2023-11-20 15:43:22,498 INFO [train_asr.py:1289] (3/4) Training started 2023-11-20 15:43:22,498 INFO [train_asr.py:1299] (3/4) Device: cuda:3 2023-11-20 15:43:22,503 INFO [train_asr.py:1311] (3/4) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 50, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'warm_step': 2000, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '2b2ac14b326d61d79d04e53fbd69b1ff6d630411', 'k2-git-date': 'Thu Aug 24 05:58:26 2023', 'lhotse-version': '1.16.0', 'torch-version': '2.0.1+cu117', 'torch-cuda-available': True, 'torch-cuda-version': '11.7', 'python-version': '3.1', 'icefall-git-branch': 'multi_KD', 'icefall-git-sha1': '16e77b48-dirty', 'icefall-git-date': 'Mon Nov 20 11:32:19 2023', 'icefall-path': '/star-xy/softwares/icefall_development/icefall_multi_KD', 'k2-path': '/star-xy/softwares/k2_development/k2/k2/python/k2/__init__.py', 'lhotse-path': '/star-xy/softwares/anaconda3/envs/multi_KD/lib/python3.10/site-packages/lhotse/__init__.py', 'hostname': 'de-74279-k2-train-6-0423201309-7c68fd68fb-qfn6b', 'IP address': '10.177.58.19'}, 'world_size': 4, 'master_port': 13490, 'tensorboard': True, 'num_epochs': 40, 'start_epoch': 15, 'start_batch': 0, 'exp_dir': PosixPath('multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0'), 'bpe_model': 'data/lang_bpe_500/bpe.model', 'base_lr': 0.045, 'lr_batches': 7500, 'lr_epochs': 3.5, 'ref_duration': 600, 'context_size': 2, 'prune_range': 5, 'lm_scale': 0.25, 'am_scale': 0.0, 'simple_loss_scale': 0.5, 'ctc_loss_scale': 0.2, 'audio_tagging_loss_scale': 1.0, 'seed': 42, 'print_diagnostics': False, 'inf_check': False, 'save_every_n': 4000, 'keep_last_k': 30, 'average_period': 200, 'use_fp16': True, 'do_finetune': False, 'init_modules': None, 'freeze_modules': None, 'finetune_ckpt': None, 'num_encoder_layers': '2,2,3,4,3,2', 'downsampling_factor': '1,2,4,8,4,2', 'feedforward_dim': '512,768,1024,1536,1024,768', 'num_heads': '4,4,4,8,4,4', 'encoder_dim': '192,256,384,512,384,256', 'query_head_dim': '32', 'value_head_dim': '12', 'pos_head_dim': '4', 'pos_dim': 48, 'encoder_unmasked_dim': '192,192,256,256,256,192', 'cnn_module_kernel': '31,31,15,15,15,31', 'decoder_dim': 512, 'joiner_dim': 512, 'causal': False, 'chunk_size': '16,32,64,-1', 'left_context_frames': '64,128,256,-1', 'use_transducer': True, 'use_ctc': False, 'do_audio_tagging': True, 'use_encoder_projection': False, 'encoder_projection_dim': -1, 'freeze_encoder': False, 'freezing_encoder_layer_index': '-1', 'freeze_encoder_steps': -1, 'encoder_lr_scale': 1.0, 'full_libri': True, 'mini_libri': False, 'use_vox2': False, 'use_libriheavy': False, 'libriheavy_subset': 'small', 'use_audioset': True, 'audioset_subset': 'unbalanced', 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 1000, 'bucketing_sampler': False, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'enable_audioset': False, 'use_musan_separately': False, 'input_strategy': 'PrecomputedFeatures', 'drop_features': False, 'return_audio': False, 'use_beats': True, 'use_ecapa': True, 'use_whisper': True, 'whisper_mvq': False, 'beats_ckpt': 'data/models/BEATs/BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt', 'whisper_version': 'small.en', 'blank_id': 0, 'vocab_size': 500} 2023-11-20 15:43:22,503 INFO [train_asr.py:1320] (3/4) About to create model 2023-11-20 15:43:23,554 INFO [train_asr.py:1324] (3/4) Number of model parameters: 65819362 2023-11-20 15:43:23,555 INFO [checkpoint.py:112] (3/4) Loading checkpoint from multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-14.pt 2023-11-20 15:43:27,130 INFO [train_asr.py:1352] (3/4) Setting the lr scale of parameters in encoder and encoder_embed to 1.0 2023-11-20 15:43:31,915 INFO [train_asr.py:1361] (3/4) Using DDP 2023-11-20 15:43:32,373 INFO [train_asr.py:1384] (3/4) Loading optimizer state dict 2023-11-20 15:43:33,139 INFO [train_asr.py:1392] (3/4) Loading scheduler state dict 2023-11-20 15:43:33,159 INFO [train_asr.py:1414] (3/4) Getting audioset cuts 2023-11-20 15:43:33,159 INFO [kd_datamodule.py:796] (3/4) About to get the audioset cuts. 2023-11-20 15:43:33,171 INFO [train_asr.py:1420] (3/4) Using mux to combine Librispeech with audioset 2023-11-20 15:43:33,171 INFO [train_asr.py:1430] (3/4) CutSet(len=2748469) [underlying data type: ] 2023-11-20 15:43:48,707 INFO [kd_datamodule.py:396] (3/4) Enable MUSAN 2023-11-20 15:43:48,708 INFO [kd_datamodule.py:397] (3/4) About to get Musan cuts 2023-11-20 15:43:52,315 INFO [kd_datamodule.py:427] (3/4) Enable SpecAugment 2023-11-20 15:43:52,315 INFO [kd_datamodule.py:428] (3/4) Time warp factor: 80 2023-11-20 15:43:52,315 INFO [kd_datamodule.py:438] (3/4) Num frame mask: 10 2023-11-20 15:43:52,315 INFO [kd_datamodule.py:451] (3/4) About to create train dataset 2023-11-20 15:43:52,317 INFO [kd_datamodule.py:487] (3/4) Using SimpleCutSampler 2023-11-20 15:43:52,317 INFO [kd_datamodule.py:495] (3/4) About to create train dataloader 2023-11-20 15:43:52,320 INFO [kd_datamodule.py:814] (3/4) About to get the audioset eval cuts. 2023-11-20 15:43:52,321 INFO [train_asr.py:1494] (3/4) CutSet(len=20681) [underlying data type: ] 2023-11-20 15:43:52,418 INFO [kd_datamodule.py:529] (3/4) About to create dev dataset 2023-11-20 15:43:53,233 INFO [kd_datamodule.py:550] (3/4) About to create dev dataloader 2023-11-20 15:43:53,234 INFO [train_asr.py:1508] (3/4) Loading grad scaler state dict 2023-11-20 15:44:29,644 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.78 vs. limit=15.0 2023-11-20 15:44:29,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.35 vs. limit=22.5 2023-11-20 15:44:30,173 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 0, loss[loss=0.1002, simple_loss=0.1183, pruned_loss=0.02247, audio_tagging_loss=0.01856, over 15251.00 frames. ], tot_loss[loss=0.1002, simple_loss=0.1183, pruned_loss=0.02247, audio_tagging_loss=0.01856, over 15251.00 frames. ], batch size: 55, lr: 4.68e-03, grad_scale: 32.0 2023-11-20 15:44:30,173 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 15:45:06,708 INFO [train_asr.py:1253] (3/4) Epoch 15, validation: loss=0.06153, simple_loss=0.05347, pruned_loss=0.005654, audio_tagging_loss=0.02914, over 4681554.00 frames. 2023-11-20 15:45:06,709 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 15:45:08,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1122200.0, ans=0.1 2023-11-20 15:45:12,401 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.292e+01 9.006e+01 9.945e+01 1.226e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-20 15:45:32,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168350 2023-11-20 15:45:38,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1122333.3333333333, ans=0.0 2023-11-20 15:45:39,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1122333.3333333333, ans=0.125 2023-11-20 15:45:49,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1122400.0, ans=0.125 2023-11-20 15:46:12,529 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 50, loss[loss=0.06827, simple_loss=0.07209, pruned_loss=0.01185, audio_tagging_loss=0.02038, over 14390.00 frames. ], tot_loss[loss=0.09002, simple_loss=0.1025, pruned_loss=0.01997, audio_tagging_loss=0.01882, over 690218.93 frames. ], batch size: 55, lr: 4.67e-03, grad_scale: 32.0 2023-11-20 15:46:36,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1122600.0, ans=0.125 2023-11-20 15:46:38,043 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168400 2023-11-20 15:46:50,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1122666.6666666667, ans=0.0 2023-11-20 15:47:00,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1122733.3333333333, ans=0.0 2023-11-20 15:47:11,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1122800.0, ans=0.125 2023-11-20 15:47:21,496 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 100, loss[loss=0.1007, simple_loss=0.1203, pruned_loss=0.02629, audio_tagging_loss=0.01424, over 14842.00 frames. ], tot_loss[loss=0.08802, simple_loss=0.1005, pruned_loss=0.01948, audio_tagging_loss=0.0183, over 1215326.34 frames. ], batch size: 54, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:47:27,773 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.479e+01 8.856e+01 9.417e+01 9.999e+01 1.434e+02, threshold=1.883e+02, percent-clipped=0.0 2023-11-20 15:47:44,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168450 2023-11-20 15:48:07,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1123066.6666666667, ans=0.125 2023-11-20 15:48:14,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1123133.3333333333, ans=0.0 2023-11-20 15:48:22,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1123133.3333333333, ans=0.07 2023-11-20 15:48:27,317 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 150, loss[loss=0.1055, simple_loss=0.1349, pruned_loss=0.02787, audio_tagging_loss=0.0102, over 15468.00 frames. ], tot_loss[loss=0.08541, simple_loss=0.09946, pruned_loss=0.01906, audio_tagging_loss=0.01662, over 1620257.23 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:48:32,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1123200.0, ans=0.0 2023-11-20 15:48:41,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1123266.6666666667, ans=0.0 2023-11-20 15:48:50,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168500 2023-11-20 15:49:03,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1123333.3333333333, ans=0.025 2023-11-20 15:49:13,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1123400.0, ans=0.125 2023-11-20 15:49:31,873 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 200, loss[loss=0.05276, simple_loss=0.06343, pruned_loss=0.01197, audio_tagging_loss=0.009079, over 15379.00 frames. ], tot_loss[loss=0.08348, simple_loss=0.09958, pruned_loss=0.0191, audio_tagging_loss=0.01459, over 1930296.22 frames. ], batch size: 59, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:49:32,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1123533.3333333333, ans=0.0 2023-11-20 15:49:32,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1123533.3333333333, ans=0.0 2023-11-20 15:49:32,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1123533.3333333333, ans=0.1 2023-11-20 15:49:38,088 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.283e+01 8.917e+01 9.795e+01 1.407e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-20 15:49:40,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.53 vs. limit=6.0 2023-11-20 15:49:56,499 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168550 2023-11-20 15:50:25,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1123800.0, ans=0.125 2023-11-20 15:50:31,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1123800.0, ans=0.0 2023-11-20 15:50:38,420 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 250, loss[loss=0.0957, simple_loss=0.1236, pruned_loss=0.02491, audio_tagging_loss=0.008967, over 16360.00 frames. ], tot_loss[loss=0.08297, simple_loss=0.1007, pruned_loss=0.01947, audio_tagging_loss=0.01317, over 2178424.26 frames. ], batch size: 59, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:50:49,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1123933.3333333333, ans=0.125 2023-11-20 15:51:00,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168600 2023-11-20 15:51:20,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1124066.6666666667, ans=0.0 2023-11-20 15:51:30,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1124133.3333333333, ans=0.125 2023-11-20 15:51:31,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1124133.3333333333, ans=0.125 2023-11-20 15:51:43,535 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 300, loss[loss=0.06349, simple_loss=0.08634, pruned_loss=0.01301, audio_tagging_loss=0.007309, over 15904.00 frames. ], tot_loss[loss=0.08176, simple_loss=0.1002, pruned_loss=0.01942, audio_tagging_loss=0.01223, over 2369471.18 frames. ], batch size: 58, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:51:49,698 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.496e+01 9.198e+01 1.000e+02 1.340e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-20 15:52:01,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1124266.6666666667, ans=0.0 2023-11-20 15:52:05,915 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168650 2023-11-20 15:52:29,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.44 vs. limit=12.0 2023-11-20 15:52:47,374 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 350, loss[loss=0.1118, simple_loss=0.136, pruned_loss=0.03374, audio_tagging_loss=0.01006, over 15229.00 frames. ], tot_loss[loss=0.08079, simple_loss=0.1003, pruned_loss=0.01918, audio_tagging_loss=0.01145, over 2526344.93 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:53:04,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1124600.0, ans=0.015 2023-11-20 15:53:06,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1124600.0, ans=0.0 2023-11-20 15:53:11,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168700 2023-11-20 15:53:11,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1124600.0, ans=0.2 2023-11-20 15:53:15,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1124666.6666666667, ans=0.125 2023-11-20 15:53:18,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1124666.6666666667, ans=0.125 2023-11-20 15:53:20,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1124666.6666666667, ans=0.125 2023-11-20 15:53:24,283 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 15:53:40,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1124800.0, ans=0.125 2023-11-20 15:53:44,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.67 vs. limit=15.0 2023-11-20 15:53:50,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1124800.0, ans=0.0 2023-11-20 15:53:52,668 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 400, loss[loss=0.08451, simple_loss=0.1114, pruned_loss=0.01901, audio_tagging_loss=0.009777, over 15416.00 frames. ], tot_loss[loss=0.08115, simple_loss=0.1015, pruned_loss=0.01939, audio_tagging_loss=0.01098, over 2643329.14 frames. ], batch size: 58, lr: 4.67e-03, grad_scale: 32.0 2023-11-20 15:53:59,456 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.311e+01 9.076e+01 1.027e+02 1.296e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-20 15:54:03,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1124866.6666666667, ans=0.0 2023-11-20 15:54:15,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168750 2023-11-20 15:54:33,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.76 vs. limit=15.0 2023-11-20 15:54:56,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1125200.0, ans=0.07 2023-11-20 15:54:57,583 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 450, loss[loss=0.06654, simple_loss=0.07663, pruned_loss=0.01761, audio_tagging_loss=0.01062, over 15322.00 frames. ], tot_loss[loss=0.08048, simple_loss=0.1007, pruned_loss=0.01937, audio_tagging_loss=0.01073, over 2732960.94 frames. ], batch size: 59, lr: 4.67e-03, grad_scale: 32.0 2023-11-20 15:55:09,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.15 vs. limit=15.0 2023-11-20 15:55:20,004 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168800 2023-11-20 15:55:20,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=12.0 2023-11-20 15:55:20,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.96 vs. limit=12.0 2023-11-20 15:55:36,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1125400.0, ans=0.125 2023-11-20 15:55:36,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.11 vs. limit=22.5 2023-11-20 15:55:58,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.47 vs. limit=12.0 2023-11-20 15:56:00,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1125466.6666666667, ans=0.125 2023-11-20 15:56:03,106 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 500, loss[loss=0.09067, simple_loss=0.1187, pruned_loss=0.02094, audio_tagging_loss=0.0104, over 15537.00 frames. ], tot_loss[loss=0.08067, simple_loss=0.1014, pruned_loss=0.01945, audio_tagging_loss=0.01053, over 2804405.40 frames. ], batch size: 58, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:56:10,680 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.084e+01 8.651e+01 9.631e+01 1.244e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-20 15:56:22,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1125600.0, ans=0.125 2023-11-20 15:56:27,769 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168850 2023-11-20 15:56:38,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1125666.6666666667, ans=0.0 2023-11-20 15:56:38,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1125666.6666666667, ans=0.125 2023-11-20 15:56:39,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1125666.6666666667, ans=0.125 2023-11-20 15:56:56,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1125800.0, ans=0.1 2023-11-20 15:57:07,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.81 vs. limit=22.5 2023-11-20 15:57:10,365 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 550, loss[loss=0.0843, simple_loss=0.111, pruned_loss=0.01917, audio_tagging_loss=0.009628, over 15087.00 frames. ], tot_loss[loss=0.08012, simple_loss=0.101, pruned_loss=0.01928, audio_tagging_loss=0.01035, over 2854692.39 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:57:15,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1125866.6666666667, ans=0.0 2023-11-20 15:57:25,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1125933.3333333333, ans=0.1 2023-11-20 15:57:26,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1125933.3333333333, ans=0.125 2023-11-20 15:57:34,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168900 2023-11-20 15:57:47,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1126066.6666666667, ans=0.015 2023-11-20 15:58:02,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1126133.3333333333, ans=0.0 2023-11-20 15:58:16,246 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 600, loss[loss=0.08619, simple_loss=0.1116, pruned_loss=0.0221, audio_tagging_loss=0.008287, over 16119.00 frames. ], tot_loss[loss=0.08007, simple_loss=0.1008, pruned_loss=0.01937, audio_tagging_loss=0.01028, over 2894396.71 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:58:23,548 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 7.985e+01 8.751e+01 9.752e+01 1.592e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-20 15:58:38,518 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 168950 2023-11-20 15:58:50,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1126333.3333333333, ans=0.0 2023-11-20 15:58:59,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1126400.0, ans=0.125 2023-11-20 15:59:00,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1126400.0, ans=0.1 2023-11-20 15:59:20,630 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 650, loss[loss=0.09151, simple_loss=0.115, pruned_loss=0.02232, audio_tagging_loss=0.01167, over 16579.00 frames. ], tot_loss[loss=0.08002, simple_loss=0.101, pruned_loss=0.01927, audio_tagging_loss=0.01026, over 2929661.65 frames. ], batch size: 60, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 15:59:27,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1126533.3333333333, ans=0.0 2023-11-20 15:59:41,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1126600.0, ans=0.125 2023-11-20 15:59:44,726 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169000 2023-11-20 15:59:46,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.84 vs. limit=10.0 2023-11-20 15:59:48,066 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 15:59:52,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.52 vs. limit=22.5 2023-11-20 16:00:02,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1126733.3333333333, ans=15.0 2023-11-20 16:00:09,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1126733.3333333333, ans=0.0 2023-11-20 16:00:14,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1126800.0, ans=0.2 2023-11-20 16:00:26,793 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 700, loss[loss=0.1001, simple_loss=0.1241, pruned_loss=0.02909, audio_tagging_loss=0.008978, over 14427.00 frames. ], tot_loss[loss=0.07969, simple_loss=0.1008, pruned_loss=0.01907, audio_tagging_loss=0.0102, over 2962144.29 frames. ], batch size: 55, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 16:00:31,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1126866.6666666667, ans=0.1 2023-11-20 16:00:34,830 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.436e+01 8.027e+01 8.656e+01 9.309e+01 1.974e+02, threshold=1.731e+02, percent-clipped=1.0 2023-11-20 16:00:50,221 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169050 2023-11-20 16:00:55,536 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.90 vs. limit=15.0 2023-11-20 16:00:56,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1127000.0, ans=0.125 2023-11-20 16:00:59,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1127000.0, ans=0.125 2023-11-20 16:01:08,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1127066.6666666667, ans=0.125 2023-11-20 16:01:14,879 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.54 vs. limit=22.5 2023-11-20 16:01:17,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1127133.3333333333, ans=0.125 2023-11-20 16:01:23,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1127133.3333333333, ans=0.125 2023-11-20 16:01:28,654 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.10 vs. limit=12.0 2023-11-20 16:01:31,606 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 750, loss[loss=0.1036, simple_loss=0.132, pruned_loss=0.02814, audio_tagging_loss=0.009436, over 15493.00 frames. ], tot_loss[loss=0.07965, simple_loss=0.1008, pruned_loss=0.01904, audio_tagging_loss=0.01018, over 2988320.92 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0 2023-11-20 16:01:36,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1127200.0, ans=0.125 2023-11-20 16:01:48,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1127266.6666666667, ans=0.125 2023-11-20 16:01:53,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1127266.6666666667, ans=0.125 2023-11-20 16:01:54,451 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169100 2023-11-20 16:01:57,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1127333.3333333333, ans=0.2 2023-11-20 16:02:04,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1127333.3333333333, ans=0.2 2023-11-20 16:02:14,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1127400.0, ans=0.5 2023-11-20 16:02:36,486 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 800, loss[loss=0.09004, simple_loss=0.1065, pruned_loss=0.02672, audio_tagging_loss=0.01006, over 15715.00 frames. ], tot_loss[loss=0.07973, simple_loss=0.101, pruned_loss=0.01913, audio_tagging_loss=0.01008, over 3004081.93 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:02:43,908 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.081e+01 8.651e+01 9.218e+01 1.161e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-20 16:03:00,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169150 2023-11-20 16:03:04,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1127666.6666666667, ans=0.1 2023-11-20 16:03:10,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1127666.6666666667, ans=0.0 2023-11-20 16:03:11,652 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.47 vs. limit=8.0 2023-11-20 16:03:27,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1127800.0, ans=0.1 2023-11-20 16:03:41,745 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 850, loss[loss=0.07711, simple_loss=0.1095, pruned_loss=0.01632, audio_tagging_loss=0.006065, over 14754.00 frames. ], tot_loss[loss=0.0797, simple_loss=0.101, pruned_loss=0.01901, audio_tagging_loss=0.01021, over 3015796.50 frames. ], batch size: 55, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:03:59,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1127933.3333333333, ans=0.125 2023-11-20 16:04:05,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169200 2023-11-20 16:04:11,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.09 vs. limit=15.0 2023-11-20 16:04:20,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1128066.6666666667, ans=0.2 2023-11-20 16:04:42,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1128133.3333333333, ans=0.125 2023-11-20 16:04:45,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1128133.3333333333, ans=0.0 2023-11-20 16:04:48,001 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 900, loss[loss=0.07756, simple_loss=0.1006, pruned_loss=0.01823, audio_tagging_loss=0.009016, over 15184.00 frames. ], tot_loss[loss=0.07874, simple_loss=0.09977, pruned_loss=0.01863, audio_tagging_loss=0.01022, over 3021829.11 frames. ], batch size: 56, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:04:49,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1128200.0, ans=0.0 2023-11-20 16:04:53,712 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.75 vs. limit=12.0 2023-11-20 16:04:54,800 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.99 vs. limit=12.0 2023-11-20 16:04:55,436 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.050e+01 8.717e+01 9.449e+01 1.348e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-20 16:05:00,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1128266.6666666667, ans=0.0 2023-11-20 16:05:11,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169250 2023-11-20 16:05:15,994 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.81 vs. limit=6.0 2023-11-20 16:05:48,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1128466.6666666667, ans=0.1 2023-11-20 16:05:52,999 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 950, loss[loss=0.06146, simple_loss=0.08932, pruned_loss=0.00911, audio_tagging_loss=0.007685, over 16176.00 frames. ], tot_loss[loss=0.07883, simple_loss=0.1001, pruned_loss=0.01872, audio_tagging_loss=0.01007, over 3025863.60 frames. ], batch size: 58, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:05:56,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1128533.3333333333, ans=0.125 2023-11-20 16:06:17,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169300 2023-11-20 16:06:28,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1128666.6666666667, ans=0.1 2023-11-20 16:06:35,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1128733.3333333333, ans=0.07 2023-11-20 16:06:46,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1128800.0, ans=0.125 2023-11-20 16:06:57,909 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1000, loss[loss=0.07838, simple_loss=0.1039, pruned_loss=0.01968, audio_tagging_loss=0.006775, over 15445.00 frames. ], tot_loss[loss=0.07843, simple_loss=0.09928, pruned_loss=0.01882, audio_tagging_loss=0.009973, over 3027738.62 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:07:04,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1128866.6666666667, ans=0.0 2023-11-20 16:07:04,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1128866.6666666667, ans=0.2 2023-11-20 16:07:06,526 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.366e+01 7.987e+01 8.758e+01 9.641e+01 1.276e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-20 16:07:22,269 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169350 2023-11-20 16:07:24,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1129000.0, ans=0.0 2023-11-20 16:07:25,956 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 16:07:52,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.74 vs. limit=15.0 2023-11-20 16:08:01,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.39 vs. limit=15.0 2023-11-20 16:08:04,566 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1050, loss[loss=0.05437, simple_loss=0.06702, pruned_loss=0.01154, audio_tagging_loss=0.009314, over 15300.00 frames. ], tot_loss[loss=0.0786, simple_loss=0.09937, pruned_loss=0.01896, audio_tagging_loss=0.009956, over 3027581.16 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:08:06,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1129200.0, ans=0.125 2023-11-20 16:08:07,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1129200.0, ans=0.0 2023-11-20 16:08:27,024 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169400 2023-11-20 16:09:09,317 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1100, loss[loss=0.08631, simple_loss=0.1165, pruned_loss=0.02084, audio_tagging_loss=0.007236, over 15343.00 frames. ], tot_loss[loss=0.07859, simple_loss=0.0994, pruned_loss=0.01897, audio_tagging_loss=0.009919, over 3030663.32 frames. ], batch size: 60, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:09:11,878 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 16:09:13,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1129533.3333333333, ans=0.0 2023-11-20 16:09:13,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1129533.3333333333, ans=0.1 2023-11-20 16:09:16,778 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.137e+01 8.767e+01 9.510e+01 1.319e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-20 16:09:18,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1129533.3333333333, ans=0.125 2023-11-20 16:09:20,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1129600.0, ans=0.125 2023-11-20 16:09:32,993 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169450 2023-11-20 16:09:44,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1129666.6666666667, ans=0.2 2023-11-20 16:09:47,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1129733.3333333333, ans=0.2 2023-11-20 16:10:06,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1129800.0, ans=0.0 2023-11-20 16:10:07,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1129800.0, ans=0.1 2023-11-20 16:10:08,242 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.25 vs. limit=22.5 2023-11-20 16:10:13,865 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1150, loss[loss=0.07823, simple_loss=0.1015, pruned_loss=0.01977, audio_tagging_loss=0.007735, over 15211.00 frames. ], tot_loss[loss=0.07838, simple_loss=0.09931, pruned_loss=0.01882, audio_tagging_loss=0.009906, over 3029901.75 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:10:27,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1129933.3333333333, ans=0.0 2023-11-20 16:10:38,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169500 2023-11-20 16:10:45,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1130000.0, ans=0.0 2023-11-20 16:11:01,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1130066.6666666667, ans=0.125 2023-11-20 16:11:01,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1130066.6666666667, ans=0.0 2023-11-20 16:11:06,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1130133.3333333333, ans=0.125 2023-11-20 16:11:19,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.19 vs. limit=10.0 2023-11-20 16:11:21,453 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1200, loss[loss=0.08704, simple_loss=0.1089, pruned_loss=0.02134, audio_tagging_loss=0.01125, over 15426.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.09967, pruned_loss=0.01888, audio_tagging_loss=0.009898, over 3034519.44 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 32.0 2023-11-20 16:11:30,082 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.141e+01 8.800e+01 9.449e+01 1.223e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-20 16:11:34,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.14 vs. limit=15.0 2023-11-20 16:11:42,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1130266.6666666667, ans=0.125 2023-11-20 16:11:43,989 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169550 2023-11-20 16:12:17,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1130466.6666666667, ans=0.125 2023-11-20 16:12:26,132 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1250, loss[loss=0.09368, simple_loss=0.1203, pruned_loss=0.02146, audio_tagging_loss=0.01205, over 14821.00 frames. ], tot_loss[loss=0.07914, simple_loss=0.1001, pruned_loss=0.01925, audio_tagging_loss=0.009832, over 3041556.22 frames. ], batch size: 54, lr: 4.66e-03, grad_scale: 16.0 2023-11-20 16:12:31,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1130533.3333333333, ans=0.125 2023-11-20 16:12:45,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1130600.0, ans=0.125 2023-11-20 16:12:49,062 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169600 2023-11-20 16:13:06,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1130733.3333333333, ans=0.0 2023-11-20 16:13:17,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1130800.0, ans=0.2 2023-11-20 16:13:23,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=15.97 vs. limit=15.0 2023-11-20 16:13:30,638 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1300, loss[loss=0.09548, simple_loss=0.1367, pruned_loss=0.02034, audio_tagging_loss=0.006775, over 15106.00 frames. ], tot_loss[loss=0.07882, simple_loss=0.09998, pruned_loss=0.01905, audio_tagging_loss=0.009773, over 3037706.74 frames. ], batch size: 55, lr: 4.66e-03, grad_scale: 16.0 2023-11-20 16:13:41,688 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.174e+01 8.827e+01 9.605e+01 1.804e+02, threshold=1.765e+02, percent-clipped=1.0 2023-11-20 16:13:43,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1130933.3333333333, ans=0.0 2023-11-20 16:13:46,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1130933.3333333333, ans=0.125 2023-11-20 16:13:54,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1130933.3333333333, ans=0.125 2023-11-20 16:13:55,581 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169650 2023-11-20 16:14:20,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1131066.6666666667, ans=0.2 2023-11-20 16:14:25,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1131133.3333333333, ans=0.0 2023-11-20 16:14:33,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.08 vs. limit=22.5 2023-11-20 16:14:37,372 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1350, loss[loss=0.05898, simple_loss=0.06502, pruned_loss=0.01115, audio_tagging_loss=0.01531, over 15030.00 frames. ], tot_loss[loss=0.07777, simple_loss=0.09856, pruned_loss=0.01861, audio_tagging_loss=0.009885, over 3036976.23 frames. ], batch size: 56, lr: 4.66e-03, grad_scale: 16.0 2023-11-20 16:14:50,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1131266.6666666667, ans=0.125 2023-11-20 16:15:00,489 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169700 2023-11-20 16:15:14,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1131400.0, ans=0.125 2023-11-20 16:15:18,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1131400.0, ans=0.125 2023-11-20 16:15:23,972 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 16:15:32,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1131466.6666666667, ans=0.125 2023-11-20 16:15:33,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1131466.6666666667, ans=0.0 2023-11-20 16:15:36,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1131466.6666666667, ans=0.125 2023-11-20 16:15:39,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1131466.6666666667, ans=0.2 2023-11-20 16:15:42,458 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1400, loss[loss=0.08679, simple_loss=0.109, pruned_loss=0.02214, audio_tagging_loss=0.01015, over 15433.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09923, pruned_loss=0.01894, audio_tagging_loss=0.009916, over 3041411.80 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 16.0 2023-11-20 16:15:42,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1131533.3333333333, ans=0.0 2023-11-20 16:15:52,445 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 7.986e+01 8.672e+01 9.769e+01 1.289e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-20 16:15:54,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1131600.0, ans=0.0 2023-11-20 16:16:05,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169750 2023-11-20 16:16:21,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1131733.3333333333, ans=0.125 2023-11-20 16:16:36,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1131800.0, ans=0.125 2023-11-20 16:16:40,511 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.64 vs. limit=15.0 2023-11-20 16:16:47,119 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1450, loss[loss=0.07863, simple_loss=0.09802, pruned_loss=0.02022, audio_tagging_loss=0.009403, over 14844.00 frames. ], tot_loss[loss=0.07852, simple_loss=0.09911, pruned_loss=0.01898, audio_tagging_loss=0.009992, over 3039816.67 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 16.0 2023-11-20 16:16:48,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1131866.6666666667, ans=0.0 2023-11-20 16:16:53,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1131866.6666666667, ans=0.125 2023-11-20 16:17:10,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1131933.3333333333, ans=0.0 2023-11-20 16:17:11,308 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169800 2023-11-20 16:17:14,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1132000.0, ans=0.0 2023-11-20 16:17:15,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1132000.0, ans=0.0 2023-11-20 16:17:24,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1132000.0, ans=0.125 2023-11-20 16:17:42,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1132133.3333333333, ans=0.2 2023-11-20 16:17:52,589 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1500, loss[loss=0.0503, simple_loss=0.05175, pruned_loss=0.0117, audio_tagging_loss=0.01272, over 14207.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.09809, pruned_loss=0.01885, audio_tagging_loss=0.01014, over 3034293.07 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 8.0 2023-11-20 16:18:02,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1132200.0, ans=0.125 2023-11-20 16:18:02,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1132200.0, ans=0.125 2023-11-20 16:18:04,950 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.236e+01 9.090e+01 9.607e+01 1.235e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-20 16:18:16,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169850 2023-11-20 16:18:25,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=13.55 vs. limit=15.0 2023-11-20 16:18:43,217 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.93 vs. limit=15.0 2023-11-20 16:18:55,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1132466.6666666667, ans=0.1 2023-11-20 16:18:58,742 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1550, loss[loss=0.08633, simple_loss=0.118, pruned_loss=0.02122, audio_tagging_loss=0.006131, over 15142.00 frames. ], tot_loss[loss=0.07851, simple_loss=0.09895, pruned_loss=0.01897, audio_tagging_loss=0.01007, over 3037468.72 frames. ], batch size: 57, lr: 4.65e-03, grad_scale: 8.0 2023-11-20 16:19:20,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1132600.0, ans=0.125 2023-11-20 16:19:21,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169900 2023-11-20 16:19:46,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1132733.3333333333, ans=0.2 2023-11-20 16:20:03,521 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1600, loss[loss=0.075, simple_loss=0.1049, pruned_loss=0.01408, audio_tagging_loss=0.008488, over 15408.00 frames. ], tot_loss[loss=0.07814, simple_loss=0.0984, pruned_loss=0.01882, audio_tagging_loss=0.01013, over 3039193.95 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:20:14,716 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.371e+01 8.069e+01 8.651e+01 9.390e+01 1.565e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-20 16:20:18,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.96 vs. limit=12.0 2023-11-20 16:20:21,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1132933.3333333333, ans=0.95 2023-11-20 16:20:21,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.66 vs. limit=15.0 2023-11-20 16:20:27,764 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 169950 2023-11-20 16:20:42,349 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 16:20:43,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1133066.6666666667, ans=0.125 2023-11-20 16:20:43,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1133066.6666666667, ans=0.5 2023-11-20 16:20:45,616 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.77 vs. limit=15.0 2023-11-20 16:20:46,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1133066.6666666667, ans=0.2 2023-11-20 16:20:47,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1133066.6666666667, ans=0.0 2023-11-20 16:21:10,152 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1650, loss[loss=0.07758, simple_loss=0.09756, pruned_loss=0.01697, audio_tagging_loss=0.01183, over 16504.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09796, pruned_loss=0.01873, audio_tagging_loss=0.01021, over 3038511.75 frames. ], batch size: 61, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:21:31,738 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.80 vs. limit=6.0 2023-11-20 16:21:33,783 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170000 2023-11-20 16:21:45,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1133333.3333333333, ans=0.125 2023-11-20 16:22:01,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1133466.6666666667, ans=0.2 2023-11-20 16:22:12,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1133466.6666666667, ans=0.125 2023-11-20 16:22:15,918 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1700, loss[loss=0.07675, simple_loss=0.09678, pruned_loss=0.01809, audio_tagging_loss=0.01026, over 15294.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09774, pruned_loss=0.01852, audio_tagging_loss=0.01024, over 3036722.66 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:22:27,587 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.019e+01 8.780e+01 9.587e+01 1.403e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-20 16:22:38,977 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170050 2023-11-20 16:22:41,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1133666.6666666667, ans=0.0 2023-11-20 16:22:47,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1133666.6666666667, ans=0.0 2023-11-20 16:22:48,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1133666.6666666667, ans=0.0 2023-11-20 16:23:00,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.90 vs. limit=15.0 2023-11-20 16:23:21,173 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1750, loss[loss=0.07399, simple_loss=0.08854, pruned_loss=0.02095, audio_tagging_loss=0.008769, over 13301.00 frames. ], tot_loss[loss=0.07801, simple_loss=0.0987, pruned_loss=0.01868, audio_tagging_loss=0.009973, over 3042201.77 frames. ], batch size: 54, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:23:44,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170100 2023-11-20 16:23:50,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1134000.0, ans=0.025 2023-11-20 16:23:55,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1134000.0, ans=0.125 2023-11-20 16:24:11,419 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 16:24:12,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1134133.3333333333, ans=0.125 2023-11-20 16:24:26,781 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1800, loss[loss=0.09187, simple_loss=0.1106, pruned_loss=0.02778, audio_tagging_loss=0.008816, over 15135.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09816, pruned_loss=0.01854, audio_tagging_loss=0.01002, over 3041189.90 frames. ], batch size: 57, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:24:30,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1134200.0, ans=0.125 2023-11-20 16:24:39,064 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.001e+01 8.710e+01 9.346e+01 1.351e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-20 16:24:51,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170150 2023-11-20 16:24:52,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1134333.3333333333, ans=0.125 2023-11-20 16:25:02,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1134333.3333333333, ans=0.125 2023-11-20 16:25:20,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1134466.6666666667, ans=0.125 2023-11-20 16:25:21,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1134466.6666666667, ans=0.125 2023-11-20 16:25:32,397 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1850, loss[loss=0.08938, simple_loss=0.115, pruned_loss=0.02426, audio_tagging_loss=0.007646, over 15105.00 frames. ], tot_loss[loss=0.07855, simple_loss=0.0991, pruned_loss=0.01907, audio_tagging_loss=0.009925, over 3041178.74 frames. ], batch size: 54, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:25:32,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1134533.3333333333, ans=0.2 2023-11-20 16:25:33,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1134533.3333333333, ans=0.125 2023-11-20 16:25:34,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1134533.3333333333, ans=0.125 2023-11-20 16:25:44,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1134600.0, ans=0.1 2023-11-20 16:25:55,435 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170200 2023-11-20 16:26:02,066 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.87 vs. limit=15.0 2023-11-20 16:26:03,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1134666.6666666667, ans=0.125 2023-11-20 16:26:16,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1134733.3333333333, ans=0.125 2023-11-20 16:26:34,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1134800.0, ans=0.0 2023-11-20 16:26:38,606 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1900, loss[loss=0.08342, simple_loss=0.1057, pruned_loss=0.02228, audio_tagging_loss=0.0083, over 15189.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09877, pruned_loss=0.01896, audio_tagging_loss=0.009869, over 3042092.24 frames. ], batch size: 57, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:26:49,714 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.240e+01 7.901e+01 8.698e+01 9.524e+01 1.257e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-20 16:27:02,082 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170250 2023-11-20 16:27:11,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1135000.0, ans=0.0 2023-11-20 16:27:16,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1135066.6666666667, ans=0.2 2023-11-20 16:27:36,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1135133.3333333333, ans=0.125 2023-11-20 16:27:36,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1135133.3333333333, ans=0.125 2023-11-20 16:27:43,332 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 1950, loss[loss=0.08796, simple_loss=0.1174, pruned_loss=0.02156, audio_tagging_loss=0.007679, over 15886.00 frames. ], tot_loss[loss=0.07848, simple_loss=0.09903, pruned_loss=0.01913, audio_tagging_loss=0.009837, over 3047958.64 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:28:07,887 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170300 2023-11-20 16:28:18,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1135333.3333333333, ans=0.125 2023-11-20 16:28:26,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.09 vs. limit=22.5 2023-11-20 16:28:42,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1135466.6666666667, ans=0.125 2023-11-20 16:28:49,405 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.27 vs. limit=15.0 2023-11-20 16:28:49,867 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2000, loss[loss=0.08947, simple_loss=0.1143, pruned_loss=0.02139, audio_tagging_loss=0.01095, over 15215.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.09832, pruned_loss=0.01892, audio_tagging_loss=0.009844, over 3051150.43 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 32.0 2023-11-20 16:29:00,831 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 7.997e+01 8.751e+01 9.649e+01 1.208e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-20 16:29:12,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170350 2023-11-20 16:29:31,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1135733.3333333333, ans=0.125 2023-11-20 16:29:34,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1135733.3333333333, ans=0.125 2023-11-20 16:29:35,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.14 vs. limit=12.0 2023-11-20 16:29:54,317 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2050, loss[loss=0.09242, simple_loss=0.1063, pruned_loss=0.03037, audio_tagging_loss=0.008921, over 15063.00 frames. ], tot_loss[loss=0.07854, simple_loss=0.09906, pruned_loss=0.01917, audio_tagging_loss=0.009845, over 3053738.13 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 32.0 2023-11-20 16:29:54,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1135866.6666666667, ans=0.125 2023-11-20 16:30:15,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1135933.3333333333, ans=0.2 2023-11-20 16:30:18,792 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170400 2023-11-20 16:30:22,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1136000.0, ans=0.0 2023-11-20 16:30:29,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.99 vs. limit=15.0 2023-11-20 16:30:37,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.40 vs. limit=15.0 2023-11-20 16:30:51,231 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0 2023-11-20 16:31:00,367 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2100, loss[loss=0.06451, simple_loss=0.08561, pruned_loss=0.01133, audio_tagging_loss=0.01038, over 14879.00 frames. ], tot_loss[loss=0.07878, simple_loss=0.09954, pruned_loss=0.01918, audio_tagging_loss=0.009834, over 3051513.04 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:31:13,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1136266.6666666667, ans=0.015 2023-11-20 16:31:13,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.29 vs. limit=22.5 2023-11-20 16:31:14,459 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.060e+01 8.301e+01 9.019e+01 9.806e+01 1.324e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-20 16:31:21,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1136266.6666666667, ans=0.0 2023-11-20 16:31:25,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170450 2023-11-20 16:31:30,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1136333.3333333333, ans=0.0 2023-11-20 16:31:45,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=12.0 2023-11-20 16:32:07,325 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2150, loss[loss=0.08331, simple_loss=0.09862, pruned_loss=0.02093, audio_tagging_loss=0.01307, over 13866.00 frames. ], tot_loss[loss=0.07874, simple_loss=0.09954, pruned_loss=0.01907, audio_tagging_loss=0.009905, over 3047472.51 frames. ], batch size: 53, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:32:13,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1136533.3333333333, ans=0.125 2023-11-20 16:32:21,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.39 vs. limit=15.0 2023-11-20 16:32:30,357 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170500 2023-11-20 16:32:46,617 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 16:32:48,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1136733.3333333333, ans=0.125 2023-11-20 16:33:12,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.36 vs. limit=15.0 2023-11-20 16:33:12,855 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2200, loss[loss=0.0733, simple_loss=0.09399, pruned_loss=0.01782, audio_tagging_loss=0.008483, over 15825.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.09961, pruned_loss=0.01895, audio_tagging_loss=0.009873, over 3053828.03 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0 2023-11-20 16:33:15,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1136866.6666666667, ans=0.0 2023-11-20 16:33:25,933 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.367e+01 8.465e+01 9.006e+01 9.663e+01 1.304e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-20 16:33:26,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1136933.3333333333, ans=0.125 2023-11-20 16:33:36,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170550 2023-11-20 16:33:41,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1137000.0, ans=0.2 2023-11-20 16:33:52,429 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.88 vs. limit=6.0 2023-11-20 16:33:54,520 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 16:33:55,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1137066.6666666667, ans=0.0 2023-11-20 16:33:59,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1137066.6666666667, ans=0.125 2023-11-20 16:34:02,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1137066.6666666667, ans=0.05 2023-11-20 16:34:07,415 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.18 vs. limit=6.0 2023-11-20 16:34:18,615 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2250, loss[loss=0.07519, simple_loss=0.09393, pruned_loss=0.01801, audio_tagging_loss=0.01022, over 14941.00 frames. ], tot_loss[loss=0.07897, simple_loss=0.1, pruned_loss=0.01908, audio_tagging_loss=0.009892, over 3050257.66 frames. ], batch size: 56, lr: 4.64e-03, grad_scale: 16.0 2023-11-20 16:34:19,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1137200.0, ans=0.1 2023-11-20 16:34:27,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1137200.0, ans=0.2 2023-11-20 16:34:28,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1137200.0, ans=0.125 2023-11-20 16:34:31,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1137266.6666666667, ans=0.05 2023-11-20 16:34:43,195 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170600 2023-11-20 16:34:52,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1137333.3333333333, ans=0.125 2023-11-20 16:35:22,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1137466.6666666667, ans=0.0 2023-11-20 16:35:24,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1137533.3333333333, ans=0.0 2023-11-20 16:35:25,852 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2300, loss[loss=0.08104, simple_loss=0.1103, pruned_loss=0.01617, audio_tagging_loss=0.009723, over 14920.00 frames. ], tot_loss[loss=0.07825, simple_loss=0.09895, pruned_loss=0.01876, audio_tagging_loss=0.01002, over 3040743.64 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 16.0 2023-11-20 16:35:38,755 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.000e+01 8.643e+01 9.323e+01 1.269e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-20 16:35:48,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170650 2023-11-20 16:36:00,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1137666.6666666667, ans=0.05 2023-11-20 16:36:23,797 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 16:36:31,359 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2350, loss[loss=0.09896, simple_loss=0.135, pruned_loss=0.02207, audio_tagging_loss=0.009402, over 16765.00 frames. ], tot_loss[loss=0.07888, simple_loss=0.0998, pruned_loss=0.01902, audio_tagging_loss=0.009964, over 3034806.55 frames. ], batch size: 60, lr: 4.64e-03, grad_scale: 16.0 2023-11-20 16:36:45,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1137933.3333333333, ans=0.125 2023-11-20 16:36:51,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1137933.3333333333, ans=0.125 2023-11-20 16:36:54,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170700 2023-11-20 16:36:59,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1138000.0, ans=0.125 2023-11-20 16:37:15,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1138066.6666666667, ans=0.125 2023-11-20 16:37:19,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1138066.6666666667, ans=15.0 2023-11-20 16:37:19,960 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.89 vs. limit=15.0 2023-11-20 16:37:29,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1138133.3333333333, ans=0.125 2023-11-20 16:37:36,635 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2400, loss[loss=0.05671, simple_loss=0.06517, pruned_loss=0.01116, audio_tagging_loss=0.01296, over 14868.00 frames. ], tot_loss[loss=0.07931, simple_loss=0.1005, pruned_loss=0.0191, audio_tagging_loss=0.00995, over 3040700.27 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:37:39,751 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.12 vs. limit=15.0 2023-11-20 16:37:50,910 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.259e+01 8.944e+01 9.702e+01 1.216e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-20 16:38:01,508 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170750 2023-11-20 16:38:17,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1138400.0, ans=0.125 2023-11-20 16:38:19,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.63 vs. limit=15.0 2023-11-20 16:38:30,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1138466.6666666667, ans=0.1 2023-11-20 16:38:43,350 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2450, loss[loss=0.07849, simple_loss=0.1047, pruned_loss=0.01901, audio_tagging_loss=0.00715, over 16128.00 frames. ], tot_loss[loss=0.07904, simple_loss=0.09985, pruned_loss=0.01902, audio_tagging_loss=0.0101, over 3044183.68 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:38:44,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1138533.3333333333, ans=0.125 2023-11-20 16:38:53,150 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.80 vs. limit=15.0 2023-11-20 16:39:02,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=12.0 2023-11-20 16:39:06,339 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170800 2023-11-20 16:39:10,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1138666.6666666667, ans=0.125 2023-11-20 16:39:20,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1138733.3333333333, ans=0.125 2023-11-20 16:39:34,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1138800.0, ans=0.125 2023-11-20 16:39:34,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1138800.0, ans=0.125 2023-11-20 16:39:42,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1138800.0, ans=0.09899494936611666 2023-11-20 16:39:47,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1138866.6666666667, ans=0.125 2023-11-20 16:39:48,714 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2500, loss[loss=0.07253, simple_loss=0.08546, pruned_loss=0.0183, audio_tagging_loss=0.01149, over 16423.00 frames. ], tot_loss[loss=0.07876, simple_loss=0.09939, pruned_loss=0.01888, audio_tagging_loss=0.01019, over 3042436.39 frames. ], batch size: 63, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:39:50,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1138866.6666666667, ans=0.0 2023-11-20 16:39:55,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1138866.6666666667, ans=0.125 2023-11-20 16:39:58,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1138866.6666666667, ans=0.1 2023-11-20 16:40:01,217 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.113e+01 8.897e+01 9.641e+01 1.351e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-20 16:40:11,315 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170850 2023-11-20 16:40:14,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.25 vs. limit=15.0 2023-11-20 16:40:53,618 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2550, loss[loss=0.07114, simple_loss=0.09547, pruned_loss=0.01606, audio_tagging_loss=0.007342, over 15651.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.09938, pruned_loss=0.01884, audio_tagging_loss=0.0101, over 3041128.73 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:40:55,381 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.15 vs. limit=15.0 2023-11-20 16:41:16,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1139266.6666666667, ans=0.125 2023-11-20 16:41:17,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170900 2023-11-20 16:41:36,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1139400.0, ans=0.2 2023-11-20 16:41:39,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1139400.0, ans=0.125 2023-11-20 16:41:48,649 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0 2023-11-20 16:41:59,387 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2600, loss[loss=0.06047, simple_loss=0.08109, pruned_loss=0.0124, audio_tagging_loss=0.007523, over 15448.00 frames. ], tot_loss[loss=0.07876, simple_loss=0.09981, pruned_loss=0.01892, audio_tagging_loss=0.009941, over 3045507.01 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:41:59,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1139533.3333333333, ans=0.07 2023-11-20 16:42:00,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1139533.3333333333, ans=0.2 2023-11-20 16:42:02,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1139533.3333333333, ans=0.07 2023-11-20 16:42:02,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1139533.3333333333, ans=0.09899494936611666 2023-11-20 16:42:05,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1139533.3333333333, ans=0.1 2023-11-20 16:42:13,173 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.062e+01 8.877e+01 9.575e+01 1.292e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-20 16:42:16,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1139600.0, ans=0.125 2023-11-20 16:42:19,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1139600.0, ans=0.125 2023-11-20 16:42:23,183 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 170950 2023-11-20 16:42:27,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1139666.6666666667, ans=0.125 2023-11-20 16:42:44,985 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.41 vs. limit=22.5 2023-11-20 16:43:05,755 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2650, loss[loss=0.08577, simple_loss=0.1027, pruned_loss=0.02549, audio_tagging_loss=0.008913, over 15411.00 frames. ], tot_loss[loss=0.07931, simple_loss=0.1009, pruned_loss=0.01909, audio_tagging_loss=0.009789, over 3044356.19 frames. ], batch size: 58, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:43:07,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1139866.6666666667, ans=0.125 2023-11-20 16:43:28,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171000 2023-11-20 16:44:11,717 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2700, loss[loss=0.09869, simple_loss=0.1258, pruned_loss=0.0265, audio_tagging_loss=0.009279, over 15037.00 frames. ], tot_loss[loss=0.07905, simple_loss=0.1006, pruned_loss=0.01899, audio_tagging_loss=0.009734, over 3044570.38 frames. ], batch size: 55, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:44:22,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.88 vs. limit=8.0 2023-11-20 16:44:24,021 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.274e+01 8.361e+01 8.930e+01 9.610e+01 1.277e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-20 16:44:35,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171050 2023-11-20 16:44:38,824 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.94 vs. limit=15.0 2023-11-20 16:45:15,527 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2750, loss[loss=0.07674, simple_loss=0.1063, pruned_loss=0.0144, audio_tagging_loss=0.009181, over 16254.00 frames. ], tot_loss[loss=0.07899, simple_loss=0.1005, pruned_loss=0.01903, audio_tagging_loss=0.009702, over 3042688.81 frames. ], batch size: 61, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:45:26,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1140533.3333333333, ans=0.125 2023-11-20 16:45:36,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1140600.0, ans=0.1 2023-11-20 16:45:39,365 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.447e-01 2023-11-20 16:45:40,325 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171100 2023-11-20 16:45:51,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.56 vs. limit=15.0 2023-11-20 16:46:09,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0 2023-11-20 16:46:12,570 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 16:46:22,420 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2800, loss[loss=0.09464, simple_loss=0.1349, pruned_loss=0.01923, audio_tagging_loss=0.007953, over 16483.00 frames. ], tot_loss[loss=0.07883, simple_loss=0.1004, pruned_loss=0.01897, audio_tagging_loss=0.009645, over 3037324.25 frames. ], batch size: 60, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:46:25,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1140866.6666666667, ans=0.0 2023-11-20 16:46:26,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1140866.6666666667, ans=0.125 2023-11-20 16:46:34,713 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 7.973e+01 8.545e+01 9.454e+01 1.250e+02, threshold=1.709e+02, percent-clipped=0.0 2023-11-20 16:46:35,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1140933.3333333333, ans=0.125 2023-11-20 16:46:44,788 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171150 2023-11-20 16:46:51,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1141000.0, ans=0.125 2023-11-20 16:46:59,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1141066.6666666667, ans=0.0 2023-11-20 16:47:01,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1141066.6666666667, ans=0.0 2023-11-20 16:47:01,533 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.80 vs. limit=6.0 2023-11-20 16:47:02,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1141066.6666666667, ans=0.0 2023-11-20 16:47:02,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1141066.6666666667, ans=0.125 2023-11-20 16:47:14,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1141133.3333333333, ans=0.1 2023-11-20 16:47:19,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1141133.3333333333, ans=0.125 2023-11-20 16:47:26,738 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2850, loss[loss=0.06754, simple_loss=0.08584, pruned_loss=0.01378, audio_tagging_loss=0.01084, over 15167.00 frames. ], tot_loss[loss=0.07773, simple_loss=0.0988, pruned_loss=0.01862, audio_tagging_loss=0.009717, over 3038368.93 frames. ], batch size: 56, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:47:42,231 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.37 vs. limit=15.0 2023-11-20 16:47:50,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171200 2023-11-20 16:47:58,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1141333.3333333333, ans=0.0 2023-11-20 16:48:19,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1141466.6666666667, ans=0.1 2023-11-20 16:48:22,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1141466.6666666667, ans=0.125 2023-11-20 16:48:31,831 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2900, loss[loss=0.08127, simple_loss=0.0983, pruned_loss=0.02378, audio_tagging_loss=0.00833, over 15994.00 frames. ], tot_loss[loss=0.07791, simple_loss=0.09891, pruned_loss=0.01873, audio_tagging_loss=0.009733, over 3043807.86 frames. ], batch size: 63, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:48:35,799 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.41 vs. limit=15.0 2023-11-20 16:48:45,412 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.270e+01 7.954e+01 8.776e+01 9.382e+01 1.409e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-20 16:48:55,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1141600.0, ans=0.125 2023-11-20 16:48:56,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171250 2023-11-20 16:49:01,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1141666.6666666667, ans=0.0 2023-11-20 16:49:37,935 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 2950, loss[loss=0.07693, simple_loss=0.09318, pruned_loss=0.01688, audio_tagging_loss=0.01345, over 14260.00 frames. ], tot_loss[loss=0.07833, simple_loss=0.09925, pruned_loss=0.01884, audio_tagging_loss=0.009868, over 3040130.63 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 32.0 2023-11-20 16:49:57,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.83 vs. limit=10.0 2023-11-20 16:50:01,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171300 2023-11-20 16:50:12,062 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.66 vs. limit=15.0 2023-11-20 16:50:14,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1142000.0, ans=0.125 2023-11-20 16:50:17,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1142066.6666666667, ans=0.125 2023-11-20 16:50:40,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1142133.3333333333, ans=0.0 2023-11-20 16:50:43,601 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3000, loss[loss=0.0932, simple_loss=0.1234, pruned_loss=0.02336, audio_tagging_loss=0.00815, over 15188.00 frames. ], tot_loss[loss=0.0786, simple_loss=0.09989, pruned_loss=0.01887, audio_tagging_loss=0.009788, over 3046740.94 frames. ], batch size: 54, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:50:43,602 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 16:51:20,104 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8813, 3.4114, 4.8019, 4.4830], device='cuda:3') 2023-11-20 16:51:23,417 INFO [train_asr.py:1253] (3/4) Epoch 15, validation: loss=0.06163, simple_loss=0.05329, pruned_loss=0.005569, audio_tagging_loss=0.02942, over 4681554.00 frames. 2023-11-20 16:51:23,418 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 16:51:36,341 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.037e+01 8.379e+01 9.112e+01 1.041e+02 3.133e+02, threshold=1.822e+02, percent-clipped=1.0 2023-11-20 16:51:36,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1142266.6666666667, ans=0.2 2023-11-20 16:51:46,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171350 2023-11-20 16:51:48,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.69 vs. limit=15.0 2023-11-20 16:52:17,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1142466.6666666667, ans=0.0 2023-11-20 16:52:22,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1142466.6666666667, ans=0.95 2023-11-20 16:52:25,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1142466.6666666667, ans=0.125 2023-11-20 16:52:28,803 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3050, loss[loss=0.1063, simple_loss=0.1296, pruned_loss=0.02994, audio_tagging_loss=0.0116, over 15309.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09951, pruned_loss=0.01882, audio_tagging_loss=0.009893, over 3046867.72 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:52:33,429 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.88 vs. limit=15.0 2023-11-20 16:52:51,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171400 2023-11-20 16:53:07,717 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 16:53:27,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1142800.0, ans=0.95 2023-11-20 16:53:33,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1142866.6666666667, ans=0.1 2023-11-20 16:53:34,120 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3100, loss[loss=0.07956, simple_loss=0.09857, pruned_loss=0.02223, audio_tagging_loss=0.008046, over 14430.00 frames. ], tot_loss[loss=0.07947, simple_loss=0.1006, pruned_loss=0.01923, audio_tagging_loss=0.009947, over 3036658.34 frames. ], batch size: 55, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:53:46,515 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.278e+01 8.696e+01 9.346e+01 1.301e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-20 16:53:57,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171450 2023-11-20 16:54:04,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1143000.0, ans=0.1 2023-11-20 16:54:27,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1143133.3333333333, ans=0.035 2023-11-20 16:54:29,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1143133.3333333333, ans=0.125 2023-11-20 16:54:35,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1143133.3333333333, ans=10.0 2023-11-20 16:54:39,658 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3150, loss[loss=0.07566, simple_loss=0.1028, pruned_loss=0.01462, audio_tagging_loss=0.009612, over 15349.00 frames. ], tot_loss[loss=0.07971, simple_loss=0.1009, pruned_loss=0.01917, audio_tagging_loss=0.01011, over 3045172.08 frames. ], batch size: 57, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:54:45,869 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 16:54:51,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1143200.0, ans=0.125 2023-11-20 16:55:03,861 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171500 2023-11-20 16:55:04,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1143266.6666666667, ans=0.125 2023-11-20 16:55:20,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.55 vs. limit=15.0 2023-11-20 16:55:22,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1143400.0, ans=0.125 2023-11-20 16:55:38,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1143466.6666666667, ans=0.125 2023-11-20 16:55:45,984 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3200, loss[loss=0.07156, simple_loss=0.09637, pruned_loss=0.01513, audio_tagging_loss=0.008249, over 15124.00 frames. ], tot_loss[loss=0.07931, simple_loss=0.1006, pruned_loss=0.01895, audio_tagging_loss=0.01005, over 3046312.84 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:55:50,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.28 vs. limit=15.0 2023-11-20 16:55:58,480 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.865e+01 8.146e+01 8.709e+01 9.815e+01 1.794e+02, threshold=1.742e+02, percent-clipped=1.0 2023-11-20 16:56:09,292 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171550 2023-11-20 16:56:40,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1143800.0, ans=0.125 2023-11-20 16:56:48,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1143800.0, ans=0.125 2023-11-20 16:56:50,358 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3250, loss[loss=0.09162, simple_loss=0.1114, pruned_loss=0.02172, audio_tagging_loss=0.01423, over 14651.00 frames. ], tot_loss[loss=0.07879, simple_loss=0.09947, pruned_loss=0.01883, audio_tagging_loss=0.01022, over 3045986.31 frames. ], batch size: 55, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:57:14,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171600 2023-11-20 16:57:25,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1144000.0, ans=10.0 2023-11-20 16:57:41,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1144133.3333333333, ans=0.0 2023-11-20 16:57:55,263 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3300, loss[loss=0.08484, simple_loss=0.1, pruned_loss=0.02651, audio_tagging_loss=0.008318, over 15615.00 frames. ], tot_loss[loss=0.0784, simple_loss=0.09881, pruned_loss=0.0187, audio_tagging_loss=0.01029, over 3050916.35 frames. ], batch size: 61, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:58:03,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.30 vs. limit=15.0 2023-11-20 16:58:07,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1144200.0, ans=0.125 2023-11-20 16:58:08,616 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.97 vs. limit=15.0 2023-11-20 16:58:09,275 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.004e+01 8.604e+01 9.340e+01 1.280e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-20 16:58:09,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1144266.6666666667, ans=0.125 2023-11-20 16:58:20,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171650 2023-11-20 16:58:31,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1144333.3333333333, ans=0.0 2023-11-20 16:59:02,015 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3350, loss[loss=0.06826, simple_loss=0.08417, pruned_loss=0.0165, audio_tagging_loss=0.009672, over 14896.00 frames. ], tot_loss[loss=0.07825, simple_loss=0.09869, pruned_loss=0.01874, audio_tagging_loss=0.01017, over 3055541.08 frames. ], batch size: 57, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 16:59:05,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1144533.3333333333, ans=0.125 2023-11-20 16:59:17,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1144600.0, ans=0.0 2023-11-20 16:59:24,712 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171700 2023-11-20 17:00:04,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1144800.0, ans=0.0 2023-11-20 17:00:06,497 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3400, loss[loss=0.07949, simple_loss=0.114, pruned_loss=0.01533, audio_tagging_loss=0.00718, over 14883.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.09869, pruned_loss=0.01855, audio_tagging_loss=0.01, over 3054588.09 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 16.0 2023-11-20 17:00:10,922 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.17 vs. limit=22.5 2023-11-20 17:00:15,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1144866.6666666667, ans=0.125 2023-11-20 17:00:20,869 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.042e+01 8.596e+01 9.172e+01 1.141e+02, threshold=1.719e+02, percent-clipped=0.0 2023-11-20 17:00:23,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.57 vs. limit=10.0 2023-11-20 17:00:30,342 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171750 2023-11-20 17:00:30,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.28 vs. limit=15.0 2023-11-20 17:00:51,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1145066.6666666667, ans=0.125 2023-11-20 17:00:56,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1145066.6666666667, ans=0.025 2023-11-20 17:01:12,075 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3450, loss[loss=0.08648, simple_loss=0.1157, pruned_loss=0.02, audio_tagging_loss=0.008644, over 15321.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09891, pruned_loss=0.01844, audio_tagging_loss=0.009915, over 3052076.06 frames. ], batch size: 55, lr: 4.63e-03, grad_scale: 16.0 2023-11-20 17:01:23,856 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.66 vs. limit=15.0 2023-11-20 17:01:31,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1145266.6666666667, ans=0.125 2023-11-20 17:01:32,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1145266.6666666667, ans=0.125 2023-11-20 17:01:36,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171800 2023-11-20 17:01:51,274 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.91 vs. limit=15.0 2023-11-20 17:01:53,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1145400.0, ans=0.0 2023-11-20 17:02:13,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1145466.6666666667, ans=0.125 2023-11-20 17:02:18,794 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3500, loss[loss=0.07763, simple_loss=0.09243, pruned_loss=0.01968, audio_tagging_loss=0.01174, over 14904.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.09799, pruned_loss=0.01824, audio_tagging_loss=0.009818, over 3055453.73 frames. ], batch size: 54, lr: 4.63e-03, grad_scale: 16.0 2023-11-20 17:02:33,118 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.582e+01 8.137e+01 8.586e+01 9.265e+01 1.254e+02, threshold=1.717e+02, percent-clipped=0.0 2023-11-20 17:02:41,989 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171850 2023-11-20 17:02:41,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1145600.0, ans=0.125 2023-11-20 17:02:43,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1145666.6666666667, ans=0.1 2023-11-20 17:02:51,712 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 17:02:59,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1145733.3333333333, ans=0.125 2023-11-20 17:03:20,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1145800.0, ans=0.125 2023-11-20 17:03:24,287 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3550, loss[loss=0.0772, simple_loss=0.09483, pruned_loss=0.02199, audio_tagging_loss=0.007792, over 15500.00 frames. ], tot_loss[loss=0.07724, simple_loss=0.09836, pruned_loss=0.01825, audio_tagging_loss=0.009812, over 3051831.48 frames. ], batch size: 57, lr: 4.63e-03, grad_scale: 16.0 2023-11-20 17:03:33,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1145866.6666666667, ans=0.1 2023-11-20 17:03:34,792 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.41 vs. limit=15.0 2023-11-20 17:03:47,510 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171900 2023-11-20 17:03:57,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.72 vs. limit=12.0 2023-11-20 17:03:58,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1146000.0, ans=0.0 2023-11-20 17:04:19,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1146133.3333333333, ans=0.0 2023-11-20 17:04:29,142 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3600, loss[loss=0.09062, simple_loss=0.1151, pruned_loss=0.02475, audio_tagging_loss=0.008322, over 14854.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09871, pruned_loss=0.01842, audio_tagging_loss=0.009824, over 3050384.52 frames. ], batch size: 54, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 17:04:30,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1146200.0, ans=0.2 2023-11-20 17:04:40,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1146200.0, ans=0.125 2023-11-20 17:04:42,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1146266.6666666667, ans=0.125 2023-11-20 17:04:44,612 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.710e+01 8.052e+01 8.613e+01 9.290e+01 1.173e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-20 17:04:53,928 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 171950 2023-11-20 17:05:01,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1146333.3333333333, ans=0.125 2023-11-20 17:05:09,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.27 vs. limit=15.0 2023-11-20 17:05:11,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1146400.0, ans=0.0 2023-11-20 17:05:25,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1146466.6666666667, ans=0.04949747468305833 2023-11-20 17:05:34,601 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3650, loss[loss=0.07559, simple_loss=0.08855, pruned_loss=0.01952, audio_tagging_loss=0.01179, over 15253.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09881, pruned_loss=0.01851, audio_tagging_loss=0.009822, over 3051330.97 frames. ], batch size: 57, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 17:05:43,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1146533.3333333333, ans=0.0 2023-11-20 17:05:52,168 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.65 vs. limit=6.0 2023-11-20 17:05:57,552 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172000 2023-11-20 17:06:09,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1146666.6666666667, ans=0.125 2023-11-20 17:06:17,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.46 vs. limit=22.5 2023-11-20 17:06:37,135 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.16 vs. limit=15.0 2023-11-20 17:06:42,353 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3700, loss[loss=0.08477, simple_loss=0.1079, pruned_loss=0.02306, audio_tagging_loss=0.007749, over 14503.00 frames. ], tot_loss[loss=0.07889, simple_loss=0.1002, pruned_loss=0.01897, audio_tagging_loss=0.00982, over 3058728.91 frames. ], batch size: 55, lr: 4.63e-03, grad_scale: 32.0 2023-11-20 17:06:56,086 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.121e+01 8.690e+01 9.493e+01 1.302e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-20 17:07:01,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1146933.3333333333, ans=0.125 2023-11-20 17:07:05,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172050 2023-11-20 17:07:09,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1147000.0, ans=0.125 2023-11-20 17:07:34,991 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.72 vs. limit=6.0 2023-11-20 17:07:35,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1147133.3333333333, ans=0.0 2023-11-20 17:07:46,857 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3750, loss[loss=0.06734, simple_loss=0.08555, pruned_loss=0.01362, audio_tagging_loss=0.01094, over 15138.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.09975, pruned_loss=0.01889, audio_tagging_loss=0.009859, over 3054963.90 frames. ], batch size: 58, lr: 4.62e-03, grad_scale: 32.0 2023-11-20 17:08:03,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1147266.6666666667, ans=0.125 2023-11-20 17:08:11,568 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172100 2023-11-20 17:08:32,666 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 17:08:44,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1147466.6666666667, ans=0.125 2023-11-20 17:08:52,213 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3800, loss[loss=0.08104, simple_loss=0.1037, pruned_loss=0.0172, audio_tagging_loss=0.01201, over 15056.00 frames. ], tot_loss[loss=0.07916, simple_loss=0.1004, pruned_loss=0.019, audio_tagging_loss=0.009966, over 3054041.91 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:09:08,178 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.643e+01 8.118e+01 8.690e+01 9.561e+01 1.262e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-20 17:09:08,927 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.81 vs. limit=22.5 2023-11-20 17:09:09,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1147600.0, ans=0.125 2023-11-20 17:09:09,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1147600.0, ans=0.05 2023-11-20 17:09:15,874 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172150 2023-11-20 17:09:28,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1147666.6666666667, ans=0.0 2023-11-20 17:09:58,580 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3850, loss[loss=0.07499, simple_loss=0.09374, pruned_loss=0.01494, audio_tagging_loss=0.01318, over 15544.00 frames. ], tot_loss[loss=0.07871, simple_loss=0.09997, pruned_loss=0.01879, audio_tagging_loss=0.009935, over 3052253.99 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:10:12,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1147933.3333333333, ans=0.2 2023-11-20 17:10:16,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1147933.3333333333, ans=0.125 2023-11-20 17:10:21,418 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172200 2023-11-20 17:10:46,102 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0 2023-11-20 17:10:54,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1148133.3333333333, ans=0.2 2023-11-20 17:11:04,021 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3900, loss[loss=0.06074, simple_loss=0.06444, pruned_loss=0.01558, audio_tagging_loss=0.01294, over 14124.00 frames. ], tot_loss[loss=0.07869, simple_loss=0.09961, pruned_loss=0.01886, audio_tagging_loss=0.01002, over 3034847.56 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:11:06,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1148200.0, ans=0.2 2023-11-20 17:11:17,316 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.70 vs. limit=22.5 2023-11-20 17:11:19,572 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.094e+01 8.942e+01 9.624e+01 1.235e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-20 17:11:23,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1148266.6666666667, ans=0.125 2023-11-20 17:11:25,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1148266.6666666667, ans=0.2 2023-11-20 17:11:28,388 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172250 2023-11-20 17:11:29,284 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.38 vs. limit=5.0 2023-11-20 17:11:36,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1148333.3333333333, ans=0.2 2023-11-20 17:11:38,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1148333.3333333333, ans=0.125 2023-11-20 17:11:51,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1148400.0, ans=0.125 2023-11-20 17:11:53,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.57 vs. limit=15.0 2023-11-20 17:11:58,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1148466.6666666667, ans=0.0 2023-11-20 17:12:08,864 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 3950, loss[loss=0.08257, simple_loss=0.1073, pruned_loss=0.02063, audio_tagging_loss=0.008292, over 15517.00 frames. ], tot_loss[loss=0.07959, simple_loss=0.1009, pruned_loss=0.01921, audio_tagging_loss=0.009908, over 3044716.62 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:12:12,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1148533.3333333333, ans=0.09899494936611666 2023-11-20 17:12:17,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1148533.3333333333, ans=0.125 2023-11-20 17:12:30,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1148600.0, ans=0.0 2023-11-20 17:12:32,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172300 2023-11-20 17:12:33,178 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.16 vs. limit=15.0 2023-11-20 17:12:34,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1148666.6666666667, ans=0.0 2023-11-20 17:12:35,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1148666.6666666667, ans=0.0 2023-11-20 17:12:51,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1148733.3333333333, ans=0.04949747468305833 2023-11-20 17:12:53,297 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.56 vs. limit=6.0 2023-11-20 17:12:59,809 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 17:13:14,349 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4000, loss[loss=0.1056, simple_loss=0.135, pruned_loss=0.02985, audio_tagging_loss=0.00822, over 15914.00 frames. ], tot_loss[loss=0.08003, simple_loss=0.1015, pruned_loss=0.01929, audio_tagging_loss=0.01001, over 3047125.70 frames. ], batch size: 60, lr: 4.62e-03, grad_scale: 32.0 2023-11-20 17:13:14,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1148866.6666666667, ans=0.1 2023-11-20 17:13:14,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1148866.6666666667, ans=0.125 2023-11-20 17:13:18,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1148866.6666666667, ans=0.035 2023-11-20 17:13:23,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1148866.6666666667, ans=0.0 2023-11-20 17:13:25,991 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 17:13:29,460 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 7.998e+01 8.708e+01 9.550e+01 1.131e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-20 17:13:35,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.09 vs. limit=15.0 2023-11-20 17:13:37,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172350 2023-11-20 17:14:01,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1149066.6666666667, ans=0.0 2023-11-20 17:14:08,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1149133.3333333333, ans=0.1 2023-11-20 17:14:18,630 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4050, loss[loss=0.09558, simple_loss=0.1249, pruned_loss=0.02471, audio_tagging_loss=0.008399, over 15772.00 frames. ], tot_loss[loss=0.08038, simple_loss=0.1021, pruned_loss=0.01942, audio_tagging_loss=0.009924, over 3048256.93 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 32.0 2023-11-20 17:14:18,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1149200.0, ans=0.2 2023-11-20 17:14:22,400 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 17:14:42,486 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172400 2023-11-20 17:14:44,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.02 vs. limit=22.5 2023-11-20 17:14:46,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1149333.3333333333, ans=0.0 2023-11-20 17:15:08,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1149400.0, ans=0.1 2023-11-20 17:15:09,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1149400.0, ans=0.125 2023-11-20 17:15:24,179 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4100, loss[loss=0.08121, simple_loss=0.1046, pruned_loss=0.01914, audio_tagging_loss=0.00977, over 14559.00 frames. ], tot_loss[loss=0.07976, simple_loss=0.1012, pruned_loss=0.01922, audio_tagging_loss=0.009946, over 3043877.48 frames. ], batch size: 54, lr: 4.62e-03, grad_scale: 32.0 2023-11-20 17:15:31,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1149533.3333333333, ans=0.0 2023-11-20 17:15:42,057 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.166e+01 8.151e+01 8.717e+01 9.713e+01 1.333e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-20 17:15:48,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172450 2023-11-20 17:16:04,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1149733.3333333333, ans=0.125 2023-11-20 17:16:13,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1149733.3333333333, ans=0.0 2023-11-20 17:16:14,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1149733.3333333333, ans=0.1 2023-11-20 17:16:28,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1149800.0, ans=0.0 2023-11-20 17:16:29,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1149866.6666666667, ans=0.125 2023-11-20 17:16:30,765 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4150, loss[loss=0.09339, simple_loss=0.1204, pruned_loss=0.02468, audio_tagging_loss=0.00849, over 16113.00 frames. ], tot_loss[loss=0.07968, simple_loss=0.1013, pruned_loss=0.01923, audio_tagging_loss=0.009807, over 3042620.87 frames. ], batch size: 58, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:16:38,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1149866.6666666667, ans=0.015 2023-11-20 17:16:48,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1149933.3333333333, ans=0.2 2023-11-20 17:16:53,827 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172500 2023-11-20 17:16:57,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1150000.0, ans=0.0 2023-11-20 17:17:02,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1150000.0, ans=0.1 2023-11-20 17:17:18,186 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 17:17:19,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1150066.6666666667, ans=0.125 2023-11-20 17:17:36,215 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4200, loss[loss=0.09796, simple_loss=0.1199, pruned_loss=0.02854, audio_tagging_loss=0.009464, over 14687.00 frames. ], tot_loss[loss=0.07928, simple_loss=0.1008, pruned_loss=0.01917, audio_tagging_loss=0.009734, over 3042404.71 frames. ], batch size: 55, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:17:52,270 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.093e+01 8.811e+01 9.474e+01 1.183e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-20 17:17:58,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1150266.6666666667, ans=0.0 2023-11-20 17:17:59,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172550 2023-11-20 17:18:15,824 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 17:18:17,652 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.599e-01 2023-11-20 17:18:40,992 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4250, loss[loss=0.0739, simple_loss=0.09425, pruned_loss=0.01639, audio_tagging_loss=0.01039, over 16061.00 frames. ], tot_loss[loss=0.07948, simple_loss=0.1012, pruned_loss=0.01922, audio_tagging_loss=0.009674, over 3050308.24 frames. ], batch size: 62, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:18:59,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1150600.0, ans=0.125 2023-11-20 17:19:04,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1150600.0, ans=0.125 2023-11-20 17:19:05,198 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172600 2023-11-20 17:19:06,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1150666.6666666667, ans=0.125 2023-11-20 17:19:15,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1150666.6666666667, ans=0.125 2023-11-20 17:19:18,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1150666.6666666667, ans=0.2 2023-11-20 17:19:23,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=23.73 vs. limit=22.5 2023-11-20 17:19:31,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1150733.3333333333, ans=0.125 2023-11-20 17:19:39,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1150800.0, ans=0.125 2023-11-20 17:19:47,501 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4300, loss[loss=0.07449, simple_loss=0.09194, pruned_loss=0.01604, audio_tagging_loss=0.01248, over 15689.00 frames. ], tot_loss[loss=0.07959, simple_loss=0.1015, pruned_loss=0.01919, audio_tagging_loss=0.009674, over 3054771.08 frames. ], batch size: 58, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:20:04,275 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.964e+01 7.979e+01 8.573e+01 9.478e+01 1.236e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-20 17:20:10,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172650 2023-11-20 17:20:14,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.63 vs. limit=15.0 2023-11-20 17:20:18,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1151000.0, ans=0.0 2023-11-20 17:20:40,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1151133.3333333333, ans=0.2 2023-11-20 17:20:52,661 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4350, loss[loss=0.08301, simple_loss=0.1057, pruned_loss=0.02137, audio_tagging_loss=0.008784, over 15732.00 frames. ], tot_loss[loss=0.07961, simple_loss=0.1014, pruned_loss=0.01923, audio_tagging_loss=0.009658, over 3054536.34 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:20:54,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.87 vs. limit=15.0 2023-11-20 17:20:59,649 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.53 vs. limit=15.0 2023-11-20 17:21:15,693 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172700 2023-11-20 17:21:27,453 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.51 vs. limit=22.5 2023-11-20 17:21:33,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1151400.0, ans=0.125 2023-11-20 17:21:40,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1151400.0, ans=0.1 2023-11-20 17:21:57,236 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4400, loss[loss=0.09303, simple_loss=0.1164, pruned_loss=0.02772, audio_tagging_loss=0.007126, over 15760.00 frames. ], tot_loss[loss=0.0794, simple_loss=0.1012, pruned_loss=0.0192, audio_tagging_loss=0.009601, over 3050948.00 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 32.0 2023-11-20 17:22:16,306 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.328e+01 8.168e+01 8.779e+01 9.674e+01 1.363e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-20 17:22:20,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1151600.0, ans=0.05 2023-11-20 17:22:21,334 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172750 2023-11-20 17:23:02,740 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4450, loss[loss=0.06999, simple_loss=0.08907, pruned_loss=0.01793, audio_tagging_loss=0.007521, over 15665.00 frames. ], tot_loss[loss=0.07878, simple_loss=0.1001, pruned_loss=0.01901, audio_tagging_loss=0.009709, over 3052498.49 frames. ], batch size: 59, lr: 4.62e-03, grad_scale: 16.0 2023-11-20 17:23:03,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1151866.6666666667, ans=0.125 2023-11-20 17:23:09,213 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0 2023-11-20 17:23:16,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1151933.3333333333, ans=0.125 2023-11-20 17:23:26,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172800 2023-11-20 17:23:59,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1152133.3333333333, ans=0.015 2023-11-20 17:24:05,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.45 vs. limit=6.0 2023-11-20 17:24:08,395 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4500, loss[loss=0.07444, simple_loss=0.09414, pruned_loss=0.01666, audio_tagging_loss=0.01071, over 15559.00 frames. ], tot_loss[loss=0.07906, simple_loss=0.1006, pruned_loss=0.01905, audio_tagging_loss=0.009697, over 3052912.11 frames. ], batch size: 59, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:24:15,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1152200.0, ans=0.0 2023-11-20 17:24:20,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1152266.6666666667, ans=0.125 2023-11-20 17:24:21,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1152266.6666666667, ans=0.125 2023-11-20 17:24:26,181 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.191e+01 8.215e+01 8.732e+01 9.444e+01 1.886e+02, threshold=1.746e+02, percent-clipped=1.0 2023-11-20 17:24:31,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172850 2023-11-20 17:24:53,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1152400.0, ans=0.1 2023-11-20 17:25:00,507 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.22 vs. limit=12.0 2023-11-20 17:25:13,122 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4550, loss[loss=0.09569, simple_loss=0.1173, pruned_loss=0.0235, audio_tagging_loss=0.01352, over 15274.00 frames. ], tot_loss[loss=0.07906, simple_loss=0.1005, pruned_loss=0.01909, audio_tagging_loss=0.009702, over 3052466.07 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:25:18,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1152533.3333333333, ans=0.0 2023-11-20 17:25:19,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1152533.3333333333, ans=0.0 2023-11-20 17:25:22,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1152533.3333333333, ans=0.0 2023-11-20 17:25:28,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.74 vs. limit=10.0 2023-11-20 17:25:36,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1152600.0, ans=0.015 2023-11-20 17:25:37,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172900 2023-11-20 17:25:39,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1152666.6666666667, ans=0.125 2023-11-20 17:25:40,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1152666.6666666667, ans=0.125 2023-11-20 17:26:02,828 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 17:26:18,996 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4600, loss[loss=0.06376, simple_loss=0.0856, pruned_loss=0.01147, audio_tagging_loss=0.009496, over 14814.00 frames. ], tot_loss[loss=0.07936, simple_loss=0.1009, pruned_loss=0.01917, audio_tagging_loss=0.009761, over 3045765.40 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:26:20,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1152866.6666666667, ans=0.025 2023-11-20 17:26:34,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1152933.3333333333, ans=0.0 2023-11-20 17:26:36,943 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 8.346e+01 9.151e+01 1.010e+02 1.307e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-20 17:26:42,011 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 172950 2023-11-20 17:27:24,143 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4650, loss[loss=0.09705, simple_loss=0.1268, pruned_loss=0.02476, audio_tagging_loss=0.008878, over 16791.00 frames. ], tot_loss[loss=0.07867, simple_loss=0.09974, pruned_loss=0.01887, audio_tagging_loss=0.009926, over 3049506.18 frames. ], batch size: 61, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:27:35,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1153266.6666666667, ans=0.125 2023-11-20 17:27:43,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1153266.6666666667, ans=0.125 2023-11-20 17:27:44,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1153266.6666666667, ans=0.0 2023-11-20 17:27:46,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173000 2023-11-20 17:28:29,539 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4700, loss[loss=0.08423, simple_loss=0.102, pruned_loss=0.02314, audio_tagging_loss=0.01009, over 14980.00 frames. ], tot_loss[loss=0.07859, simple_loss=0.09967, pruned_loss=0.01879, audio_tagging_loss=0.009961, over 3050477.58 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:28:39,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1153533.3333333333, ans=0.0 2023-11-20 17:28:49,106 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 7.947e+01 8.690e+01 9.286e+01 1.416e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-20 17:28:54,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173050 2023-11-20 17:28:56,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1153666.6666666667, ans=0.09899494936611666 2023-11-20 17:29:16,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1153733.3333333333, ans=0.125 2023-11-20 17:29:31,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0 2023-11-20 17:29:34,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1153800.0, ans=0.04949747468305833 2023-11-20 17:29:36,583 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4750, loss[loss=0.08726, simple_loss=0.1157, pruned_loss=0.01876, audio_tagging_loss=0.01063, over 14883.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.09951, pruned_loss=0.01883, audio_tagging_loss=0.01003, over 3051626.57 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:29:40,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1153866.6666666667, ans=0.0 2023-11-20 17:29:59,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173100 2023-11-20 17:30:16,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.12 vs. limit=15.0 2023-11-20 17:30:17,844 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.48 vs. limit=10.0 2023-11-20 17:30:35,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1154133.3333333333, ans=0.125 2023-11-20 17:30:42,288 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4800, loss[loss=0.05641, simple_loss=0.05986, pruned_loss=0.01256, audio_tagging_loss=0.01393, over 16091.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09857, pruned_loss=0.01878, audio_tagging_loss=0.01015, over 3055327.30 frames. ], batch size: 65, lr: 4.61e-03, grad_scale: 32.0 2023-11-20 17:30:48,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.14 vs. limit=15.0 2023-11-20 17:30:49,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1154200.0, ans=0.125 2023-11-20 17:30:59,595 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.494e+01 7.777e+01 8.468e+01 9.403e+01 1.279e+02, threshold=1.694e+02, percent-clipped=0.0 2023-11-20 17:31:04,470 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173150 2023-11-20 17:31:47,281 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4850, loss[loss=0.1024, simple_loss=0.1332, pruned_loss=0.02696, audio_tagging_loss=0.008843, over 15608.00 frames. ], tot_loss[loss=0.07872, simple_loss=0.0994, pruned_loss=0.01885, audio_tagging_loss=0.01017, over 3051055.36 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 32.0 2023-11-20 17:31:58,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1154600.0, ans=0.0 2023-11-20 17:31:58,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1154600.0, ans=0.125 2023-11-20 17:32:08,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1154600.0, ans=0.0 2023-11-20 17:32:11,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173200 2023-11-20 17:32:19,301 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=12.0 2023-11-20 17:32:26,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=4.24 vs. limit=15.0 2023-11-20 17:32:32,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1154733.3333333333, ans=0.1 2023-11-20 17:32:51,428 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4900, loss[loss=0.08181, simple_loss=0.1002, pruned_loss=0.02138, audio_tagging_loss=0.01034, over 15350.00 frames. ], tot_loss[loss=0.07775, simple_loss=0.09834, pruned_loss=0.01845, audio_tagging_loss=0.01013, over 3050865.67 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 32.0 2023-11-20 17:33:02,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1154866.6666666667, ans=0.0 2023-11-20 17:33:09,822 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.588e+01 8.105e+01 8.801e+01 9.548e+01 1.347e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-20 17:33:14,831 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173250 2023-11-20 17:33:16,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.78 vs. limit=22.5 2023-11-20 17:33:16,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.87 vs. limit=6.0 2023-11-20 17:33:18,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0 2023-11-20 17:33:25,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.18 vs. limit=15.0 2023-11-20 17:33:35,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1155066.6666666667, ans=0.2 2023-11-20 17:33:37,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.33 vs. limit=15.0 2023-11-20 17:33:41,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.96 vs. limit=15.0 2023-11-20 17:33:55,038 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 4950, loss[loss=0.07559, simple_loss=0.09202, pruned_loss=0.02216, audio_tagging_loss=0.007421, over 14676.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.09886, pruned_loss=0.01864, audio_tagging_loss=0.009881, over 3048637.27 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 32.0 2023-11-20 17:33:55,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1155200.0, ans=0.0 2023-11-20 17:33:56,935 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.75 vs. limit=15.0 2023-11-20 17:33:59,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1155200.0, ans=0.0 2023-11-20 17:34:14,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.48 vs. limit=22.5 2023-11-20 17:34:16,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173300 2023-11-20 17:34:17,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1155266.6666666667, ans=0.125 2023-11-20 17:34:18,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1155333.3333333333, ans=0.125 2023-11-20 17:34:50,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1155466.6666666667, ans=0.0 2023-11-20 17:34:56,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1155533.3333333333, ans=0.1 2023-11-20 17:34:57,669 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5000, loss[loss=0.09983, simple_loss=0.1271, pruned_loss=0.02623, audio_tagging_loss=0.01007, over 15054.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.09991, pruned_loss=0.01893, audio_tagging_loss=0.009734, over 3046015.90 frames. ], batch size: 54, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:34:58,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1155533.3333333333, ans=0.125 2023-11-20 17:34:58,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1155533.3333333333, ans=0.0 2023-11-20 17:35:02,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1155533.3333333333, ans=0.0 2023-11-20 17:35:16,433 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.798e+01 8.227e+01 8.824e+01 9.880e+01 1.350e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-20 17:35:20,291 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173350 2023-11-20 17:35:37,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1155733.3333333333, ans=0.0 2023-11-20 17:35:43,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1155733.3333333333, ans=0.125 2023-11-20 17:35:43,826 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.52 vs. limit=22.5 2023-11-20 17:35:46,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.89 vs. limit=12.0 2023-11-20 17:35:59,935 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5050, loss[loss=0.07087, simple_loss=0.09384, pruned_loss=0.01484, audio_tagging_loss=0.00911, over 14437.00 frames. ], tot_loss[loss=0.07866, simple_loss=0.1004, pruned_loss=0.01882, audio_tagging_loss=0.009645, over 3042997.89 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:36:19,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.73 vs. limit=22.5 2023-11-20 17:36:23,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173400 2023-11-20 17:36:29,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1156000.0, ans=0.125 2023-11-20 17:36:45,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.22 vs. limit=15.0 2023-11-20 17:37:04,831 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5100, loss[loss=0.07251, simple_loss=0.1035, pruned_loss=0.01402, audio_tagging_loss=0.00676, over 15133.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09958, pruned_loss=0.01865, audio_tagging_loss=0.009672, over 3033432.86 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:37:19,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1156266.6666666667, ans=0.125 2023-11-20 17:37:23,037 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.188e+01 8.649e+01 9.367e+01 2.807e+02, threshold=1.730e+02, percent-clipped=1.0 2023-11-20 17:37:23,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1156266.6666666667, ans=0.2 2023-11-20 17:37:26,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173450 2023-11-20 17:37:45,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1156400.0, ans=0.125 2023-11-20 17:37:48,106 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.41 vs. limit=15.0 2023-11-20 17:37:58,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1156466.6666666667, ans=0.2 2023-11-20 17:37:59,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1156466.6666666667, ans=0.0 2023-11-20 17:38:01,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1156466.6666666667, ans=0.2 2023-11-20 17:38:07,894 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5150, loss[loss=0.09581, simple_loss=0.1136, pruned_loss=0.02609, audio_tagging_loss=0.01292, over 14734.00 frames. ], tot_loss[loss=0.07829, simple_loss=0.09998, pruned_loss=0.0187, audio_tagging_loss=0.009608, over 3034546.56 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 16.0 2023-11-20 17:38:23,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1156600.0, ans=0.0 2023-11-20 17:38:30,954 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173500 2023-11-20 17:38:33,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.35 vs. limit=15.0 2023-11-20 17:38:41,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1156666.6666666667, ans=0.125 2023-11-20 17:38:49,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1156733.3333333333, ans=0.04949747468305833 2023-11-20 17:38:53,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1156733.3333333333, ans=0.125 2023-11-20 17:38:55,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.67 vs. limit=22.5 2023-11-20 17:38:56,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1156733.3333333333, ans=0.2 2023-11-20 17:39:10,977 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5200, loss[loss=0.06347, simple_loss=0.08006, pruned_loss=0.01278, audio_tagging_loss=0.01067, over 14547.00 frames. ], tot_loss[loss=0.07842, simple_loss=0.1, pruned_loss=0.01886, audio_tagging_loss=0.009544, over 3028522.38 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 32.0 2023-11-20 17:39:13,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1156866.6666666667, ans=0.0 2023-11-20 17:39:31,482 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.205e+01 8.715e+01 9.461e+01 1.258e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-20 17:39:35,260 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173550 2023-11-20 17:39:35,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1156933.3333333333, ans=0.0 2023-11-20 17:39:36,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1157000.0, ans=0.5 2023-11-20 17:39:56,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1157066.6666666667, ans=0.125 2023-11-20 17:40:02,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1157133.3333333333, ans=0.2 2023-11-20 17:40:08,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1157133.3333333333, ans=0.125 2023-11-20 17:40:13,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1157133.3333333333, ans=0.0 2023-11-20 17:40:15,224 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5250, loss[loss=0.07092, simple_loss=0.08793, pruned_loss=0.0152, audio_tagging_loss=0.01176, over 14374.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.1003, pruned_loss=0.01896, audio_tagging_loss=0.009524, over 3032390.69 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 32.0 2023-11-20 17:40:21,628 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 17:40:21,886 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.30 vs. limit=15.0 2023-11-20 17:40:22,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1157200.0, ans=0.125 2023-11-20 17:40:33,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1157266.6666666667, ans=0.125 2023-11-20 17:40:35,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.66 vs. limit=12.0 2023-11-20 17:40:37,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1157266.6666666667, ans=0.1 2023-11-20 17:40:38,351 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173600 2023-11-20 17:41:03,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1157400.0, ans=0.125 2023-11-20 17:41:16,903 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.30 vs. limit=15.0 2023-11-20 17:41:19,888 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5300, loss[loss=0.08801, simple_loss=0.1222, pruned_loss=0.01998, audio_tagging_loss=0.006925, over 15666.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09978, pruned_loss=0.01873, audio_tagging_loss=0.009596, over 3035721.64 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 32.0 2023-11-20 17:41:28,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1157533.3333333333, ans=0.125 2023-11-20 17:41:37,919 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.138e+01 8.869e+01 9.897e+01 2.566e+02, threshold=1.774e+02, percent-clipped=1.0 2023-11-20 17:41:41,711 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173650 2023-11-20 17:41:46,621 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 17:42:22,924 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5350, loss[loss=0.07227, simple_loss=0.09153, pruned_loss=0.01587, audio_tagging_loss=0.01063, over 14335.00 frames. ], tot_loss[loss=0.07767, simple_loss=0.09891, pruned_loss=0.01851, audio_tagging_loss=0.00971, over 3037016.50 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 32.0 2023-11-20 17:42:35,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1157933.3333333333, ans=0.125 2023-11-20 17:42:36,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1157933.3333333333, ans=0.125 2023-11-20 17:42:46,543 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173700 2023-11-20 17:43:09,394 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.58 vs. limit=15.0 2023-11-20 17:43:26,670 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5400, loss[loss=0.08062, simple_loss=0.1127, pruned_loss=0.01719, audio_tagging_loss=0.007076, over 15745.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.09806, pruned_loss=0.01836, audio_tagging_loss=0.009795, over 3032374.20 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 32.0 2023-11-20 17:43:36,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1158200.0, ans=0.0 2023-11-20 17:43:46,149 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 7.982e+01 8.687e+01 9.392e+01 1.840e+02, threshold=1.737e+02, percent-clipped=1.0 2023-11-20 17:43:49,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173750 2023-11-20 17:44:08,888 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0 2023-11-20 17:44:24,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1158466.6666666667, ans=0.125 2023-11-20 17:44:30,513 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5450, loss[loss=0.078, simple_loss=0.09787, pruned_loss=0.01984, audio_tagging_loss=0.009227, over 15068.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.09806, pruned_loss=0.01843, audio_tagging_loss=0.009901, over 3025462.76 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 32.0 2023-11-20 17:44:35,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1158533.3333333333, ans=0.125 2023-11-20 17:44:37,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1158533.3333333333, ans=0.125 2023-11-20 17:44:41,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1158533.3333333333, ans=0.125 2023-11-20 17:44:53,199 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173800 2023-11-20 17:45:16,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1158733.3333333333, ans=0.1 2023-11-20 17:45:23,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1158800.0, ans=0.2 2023-11-20 17:45:24,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1158800.0, ans=0.0 2023-11-20 17:45:30,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1158800.0, ans=0.1 2023-11-20 17:45:33,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1158866.6666666667, ans=0.125 2023-11-20 17:45:34,258 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5500, loss[loss=0.07505, simple_loss=0.09909, pruned_loss=0.01791, audio_tagging_loss=0.00759, over 14946.00 frames. ], tot_loss[loss=0.07784, simple_loss=0.09837, pruned_loss=0.01876, audio_tagging_loss=0.009903, over 3033766.63 frames. ], batch size: 58, lr: 4.60e-03, grad_scale: 32.0 2023-11-20 17:45:54,019 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.155e+01 8.582e+01 9.481e+01 1.342e+02, threshold=1.716e+02, percent-clipped=0.0 2023-11-20 17:45:57,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173850 2023-11-20 17:45:59,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1159000.0, ans=0.0 2023-11-20 17:46:20,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1159066.6666666667, ans=0.125 2023-11-20 17:46:36,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1159200.0, ans=0.0 2023-11-20 17:46:37,039 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5550, loss[loss=0.05266, simple_loss=0.062, pruned_loss=0.008751, audio_tagging_loss=0.01291, over 15689.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.09821, pruned_loss=0.01869, audio_tagging_loss=0.01001, over 3038286.72 frames. ], batch size: 60, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:46:37,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1159200.0, ans=0.125 2023-11-20 17:46:52,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1159266.6666666667, ans=0.04949747468305833 2023-11-20 17:46:53,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1159266.6666666667, ans=0.1 2023-11-20 17:46:59,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173900 2023-11-20 17:47:17,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.89 vs. limit=8.0 2023-11-20 17:47:20,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1159400.0, ans=0.125 2023-11-20 17:47:29,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1159466.6666666667, ans=0.05 2023-11-20 17:47:34,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1159466.6666666667, ans=0.1 2023-11-20 17:47:39,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1159533.3333333333, ans=0.125 2023-11-20 17:47:40,118 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5600, loss[loss=0.07536, simple_loss=0.101, pruned_loss=0.0155, audio_tagging_loss=0.009386, over 14882.00 frames. ], tot_loss[loss=0.07853, simple_loss=0.09908, pruned_loss=0.01889, audio_tagging_loss=0.01011, over 3042334.49 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 32.0 2023-11-20 17:47:53,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1159600.0, ans=0.1 2023-11-20 17:48:00,192 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 7.988e+01 8.538e+01 9.609e+01 1.592e+02, threshold=1.708e+02, percent-clipped=0.0 2023-11-20 17:48:00,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1159600.0, ans=0.2 2023-11-20 17:48:02,722 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 173950 2023-11-20 17:48:26,225 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 17:48:27,088 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.18 vs. limit=15.0 2023-11-20 17:48:27,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1159733.3333333333, ans=0.125 2023-11-20 17:48:30,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1159800.0, ans=0.0 2023-11-20 17:48:31,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1159800.0, ans=0.1 2023-11-20 17:48:35,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1159800.0, ans=0.0 2023-11-20 17:48:44,022 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5650, loss[loss=0.09786, simple_loss=0.1266, pruned_loss=0.02345, audio_tagging_loss=0.01109, over 15229.00 frames. ], tot_loss[loss=0.07894, simple_loss=0.09978, pruned_loss=0.01892, audio_tagging_loss=0.01012, over 3056145.92 frames. ], batch size: 58, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:49:01,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1159933.3333333333, ans=0.125 2023-11-20 17:49:02,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.40 vs. limit=15.0 2023-11-20 17:49:07,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174000 2023-11-20 17:49:48,558 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5700, loss[loss=0.07522, simple_loss=0.0875, pruned_loss=0.01769, audio_tagging_loss=0.01379, over 15260.00 frames. ], tot_loss[loss=0.07883, simple_loss=0.09997, pruned_loss=0.01879, audio_tagging_loss=0.01005, over 3056435.38 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:49:54,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1160200.0, ans=0.0 2023-11-20 17:50:09,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1160266.6666666667, ans=0.125 2023-11-20 17:50:10,026 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.368e+01 9.255e+01 1.006e+02 1.511e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-20 17:50:11,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174050 2023-11-20 17:50:15,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1160333.3333333333, ans=0.1 2023-11-20 17:50:24,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1160333.3333333333, ans=0.2 2023-11-20 17:50:25,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1160400.0, ans=0.125 2023-11-20 17:50:29,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1160400.0, ans=0.125 2023-11-20 17:50:33,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.41 vs. limit=15.0 2023-11-20 17:50:52,581 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5750, loss[loss=0.07242, simple_loss=0.09533, pruned_loss=0.01585, audio_tagging_loss=0.008908, over 14203.00 frames. ], tot_loss[loss=0.07848, simple_loss=0.09964, pruned_loss=0.01878, audio_tagging_loss=0.009882, over 3050024.38 frames. ], batch size: 54, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:51:15,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174100 2023-11-20 17:51:20,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1160666.6666666667, ans=0.0 2023-11-20 17:51:37,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1160733.3333333333, ans=0.0 2023-11-20 17:51:55,630 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5800, loss[loss=0.07717, simple_loss=0.1017, pruned_loss=0.01507, audio_tagging_loss=0.01125, over 15127.00 frames. ], tot_loss[loss=0.0783, simple_loss=0.09939, pruned_loss=0.01879, audio_tagging_loss=0.009819, over 3048620.24 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:51:58,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1160866.6666666667, ans=0.125 2023-11-20 17:51:58,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1160866.6666666667, ans=0.125 2023-11-20 17:51:59,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1160866.6666666667, ans=0.0 2023-11-20 17:52:03,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1160866.6666666667, ans=0.025 2023-11-20 17:52:06,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1160866.6666666667, ans=0.125 2023-11-20 17:52:07,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1160933.3333333333, ans=0.125 2023-11-20 17:52:17,729 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.484e+01 8.088e+01 8.646e+01 9.359e+01 1.315e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-20 17:52:18,117 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 17:52:19,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174150 2023-11-20 17:52:59,015 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5850, loss[loss=0.08586, simple_loss=0.1082, pruned_loss=0.02369, audio_tagging_loss=0.008087, over 15747.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.09831, pruned_loss=0.01872, audio_tagging_loss=0.009683, over 3043176.78 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:53:22,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174200 2023-11-20 17:53:25,728 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.97 vs. limit=15.0 2023-11-20 17:53:30,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1161333.3333333333, ans=0.2 2023-11-20 17:53:35,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1161333.3333333333, ans=0.125 2023-11-20 17:53:39,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1161400.0, ans=0.2 2023-11-20 17:53:41,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.68 vs. limit=10.0 2023-11-20 17:53:57,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1161466.6666666667, ans=0.125 2023-11-20 17:54:03,768 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5900, loss[loss=0.0664, simple_loss=0.08624, pruned_loss=0.01242, audio_tagging_loss=0.01087, over 13809.00 frames. ], tot_loss[loss=0.07806, simple_loss=0.0993, pruned_loss=0.01872, audio_tagging_loss=0.009688, over 3047219.67 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:54:07,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1161533.3333333333, ans=0.0 2023-11-20 17:54:24,230 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.636e+01 8.417e+01 8.980e+01 9.978e+01 1.286e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-20 17:54:24,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1161600.0, ans=0.0 2023-11-20 17:54:25,533 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174250 2023-11-20 17:54:30,694 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.68 vs. limit=15.0 2023-11-20 17:54:37,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1161666.6666666667, ans=0.2 2023-11-20 17:54:37,866 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.61 vs. limit=15.0 2023-11-20 17:54:46,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1161733.3333333333, ans=0.0 2023-11-20 17:55:06,722 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 5950, loss[loss=0.06525, simple_loss=0.07002, pruned_loss=0.01705, audio_tagging_loss=0.01319, over 15062.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09941, pruned_loss=0.01873, audio_tagging_loss=0.009667, over 3045200.17 frames. ], batch size: 58, lr: 4.60e-03, grad_scale: 16.0 2023-11-20 17:55:22,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1161933.3333333333, ans=0.2 2023-11-20 17:55:30,878 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174300 2023-11-20 17:55:51,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1162066.6666666667, ans=0.125 2023-11-20 17:55:53,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1162066.6666666667, ans=0.1 2023-11-20 17:55:55,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1162066.6666666667, ans=0.125 2023-11-20 17:55:59,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.82 vs. limit=10.0 2023-11-20 17:55:59,934 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 17:56:10,519 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6000, loss[loss=0.07682, simple_loss=0.09611, pruned_loss=0.01545, audio_tagging_loss=0.01332, over 15269.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09838, pruned_loss=0.01839, audio_tagging_loss=0.009783, over 3041803.51 frames. ], batch size: 60, lr: 4.59e-03, grad_scale: 32.0 2023-11-20 17:56:10,520 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 17:56:51,551 INFO [train_asr.py:1253] (3/4) Epoch 15, validation: loss=0.06114, simple_loss=0.05327, pruned_loss=0.005599, audio_tagging_loss=0.02891, over 4681554.00 frames. 2023-11-20 17:56:51,552 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 17:57:03,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1162266.6666666667, ans=0.125 2023-11-20 17:57:05,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1162266.6666666667, ans=0.1 2023-11-20 17:57:07,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.80 vs. limit=22.5 2023-11-20 17:57:12,188 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.279e+01 8.029e+01 8.706e+01 9.735e+01 1.152e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-20 17:57:13,547 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174350 2023-11-20 17:57:15,230 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.68 vs. limit=10.0 2023-11-20 17:57:21,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1162333.3333333333, ans=0.07 2023-11-20 17:57:38,279 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 17:57:53,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1162466.6666666667, ans=0.0 2023-11-20 17:57:55,645 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6050, loss[loss=0.07334, simple_loss=0.09045, pruned_loss=0.0181, audio_tagging_loss=0.01001, over 15948.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09795, pruned_loss=0.01824, audio_tagging_loss=0.009823, over 3043031.23 frames. ], batch size: 60, lr: 4.59e-03, grad_scale: 32.0 2023-11-20 17:57:59,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1162533.3333333333, ans=0.2 2023-11-20 17:58:19,554 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174400 2023-11-20 17:58:19,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1162600.0, ans=0.125 2023-11-20 17:58:29,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1162666.6666666667, ans=0.125 2023-11-20 17:58:40,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1162733.3333333333, ans=0.125 2023-11-20 17:58:41,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1162733.3333333333, ans=0.0 2023-11-20 17:58:59,442 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6100, loss[loss=0.06507, simple_loss=0.07961, pruned_loss=0.0147, audio_tagging_loss=0.01057, over 14949.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.09769, pruned_loss=0.01826, audio_tagging_loss=0.009877, over 3049636.71 frames. ], batch size: 58, lr: 4.59e-03, grad_scale: 32.0 2023-11-20 17:59:05,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1162866.6666666667, ans=0.125 2023-11-20 17:59:08,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1162866.6666666667, ans=0.1 2023-11-20 17:59:12,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1162933.3333333333, ans=0.125 2023-11-20 17:59:22,141 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 7.952e+01 8.424e+01 9.114e+01 1.496e+02, threshold=1.685e+02, percent-clipped=0.0 2023-11-20 17:59:22,722 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.21 vs. limit=15.0 2023-11-20 17:59:23,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174450 2023-11-20 18:00:04,770 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6150, loss[loss=0.06849, simple_loss=0.08721, pruned_loss=0.01757, audio_tagging_loss=0.007314, over 15115.00 frames. ], tot_loss[loss=0.07722, simple_loss=0.09802, pruned_loss=0.01837, audio_tagging_loss=0.009835, over 3053529.40 frames. ], batch size: 57, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:00:27,087 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174500 2023-11-20 18:00:39,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1163333.3333333333, ans=0.2 2023-11-20 18:00:52,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1163400.0, ans=0.0 2023-11-20 18:00:58,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1163466.6666666667, ans=0.0 2023-11-20 18:01:09,263 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6200, loss[loss=0.0726, simple_loss=0.09721, pruned_loss=0.01478, audio_tagging_loss=0.009211, over 14342.00 frames. ], tot_loss[loss=0.07734, simple_loss=0.09828, pruned_loss=0.01839, audio_tagging_loss=0.009805, over 3047014.40 frames. ], batch size: 53, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:01:11,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1163533.3333333333, ans=0.1 2023-11-20 18:01:13,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.61 vs. limit=15.0 2023-11-20 18:01:14,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1163533.3333333333, ans=0.125 2023-11-20 18:01:24,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1163600.0, ans=0.0 2023-11-20 18:01:33,401 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 8.002e+01 8.659e+01 9.317e+01 2.710e+02, threshold=1.732e+02, percent-clipped=1.0 2023-11-20 18:01:33,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174550 2023-11-20 18:01:37,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1163666.6666666667, ans=0.1 2023-11-20 18:02:08,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1163800.0, ans=0.125 2023-11-20 18:02:10,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.38 vs. limit=15.0 2023-11-20 18:02:11,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1163800.0, ans=0.035 2023-11-20 18:02:13,303 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6250, loss[loss=0.09406, simple_loss=0.1304, pruned_loss=0.02221, audio_tagging_loss=0.00664, over 14520.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09865, pruned_loss=0.0185, audio_tagging_loss=0.009892, over 3049832.51 frames. ], batch size: 54, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:02:37,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174600 2023-11-20 18:02:42,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1164000.0, ans=0.125 2023-11-20 18:02:44,637 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 18:02:55,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1164066.6666666667, ans=0.1 2023-11-20 18:02:56,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1164066.6666666667, ans=0.125 2023-11-20 18:02:57,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1164066.6666666667, ans=0.125 2023-11-20 18:03:02,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1164066.6666666667, ans=0.125 2023-11-20 18:03:07,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1164133.3333333333, ans=0.125 2023-11-20 18:03:12,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1164133.3333333333, ans=0.2 2023-11-20 18:03:15,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.25 vs. limit=15.0 2023-11-20 18:03:17,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1164200.0, ans=0.0 2023-11-20 18:03:18,945 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6300, loss[loss=0.07231, simple_loss=0.08924, pruned_loss=0.01503, audio_tagging_loss=0.01266, over 14480.00 frames. ], tot_loss[loss=0.0784, simple_loss=0.09969, pruned_loss=0.01869, audio_tagging_loss=0.009858, over 3055673.90 frames. ], batch size: 58, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:03:22,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=22.5 2023-11-20 18:03:33,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1164266.6666666667, ans=0.0 2023-11-20 18:03:41,341 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.283e+01 8.184e+01 8.938e+01 9.921e+01 1.399e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-20 18:03:41,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174650 2023-11-20 18:03:45,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1164333.3333333333, ans=0.125 2023-11-20 18:03:54,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1164333.3333333333, ans=0.125 2023-11-20 18:03:58,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1164400.0, ans=0.2 2023-11-20 18:04:07,236 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.93 vs. limit=6.0 2023-11-20 18:04:13,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1164466.6666666667, ans=0.5 2023-11-20 18:04:15,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1164466.6666666667, ans=0.125 2023-11-20 18:04:20,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1164466.6666666667, ans=0.0 2023-11-20 18:04:23,033 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6350, loss[loss=0.06772, simple_loss=0.07752, pruned_loss=0.01608, audio_tagging_loss=0.01288, over 14688.00 frames. ], tot_loss[loss=0.07796, simple_loss=0.0989, pruned_loss=0.01852, audio_tagging_loss=0.009986, over 3054472.25 frames. ], batch size: 56, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:04:25,066 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.75 vs. limit=22.5 2023-11-20 18:04:25,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1164533.3333333333, ans=0.035 2023-11-20 18:04:46,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174700 2023-11-20 18:04:49,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1164666.6666666667, ans=0.125 2023-11-20 18:04:52,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1164666.6666666667, ans=0.1 2023-11-20 18:04:53,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1164666.6666666667, ans=0.1 2023-11-20 18:05:04,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1164733.3333333333, ans=0.1 2023-11-20 18:05:09,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1164733.3333333333, ans=0.0 2023-11-20 18:05:26,449 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6400, loss[loss=0.1105, simple_loss=0.133, pruned_loss=0.03476, audio_tagging_loss=0.009196, over 13838.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09874, pruned_loss=0.01853, audio_tagging_loss=0.0102, over 3049067.08 frames. ], batch size: 53, lr: 4.59e-03, grad_scale: 32.0 2023-11-20 18:05:36,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1164866.6666666667, ans=0.2 2023-11-20 18:05:48,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1164933.3333333333, ans=0.1 2023-11-20 18:05:50,737 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.152e+01 8.050e+01 8.682e+01 9.459e+01 1.192e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-20 18:05:50,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174750 2023-11-20 18:06:17,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1165133.3333333333, ans=0.125 2023-11-20 18:06:17,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.66 vs. limit=15.0 2023-11-20 18:06:24,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.73 vs. limit=6.0 2023-11-20 18:06:31,351 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6450, loss[loss=0.05655, simple_loss=0.05921, pruned_loss=0.009659, audio_tagging_loss=0.01728, over 14673.00 frames. ], tot_loss[loss=0.07807, simple_loss=0.09861, pruned_loss=0.01851, audio_tagging_loss=0.01026, over 3046404.37 frames. ], batch size: 58, lr: 4.59e-03, grad_scale: 32.0 2023-11-20 18:06:40,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1165200.0, ans=0.125 2023-11-20 18:06:53,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.68 vs. limit=15.0 2023-11-20 18:06:54,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174800 2023-11-20 18:07:33,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1165466.6666666667, ans=0.125 2023-11-20 18:07:36,636 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6500, loss[loss=0.06506, simple_loss=0.07724, pruned_loss=0.01452, audio_tagging_loss=0.01191, over 16298.00 frames. ], tot_loss[loss=0.07859, simple_loss=0.09956, pruned_loss=0.01865, audio_tagging_loss=0.01016, over 3049888.45 frames. ], batch size: 63, lr: 4.59e-03, grad_scale: 32.0 2023-11-20 18:07:48,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1165600.0, ans=0.1 2023-11-20 18:07:58,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174850 2023-11-20 18:08:00,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 7.879e+01 8.698e+01 9.570e+01 1.330e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-20 18:08:09,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1165666.6666666667, ans=0.125 2023-11-20 18:08:40,142 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6550, loss[loss=0.07585, simple_loss=0.09678, pruned_loss=0.01586, audio_tagging_loss=0.0116, over 15128.00 frames. ], tot_loss[loss=0.07904, simple_loss=0.1005, pruned_loss=0.01879, audio_tagging_loss=0.01001, over 3052309.52 frames. ], batch size: 58, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:08:41,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1165866.6666666667, ans=0.125 2023-11-20 18:09:03,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174900 2023-11-20 18:09:16,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.69 vs. limit=15.0 2023-11-20 18:09:23,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1166066.6666666667, ans=0.125 2023-11-20 18:09:32,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1166133.3333333333, ans=0.125 2023-11-20 18:09:44,716 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6600, loss[loss=0.05941, simple_loss=0.07708, pruned_loss=0.01233, audio_tagging_loss=0.00854, over 14474.00 frames. ], tot_loss[loss=0.07833, simple_loss=0.09954, pruned_loss=0.01874, audio_tagging_loss=0.00982, over 3047966.24 frames. ], batch size: 55, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:09:56,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1166266.6666666667, ans=0.125 2023-11-20 18:10:08,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 174950 2023-11-20 18:10:08,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1166266.6666666667, ans=0.2 2023-11-20 18:10:09,439 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.156e+01 8.016e+01 8.893e+01 9.402e+01 1.270e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-20 18:10:09,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1166333.3333333333, ans=0.125 2023-11-20 18:10:09,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=1166333.3333333333, ans=0.2 2023-11-20 18:10:14,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1166333.3333333333, ans=0.0 2023-11-20 18:10:48,685 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6650, loss[loss=0.06104, simple_loss=0.0653, pruned_loss=0.01565, audio_tagging_loss=0.01275, over 14757.00 frames. ], tot_loss[loss=0.07833, simple_loss=0.09953, pruned_loss=0.01881, audio_tagging_loss=0.009748, over 3043824.63 frames. ], batch size: 57, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:10:51,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1166533.3333333333, ans=0.125 2023-11-20 18:10:52,524 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.45 vs. limit=15.0 2023-11-20 18:10:58,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1166533.3333333333, ans=0.0 2023-11-20 18:11:10,641 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.96 vs. limit=15.0 2023-11-20 18:11:11,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175000 2023-11-20 18:11:23,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=1166666.6666666667, ans=0.2 2023-11-20 18:11:39,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.98 vs. limit=22.5 2023-11-20 18:11:44,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1166800.0, ans=0.125 2023-11-20 18:11:53,078 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6700, loss[loss=0.07163, simple_loss=0.09353, pruned_loss=0.01612, audio_tagging_loss=0.008747, over 16543.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09852, pruned_loss=0.01856, audio_tagging_loss=0.009766, over 3034040.07 frames. ], batch size: 61, lr: 4.59e-03, grad_scale: 16.0 2023-11-20 18:12:09,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1166933.3333333333, ans=0.125 2023-11-20 18:12:10,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1166933.3333333333, ans=0.125 2023-11-20 18:12:15,552 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175050 2023-11-20 18:12:17,134 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.109e+01 8.688e+01 9.322e+01 1.481e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-20 18:12:36,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1167066.6666666667, ans=0.0 2023-11-20 18:12:57,099 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6750, loss[loss=0.09493, simple_loss=0.1168, pruned_loss=0.02978, audio_tagging_loss=0.006725, over 15568.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09878, pruned_loss=0.01866, audio_tagging_loss=0.009747, over 3040974.62 frames. ], batch size: 59, lr: 4.58e-03, grad_scale: 16.0 2023-11-20 18:12:57,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1167200.0, ans=0.0 2023-11-20 18:13:03,625 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 18:13:09,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1167266.6666666667, ans=0.1 2023-11-20 18:13:10,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1167266.6666666667, ans=0.0 2023-11-20 18:13:20,211 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175100 2023-11-20 18:14:01,469 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6800, loss[loss=0.08091, simple_loss=0.1062, pruned_loss=0.01941, audio_tagging_loss=0.008379, over 13812.00 frames. ], tot_loss[loss=0.07825, simple_loss=0.09948, pruned_loss=0.01877, audio_tagging_loss=0.009731, over 3039031.86 frames. ], batch size: 53, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:14:08,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1167533.3333333333, ans=0.125 2023-11-20 18:14:24,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175150 2023-11-20 18:14:25,244 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.115e+01 8.227e+01 9.120e+01 1.003e+02 1.352e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-20 18:14:54,451 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.02 vs. limit=15.0 2023-11-20 18:14:57,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1167800.0, ans=0.125 2023-11-20 18:14:58,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.02 vs. limit=6.0 2023-11-20 18:15:04,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1167866.6666666667, ans=0.125 2023-11-20 18:15:05,530 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6850, loss[loss=0.09279, simple_loss=0.1137, pruned_loss=0.02559, audio_tagging_loss=0.01034, over 15046.00 frames. ], tot_loss[loss=0.07859, simple_loss=0.0999, pruned_loss=0.01895, audio_tagging_loss=0.009684, over 3043169.42 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:15:05,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1167866.6666666667, ans=0.2 2023-11-20 18:15:09,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1167866.6666666667, ans=0.5 2023-11-20 18:15:14,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1167866.6666666667, ans=0.0 2023-11-20 18:15:28,853 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175200 2023-11-20 18:15:47,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1168066.6666666667, ans=0.125 2023-11-20 18:15:48,682 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=16.23 vs. limit=15.0 2023-11-20 18:15:50,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1168066.6666666667, ans=0.125 2023-11-20 18:15:52,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1168066.6666666667, ans=0.0 2023-11-20 18:15:52,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1168066.6666666667, ans=0.0 2023-11-20 18:15:53,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1168066.6666666667, ans=0.1 2023-11-20 18:16:09,518 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.68 vs. limit=22.5 2023-11-20 18:16:10,195 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6900, loss[loss=0.07396, simple_loss=0.09189, pruned_loss=0.01602, audio_tagging_loss=0.01198, over 15560.00 frames. ], tot_loss[loss=0.07934, simple_loss=0.1011, pruned_loss=0.01918, audio_tagging_loss=0.00964, over 3046077.04 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:16:12,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1168200.0, ans=0.1 2023-11-20 18:16:15,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1168200.0, ans=0.1 2023-11-20 18:16:27,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=22.5 2023-11-20 18:16:33,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175250 2023-11-20 18:16:34,532 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.834e+01 8.085e+01 8.884e+01 9.597e+01 1.266e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-20 18:16:52,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1168400.0, ans=0.2 2023-11-20 18:17:00,895 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 18:17:12,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.17 vs. limit=15.0 2023-11-20 18:17:14,807 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 6950, loss[loss=0.08748, simple_loss=0.1197, pruned_loss=0.02128, audio_tagging_loss=0.00636, over 15999.00 frames. ], tot_loss[loss=0.07953, simple_loss=0.1014, pruned_loss=0.01915, audio_tagging_loss=0.009691, over 3054342.79 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:17:15,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1168533.3333333333, ans=0.04949747468305833 2023-11-20 18:17:18,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1168533.3333333333, ans=0.125 2023-11-20 18:17:30,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1168600.0, ans=0.125 2023-11-20 18:17:37,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175300 2023-11-20 18:17:58,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1168733.3333333333, ans=0.09899494936611666 2023-11-20 18:17:59,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1168733.3333333333, ans=0.125 2023-11-20 18:18:09,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1168800.0, ans=15.0 2023-11-20 18:18:18,370 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7000, loss[loss=0.06155, simple_loss=0.06736, pruned_loss=0.01549, audio_tagging_loss=0.01238, over 13642.00 frames. ], tot_loss[loss=0.07882, simple_loss=0.1003, pruned_loss=0.01889, audio_tagging_loss=0.009765, over 3058018.00 frames. ], batch size: 55, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:18:20,369 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.91 vs. limit=15.0 2023-11-20 18:18:41,076 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=15.0 2023-11-20 18:18:41,575 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175350 2023-11-20 18:18:42,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten.whitening_limit, batch_count=1168933.3333333333, ans=15.0 2023-11-20 18:18:42,699 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 7.873e+01 8.691e+01 9.185e+01 1.165e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-20 18:18:59,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1169066.6666666667, ans=0.0 2023-11-20 18:18:59,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1169066.6666666667, ans=0.09899494936611666 2023-11-20 18:19:00,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1169066.6666666667, ans=0.125 2023-11-20 18:19:04,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1169066.6666666667, ans=0.125 2023-11-20 18:19:06,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1169066.6666666667, ans=0.0 2023-11-20 18:19:18,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1169133.3333333333, ans=0.1 2023-11-20 18:19:21,841 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7050, loss[loss=0.08123, simple_loss=0.1104, pruned_loss=0.01632, audio_tagging_loss=0.009704, over 15959.00 frames. ], tot_loss[loss=0.07914, simple_loss=0.1005, pruned_loss=0.01898, audio_tagging_loss=0.009883, over 3059944.41 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:19:27,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1169200.0, ans=0.1 2023-11-20 18:19:29,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.88 vs. limit=15.0 2023-11-20 18:19:45,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175400 2023-11-20 18:20:26,362 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7100, loss[loss=0.07076, simple_loss=0.08976, pruned_loss=0.01526, audio_tagging_loss=0.01062, over 15199.00 frames. ], tot_loss[loss=0.07889, simple_loss=0.09999, pruned_loss=0.0189, audio_tagging_loss=0.009991, over 3067696.35 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:20:31,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1169533.3333333333, ans=0.2 2023-11-20 18:20:48,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175450 2023-11-20 18:20:49,780 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.001e+01 8.747e+01 9.712e+01 1.225e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-20 18:20:52,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1169666.6666666667, ans=0.125 2023-11-20 18:20:59,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1169666.6666666667, ans=0.125 2023-11-20 18:21:13,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1169733.3333333333, ans=0.125 2023-11-20 18:21:24,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.47 vs. limit=22.5 2023-11-20 18:21:29,411 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7150, loss[loss=0.08987, simple_loss=0.1023, pruned_loss=0.0236, audio_tagging_loss=0.01514, over 14709.00 frames. ], tot_loss[loss=0.0789, simple_loss=0.09971, pruned_loss=0.01902, audio_tagging_loss=0.01002, over 3062426.68 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:21:29,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1169866.6666666667, ans=0.1 2023-11-20 18:21:47,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1169933.3333333333, ans=0.125 2023-11-20 18:21:53,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175500 2023-11-20 18:22:09,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1170066.6666666667, ans=0.125 2023-11-20 18:22:12,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1170066.6666666667, ans=0.0 2023-11-20 18:22:32,635 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7200, loss[loss=0.07901, simple_loss=0.09423, pruned_loss=0.01919, audio_tagging_loss=0.01271, over 16040.00 frames. ], tot_loss[loss=0.07905, simple_loss=0.09989, pruned_loss=0.01907, audio_tagging_loss=0.01004, over 3061584.48 frames. ], batch size: 60, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:22:41,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1170200.0, ans=0.0 2023-11-20 18:22:56,698 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175550 2023-11-20 18:22:57,773 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.535e+01 8.191e+01 8.742e+01 9.688e+01 2.740e+02, threshold=1.748e+02, percent-clipped=1.0 2023-11-20 18:23:14,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1170400.0, ans=0.95 2023-11-20 18:23:18,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1170400.0, ans=0.125 2023-11-20 18:23:37,147 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7250, loss[loss=0.08138, simple_loss=0.1013, pruned_loss=0.02234, audio_tagging_loss=0.008379, over 14930.00 frames. ], tot_loss[loss=0.07993, simple_loss=0.101, pruned_loss=0.01943, audio_tagging_loss=0.01001, over 3060570.94 frames. ], batch size: 53, lr: 4.58e-03, grad_scale: 32.0 2023-11-20 18:23:59,847 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175600 2023-11-20 18:24:11,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1170666.6666666667, ans=0.1 2023-11-20 18:24:15,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1170733.3333333333, ans=0.0 2023-11-20 18:24:21,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=1170733.3333333333, ans=0.05 2023-11-20 18:24:41,084 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7300, loss[loss=0.06855, simple_loss=0.09425, pruned_loss=0.01356, audio_tagging_loss=0.00786, over 14288.00 frames. ], tot_loss[loss=0.07942, simple_loss=0.1007, pruned_loss=0.0191, audio_tagging_loss=0.009959, over 3058237.79 frames. ], batch size: 53, lr: 4.58e-03, grad_scale: 16.0 2023-11-20 18:24:56,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1170933.3333333333, ans=0.0 2023-11-20 18:24:57,284 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 18:25:03,275 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175650 2023-11-20 18:25:05,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1171000.0, ans=0.1 2023-11-20 18:25:06,276 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 7.796e+01 8.638e+01 9.366e+01 1.171e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-20 18:25:14,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1171000.0, ans=0.125 2023-11-20 18:25:37,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.53 vs. limit=15.0 2023-11-20 18:25:43,779 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7350, loss[loss=0.07483, simple_loss=0.09471, pruned_loss=0.01403, audio_tagging_loss=0.01344, over 14949.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.09989, pruned_loss=0.01889, audio_tagging_loss=0.009788, over 3054283.83 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 16.0 2023-11-20 18:26:07,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175700 2023-11-20 18:26:07,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1171266.6666666667, ans=0.1 2023-11-20 18:26:22,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1171400.0, ans=0.0 2023-11-20 18:26:28,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.69 vs. limit=10.0 2023-11-20 18:26:47,853 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7400, loss[loss=0.07321, simple_loss=0.09426, pruned_loss=0.0182, audio_tagging_loss=0.007881, over 15792.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09935, pruned_loss=0.01877, audio_tagging_loss=0.00977, over 3052389.86 frames. ], batch size: 59, lr: 4.58e-03, grad_scale: 16.0 2023-11-20 18:27:10,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175750 2023-11-20 18:27:12,661 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 8.125e+01 8.808e+01 9.818e+01 1.259e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-20 18:27:47,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1171800.0, ans=0.125 2023-11-20 18:27:49,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1171800.0, ans=0.1 2023-11-20 18:27:49,667 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0 2023-11-20 18:27:51,323 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7450, loss[loss=0.09313, simple_loss=0.1286, pruned_loss=0.02269, audio_tagging_loss=0.006162, over 15371.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09843, pruned_loss=0.01859, audio_tagging_loss=0.009804, over 3053781.43 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 16.0 2023-11-20 18:27:55,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1171866.6666666667, ans=0.025 2023-11-20 18:28:00,324 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=2.531e-03 2023-11-20 18:28:03,237 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.34 vs. limit=15.0 2023-11-20 18:28:13,581 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175800 2023-11-20 18:28:18,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1172000.0, ans=0.0 2023-11-20 18:28:35,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1172066.6666666667, ans=0.125 2023-11-20 18:28:53,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1172200.0, ans=0.0 2023-11-20 18:28:54,505 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7500, loss[loss=0.1031, simple_loss=0.1386, pruned_loss=0.02565, audio_tagging_loss=0.008147, over 15168.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.09898, pruned_loss=0.0187, audio_tagging_loss=0.009829, over 3050779.55 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 16.0 2023-11-20 18:29:01,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1172200.0, ans=0.0 2023-11-20 18:29:03,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1172200.0, ans=0.0 2023-11-20 18:29:12,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1172266.6666666667, ans=0.125 2023-11-20 18:29:19,359 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175850 2023-11-20 18:29:21,646 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.057e+01 8.747e+01 9.687e+01 1.264e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-20 18:29:29,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1172333.3333333333, ans=0.125 2023-11-20 18:29:55,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1172466.6666666667, ans=0.125 2023-11-20 18:29:59,039 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7550, loss[loss=0.08643, simple_loss=0.1142, pruned_loss=0.01987, audio_tagging_loss=0.009479, over 15880.00 frames. ], tot_loss[loss=0.078, simple_loss=0.09925, pruned_loss=0.01868, audio_tagging_loss=0.009698, over 3055363.25 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 16.0 2023-11-20 18:30:05,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1172533.3333333333, ans=0.125 2023-11-20 18:30:22,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175900 2023-11-20 18:30:40,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1172733.3333333333, ans=0.0 2023-11-20 18:30:44,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1172733.3333333333, ans=0.125 2023-11-20 18:30:48,375 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.13 vs. limit=15.0 2023-11-20 18:30:50,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1172800.0, ans=0.125 2023-11-20 18:31:03,463 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7600, loss[loss=0.07399, simple_loss=0.09646, pruned_loss=0.01526, audio_tagging_loss=0.01049, over 14754.00 frames. ], tot_loss[loss=0.07838, simple_loss=0.09978, pruned_loss=0.01879, audio_tagging_loss=0.009701, over 3050290.59 frames. ], batch size: 54, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:31:04,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1172866.6666666667, ans=22.5 2023-11-20 18:31:08,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1172866.6666666667, ans=0.125 2023-11-20 18:31:17,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1172933.3333333333, ans=0.5 2023-11-20 18:31:17,604 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.46 vs. limit=15.0 2023-11-20 18:31:19,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1172933.3333333333, ans=0.1 2023-11-20 18:31:25,584 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 175950 2023-11-20 18:31:25,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1172933.3333333333, ans=0.125 2023-11-20 18:31:27,952 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.999e+01 7.835e+01 8.443e+01 9.077e+01 1.271e+02, threshold=1.689e+02, percent-clipped=0.0 2023-11-20 18:31:36,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1173000.0, ans=0.125 2023-11-20 18:31:51,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1173066.6666666667, ans=0.125 2023-11-20 18:32:01,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1173133.3333333333, ans=0.125 2023-11-20 18:32:06,913 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7650, loss[loss=0.07043, simple_loss=0.09358, pruned_loss=0.01638, audio_tagging_loss=0.007261, over 15110.00 frames. ], tot_loss[loss=0.07805, simple_loss=0.09929, pruned_loss=0.01861, audio_tagging_loss=0.009791, over 3044587.87 frames. ], batch size: 57, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:32:09,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1173200.0, ans=0.125 2023-11-20 18:32:23,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1173266.6666666667, ans=0.1 2023-11-20 18:32:30,206 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176000 2023-11-20 18:32:45,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1173333.3333333333, ans=0.125 2023-11-20 18:32:48,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1173400.0, ans=0.125 2023-11-20 18:32:51,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.24 vs. limit=22.5 2023-11-20 18:32:58,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1173400.0, ans=0.0 2023-11-20 18:33:06,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1173466.6666666667, ans=0.0 2023-11-20 18:33:14,111 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7700, loss[loss=0.05096, simple_loss=0.0656, pruned_loss=0.008078, audio_tagging_loss=0.01008, over 14072.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.0986, pruned_loss=0.01826, audio_tagging_loss=0.009757, over 3041217.91 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:33:27,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1173600.0, ans=0.125 2023-11-20 18:33:37,305 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176050 2023-11-20 18:33:39,690 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.684e+01 8.040e+01 8.573e+01 9.268e+01 1.195e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-20 18:33:41,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1173666.6666666667, ans=0.2 2023-11-20 18:34:04,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1173800.0, ans=0.2 2023-11-20 18:34:18,335 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7750, loss[loss=0.07321, simple_loss=0.09017, pruned_loss=0.01566, audio_tagging_loss=0.01247, over 16160.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09949, pruned_loss=0.01842, audio_tagging_loss=0.009774, over 3047923.96 frames. ], batch size: 61, lr: 4.57e-03, grad_scale: 16.0 2023-11-20 18:34:41,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176100 2023-11-20 18:34:51,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.74 vs. limit=15.0 2023-11-20 18:34:58,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1174066.6666666667, ans=0.0 2023-11-20 18:34:59,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1174066.6666666667, ans=0.125 2023-11-20 18:34:59,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1174066.6666666667, ans=0.125 2023-11-20 18:35:15,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1174133.3333333333, ans=0.1 2023-11-20 18:35:22,301 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7800, loss[loss=0.07107, simple_loss=0.07088, pruned_loss=0.01659, audio_tagging_loss=0.01904, over 15083.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.0994, pruned_loss=0.01861, audio_tagging_loss=0.009889, over 3038977.49 frames. ], batch size: 62, lr: 4.57e-03, grad_scale: 16.0 2023-11-20 18:35:27,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1174200.0, ans=0.125 2023-11-20 18:35:40,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1174266.6666666667, ans=0.0 2023-11-20 18:35:41,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1174266.6666666667, ans=0.125 2023-11-20 18:35:45,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176150 2023-11-20 18:35:48,939 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.331e+01 9.045e+01 1.006e+02 1.273e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-20 18:35:55,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.74 vs. limit=22.5 2023-11-20 18:35:56,352 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.78 vs. limit=10.0 2023-11-20 18:36:14,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1174466.6666666667, ans=0.2 2023-11-20 18:36:21,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1174466.6666666667, ans=0.2 2023-11-20 18:36:26,128 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7850, loss[loss=0.06447, simple_loss=0.08266, pruned_loss=0.01217, audio_tagging_loss=0.01096, over 16233.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09969, pruned_loss=0.01865, audio_tagging_loss=0.009872, over 3038186.79 frames. ], batch size: 60, lr: 4.57e-03, grad_scale: 16.0 2023-11-20 18:36:35,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1174533.3333333333, ans=22.5 2023-11-20 18:36:43,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1174600.0, ans=0.2 2023-11-20 18:36:50,249 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176200 2023-11-20 18:36:57,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.25 vs. limit=15.0 2023-11-20 18:36:58,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1174666.6666666667, ans=0.0 2023-11-20 18:37:13,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1174733.3333333333, ans=0.125 2023-11-20 18:37:17,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1174800.0, ans=0.5 2023-11-20 18:37:23,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1174800.0, ans=0.0 2023-11-20 18:37:31,655 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7900, loss[loss=0.0935, simple_loss=0.1202, pruned_loss=0.02449, audio_tagging_loss=0.008899, over 15836.00 frames. ], tot_loss[loss=0.07868, simple_loss=0.09977, pruned_loss=0.0188, audio_tagging_loss=0.009995, over 3040869.35 frames. ], batch size: 58, lr: 4.57e-03, grad_scale: 16.0 2023-11-20 18:37:34,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.72 vs. limit=15.0 2023-11-20 18:37:38,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1174866.6666666667, ans=0.0 2023-11-20 18:37:43,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1174933.3333333333, ans=0.09899494936611666 2023-11-20 18:37:54,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176250 2023-11-20 18:37:57,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1175000.0, ans=0.1 2023-11-20 18:37:57,980 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.028e+01 8.743e+01 9.651e+01 2.118e+02, threshold=1.749e+02, percent-clipped=1.0 2023-11-20 18:38:15,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1175066.6666666667, ans=0.07 2023-11-20 18:38:25,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1175133.3333333333, ans=0.1 2023-11-20 18:38:35,860 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 7950, loss[loss=0.1077, simple_loss=0.1404, pruned_loss=0.0263, audio_tagging_loss=0.01115, over 14741.00 frames. ], tot_loss[loss=0.07896, simple_loss=0.09995, pruned_loss=0.01888, audio_tagging_loss=0.01011, over 3040900.61 frames. ], batch size: 54, lr: 4.57e-03, grad_scale: 16.0 2023-11-20 18:38:43,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1175200.0, ans=10.0 2023-11-20 18:38:51,799 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 18:38:59,274 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176300 2023-11-20 18:38:59,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1175266.6666666667, ans=0.125 2023-11-20 18:39:00,047 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=9.23 vs. limit=15.0 2023-11-20 18:39:27,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1175466.6666666667, ans=0.0 2023-11-20 18:39:28,279 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.17 vs. limit=15.0 2023-11-20 18:39:39,811 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8000, loss[loss=0.08023, simple_loss=0.1037, pruned_loss=0.01801, audio_tagging_loss=0.01034, over 15790.00 frames. ], tot_loss[loss=0.07877, simple_loss=0.09968, pruned_loss=0.01883, audio_tagging_loss=0.01009, over 3044881.96 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:39:50,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1175533.3333333333, ans=0.125 2023-11-20 18:39:50,966 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 18:40:03,684 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176350 2023-11-20 18:40:07,203 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.314e+01 8.360e+01 9.089e+01 9.769e+01 1.308e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-20 18:40:20,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1175733.3333333333, ans=0.2 2023-11-20 18:40:25,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1175733.3333333333, ans=0.05 2023-11-20 18:40:25,507 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.32 vs. limit=15.0 2023-11-20 18:40:38,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1175800.0, ans=0.0 2023-11-20 18:40:41,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.85 vs. limit=15.0 2023-11-20 18:40:44,409 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8050, loss[loss=0.05732, simple_loss=0.07075, pruned_loss=0.012, audio_tagging_loss=0.009944, over 14521.00 frames. ], tot_loss[loss=0.07892, simple_loss=0.09985, pruned_loss=0.01885, audio_tagging_loss=0.01014, over 3042526.90 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:41:01,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1175933.3333333333, ans=0.2 2023-11-20 18:41:07,862 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176400 2023-11-20 18:41:16,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1176000.0, ans=0.5 2023-11-20 18:41:43,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.22 vs. limit=10.0 2023-11-20 18:41:44,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1176133.3333333333, ans=0.125 2023-11-20 18:41:49,098 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8100, loss[loss=0.07227, simple_loss=0.08472, pruned_loss=0.01839, audio_tagging_loss=0.01152, over 14869.00 frames. ], tot_loss[loss=0.07961, simple_loss=0.1007, pruned_loss=0.01919, audio_tagging_loss=0.01008, over 3047106.26 frames. ], batch size: 54, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:41:53,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.81 vs. limit=15.0 2023-11-20 18:42:00,026 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.92 vs. limit=15.0 2023-11-20 18:42:13,027 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176450 2023-11-20 18:42:16,562 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 7.970e+01 8.520e+01 9.468e+01 1.287e+02, threshold=1.704e+02, percent-clipped=0.0 2023-11-20 18:42:23,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1176333.3333333333, ans=0.1 2023-11-20 18:42:39,759 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.57 vs. limit=6.0 2023-11-20 18:42:53,261 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8150, loss[loss=0.09107, simple_loss=0.1179, pruned_loss=0.02222, audio_tagging_loss=0.00989, over 17550.00 frames. ], tot_loss[loss=0.07924, simple_loss=0.1004, pruned_loss=0.01906, audio_tagging_loss=0.009999, over 3052826.86 frames. ], batch size: 65, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:43:09,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1176600.0, ans=0.1 2023-11-20 18:43:17,135 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176500 2023-11-20 18:43:19,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1176666.6666666667, ans=0.04949747468305833 2023-11-20 18:43:29,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1176666.6666666667, ans=0.125 2023-11-20 18:43:29,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1176666.6666666667, ans=0.125 2023-11-20 18:43:57,833 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8200, loss[loss=0.08893, simple_loss=0.1154, pruned_loss=0.02297, audio_tagging_loss=0.008275, over 14892.00 frames. ], tot_loss[loss=0.07961, simple_loss=0.101, pruned_loss=0.01925, audio_tagging_loss=0.009832, over 3043532.62 frames. ], batch size: 53, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:43:59,135 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 18:44:02,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1176866.6666666667, ans=0.1 2023-11-20 18:44:06,599 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.93 vs. limit=15.0 2023-11-20 18:44:12,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1176933.3333333333, ans=0.0 2023-11-20 18:44:20,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176550 2023-11-20 18:44:24,658 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.959e+01 8.657e+01 9.371e+01 1.307e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-20 18:45:02,156 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8250, loss[loss=0.07596, simple_loss=0.09789, pruned_loss=0.01751, audio_tagging_loss=0.009508, over 15248.00 frames. ], tot_loss[loss=0.07933, simple_loss=0.1009, pruned_loss=0.01914, audio_tagging_loss=0.009714, over 3038724.58 frames. ], batch size: 57, lr: 4.57e-03, grad_scale: 32.0 2023-11-20 18:45:04,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=1177200.0, ans=0.1 2023-11-20 18:45:25,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176600 2023-11-20 18:45:46,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1177400.0, ans=10.0 2023-11-20 18:45:47,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1177400.0, ans=0.125 2023-11-20 18:45:49,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1177400.0, ans=0.125 2023-11-20 18:45:58,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1177466.6666666667, ans=0.0 2023-11-20 18:45:59,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1177466.6666666667, ans=0.125 2023-11-20 18:46:06,157 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8300, loss[loss=0.06727, simple_loss=0.08943, pruned_loss=0.009911, audio_tagging_loss=0.01264, over 15674.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.09934, pruned_loss=0.01862, audio_tagging_loss=0.009918, over 3039592.81 frames. ], batch size: 59, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:46:27,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1177600.0, ans=0.1 2023-11-20 18:46:30,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176650 2023-11-20 18:46:33,816 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.522e+01 7.955e+01 8.600e+01 9.336e+01 1.507e+02, threshold=1.720e+02, percent-clipped=0.0 2023-11-20 18:47:00,996 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.45 vs. limit=15.0 2023-11-20 18:47:10,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1177866.6666666667, ans=0.0 2023-11-20 18:47:11,255 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8350, loss[loss=0.05171, simple_loss=0.07103, pruned_loss=0.006694, audio_tagging_loss=0.009503, over 14804.00 frames. ], tot_loss[loss=0.07782, simple_loss=0.09899, pruned_loss=0.01848, audio_tagging_loss=0.009846, over 3041761.79 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:47:16,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1177866.6666666667, ans=0.125 2023-11-20 18:47:33,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176700 2023-11-20 18:47:36,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1178000.0, ans=0.0 2023-11-20 18:47:43,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1178000.0, ans=0.2 2023-11-20 18:47:56,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1178066.6666666667, ans=0.125 2023-11-20 18:48:05,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1178133.3333333333, ans=0.125 2023-11-20 18:48:12,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1178133.3333333333, ans=0.05 2023-11-20 18:48:15,704 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8400, loss[loss=0.07535, simple_loss=0.09557, pruned_loss=0.01822, audio_tagging_loss=0.009347, over 14596.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09922, pruned_loss=0.01848, audio_tagging_loss=0.009837, over 3045165.43 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:48:19,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1178200.0, ans=0.09899494936611666 2023-11-20 18:48:38,690 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176750 2023-11-20 18:48:42,873 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.237e+01 7.973e+01 8.686e+01 9.472e+01 1.183e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-20 18:48:45,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.50 vs. limit=22.5 2023-11-20 18:49:09,369 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.51 vs. limit=15.0 2023-11-20 18:49:16,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.14 vs. limit=22.5 2023-11-20 18:49:19,821 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8450, loss[loss=0.0818, simple_loss=0.09965, pruned_loss=0.0235, audio_tagging_loss=0.008479, over 14885.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09839, pruned_loss=0.0184, audio_tagging_loss=0.01004, over 3045675.36 frames. ], batch size: 54, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:49:30,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1178533.3333333333, ans=0.0 2023-11-20 18:49:38,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1178600.0, ans=0.0 2023-11-20 18:49:39,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1178600.0, ans=0.125 2023-11-20 18:49:41,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1178600.0, ans=0.0 2023-11-20 18:49:43,451 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176800 2023-11-20 18:49:48,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.85 vs. limit=10.0 2023-11-20 18:50:00,805 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 18:50:05,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1178733.3333333333, ans=0.04949747468305833 2023-11-20 18:50:25,063 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8500, loss[loss=0.06144, simple_loss=0.07687, pruned_loss=0.01329, audio_tagging_loss=0.009711, over 15539.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09823, pruned_loss=0.01854, audio_tagging_loss=0.009971, over 3044809.41 frames. ], batch size: 62, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:50:31,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1178866.6666666667, ans=0.125 2023-11-20 18:50:45,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1178933.3333333333, ans=0.1 2023-11-20 18:50:46,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1178933.3333333333, ans=0.125 2023-11-20 18:50:47,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176850 2023-11-20 18:50:51,458 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.113e+01 8.136e+01 8.962e+01 9.902e+01 1.387e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-20 18:50:58,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1179000.0, ans=0.125 2023-11-20 18:51:05,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1179066.6666666667, ans=0.125 2023-11-20 18:51:12,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1179066.6666666667, ans=0.2 2023-11-20 18:51:20,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1179133.3333333333, ans=0.015 2023-11-20 18:51:28,718 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8550, loss[loss=0.08278, simple_loss=0.1217, pruned_loss=0.0148, audio_tagging_loss=0.007159, over 15020.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.09891, pruned_loss=0.01858, audio_tagging_loss=0.009944, over 3052283.57 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:51:32,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1179200.0, ans=0.125 2023-11-20 18:51:35,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1179200.0, ans=0.05 2023-11-20 18:51:39,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1179200.0, ans=0.2 2023-11-20 18:51:51,431 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176900 2023-11-20 18:52:02,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1179333.3333333333, ans=0.0 2023-11-20 18:52:12,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.95 vs. limit=6.0 2023-11-20 18:52:15,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1179400.0, ans=0.1 2023-11-20 18:52:18,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1179400.0, ans=0.125 2023-11-20 18:52:23,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1179466.6666666667, ans=0.2 2023-11-20 18:52:31,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1179533.3333333333, ans=0.125 2023-11-20 18:52:31,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1179533.3333333333, ans=0.125 2023-11-20 18:52:32,736 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8600, loss[loss=0.0823, simple_loss=0.1041, pruned_loss=0.02088, audio_tagging_loss=0.009372, over 16764.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09884, pruned_loss=0.01857, audio_tagging_loss=0.009885, over 3059524.02 frames. ], batch size: 65, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:52:52,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1179600.0, ans=0.125 2023-11-20 18:52:56,729 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 176950 2023-11-20 18:52:56,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1179600.0, ans=0.1 2023-11-20 18:52:59,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.03 vs. limit=22.5 2023-11-20 18:53:00,850 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.645e+01 8.095e+01 8.772e+01 9.480e+01 1.996e+02, threshold=1.754e+02, percent-clipped=1.0 2023-11-20 18:53:37,653 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8650, loss[loss=0.07499, simple_loss=0.1002, pruned_loss=0.01642, audio_tagging_loss=0.008452, over 16381.00 frames. ], tot_loss[loss=0.07777, simple_loss=0.09863, pruned_loss=0.0185, audio_tagging_loss=0.009961, over 3055955.47 frames. ], batch size: 59, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:53:44,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1179866.6666666667, ans=0.125 2023-11-20 18:53:59,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1179933.3333333333, ans=0.0 2023-11-20 18:54:01,270 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177000 2023-11-20 18:54:29,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1180133.3333333333, ans=0.0 2023-11-20 18:54:36,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1180133.3333333333, ans=0.09899494936611666 2023-11-20 18:54:43,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5 2023-11-20 18:54:43,913 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8700, loss[loss=0.1017, simple_loss=0.1321, pruned_loss=0.02513, audio_tagging_loss=0.0105, over 14959.00 frames. ], tot_loss[loss=0.07756, simple_loss=0.09829, pruned_loss=0.01841, audio_tagging_loss=0.01, over 3053588.86 frames. ], batch size: 55, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:54:54,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1180200.0, ans=0.125 2023-11-20 18:54:57,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1180266.6666666667, ans=0.1 2023-11-20 18:54:58,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1180266.6666666667, ans=0.125 2023-11-20 18:55:01,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1180266.6666666667, ans=0.2 2023-11-20 18:55:06,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177050 2023-11-20 18:55:10,012 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.109e+01 8.720e+01 9.588e+01 1.618e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-20 18:55:15,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1180333.3333333333, ans=0.125 2023-11-20 18:55:17,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1180333.3333333333, ans=0.1 2023-11-20 18:55:20,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1180333.3333333333, ans=0.125 2023-11-20 18:55:36,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1180466.6666666667, ans=0.0 2023-11-20 18:55:48,035 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8750, loss[loss=0.08278, simple_loss=0.1025, pruned_loss=0.02305, audio_tagging_loss=0.008461, over 14726.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09897, pruned_loss=0.01866, audio_tagging_loss=0.00997, over 3051749.46 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 16.0 2023-11-20 18:55:54,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.77 vs. limit=22.5 2023-11-20 18:55:58,459 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.71 vs. limit=15.0 2023-11-20 18:56:11,311 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177100 2023-11-20 18:56:20,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1180666.6666666667, ans=0.1 2023-11-20 18:56:41,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1180800.0, ans=0.1 2023-11-20 18:56:52,266 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8800, loss[loss=0.06977, simple_loss=0.0951, pruned_loss=0.01347, audio_tagging_loss=0.008743, over 15411.00 frames. ], tot_loss[loss=0.07845, simple_loss=0.09951, pruned_loss=0.01866, audio_tagging_loss=0.01004, over 3044953.16 frames. ], batch size: 58, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:57:16,502 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177150 2023-11-20 18:57:16,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1180933.3333333333, ans=0.0 2023-11-20 18:57:21,357 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.135e+01 8.370e+01 9.044e+01 9.709e+01 1.257e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-20 18:57:25,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.17 vs. limit=12.0 2023-11-20 18:57:30,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1181066.6666666667, ans=0.0 2023-11-20 18:57:46,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1181133.3333333333, ans=0.125 2023-11-20 18:57:46,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1181133.3333333333, ans=0.5 2023-11-20 18:57:51,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1181133.3333333333, ans=0.125 2023-11-20 18:57:58,609 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8850, loss[loss=0.09595, simple_loss=0.1286, pruned_loss=0.02157, audio_tagging_loss=0.0101, over 15749.00 frames. ], tot_loss[loss=0.07901, simple_loss=0.1005, pruned_loss=0.01879, audio_tagging_loss=0.009965, over 3051333.22 frames. ], batch size: 58, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:58:10,997 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 18:58:17,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1181266.6666666667, ans=0.0 2023-11-20 18:58:20,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177200 2023-11-20 18:58:26,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1181333.3333333333, ans=0.0 2023-11-20 18:58:26,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1181333.3333333333, ans=10.0 2023-11-20 18:58:36,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1181400.0, ans=0.125 2023-11-20 18:58:48,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1181400.0, ans=0.125 2023-11-20 18:58:50,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=1181466.6666666667, ans=0.1 2023-11-20 18:58:50,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1181466.6666666667, ans=0.125 2023-11-20 18:59:03,529 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8900, loss[loss=0.06333, simple_loss=0.08306, pruned_loss=0.01308, audio_tagging_loss=0.008721, over 15047.00 frames. ], tot_loss[loss=0.07881, simple_loss=0.1001, pruned_loss=0.01879, audio_tagging_loss=0.00997, over 3047676.36 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 18:59:09,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1181533.3333333333, ans=0.0 2023-11-20 18:59:12,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1181533.3333333333, ans=0.125 2023-11-20 18:59:13,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1181533.3333333333, ans=0.0 2023-11-20 18:59:27,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177250 2023-11-20 18:59:32,817 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.317e+01 8.861e+01 9.855e+01 1.345e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-20 19:00:00,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1181800.0, ans=0.125 2023-11-20 19:00:08,819 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 8950, loss[loss=0.08255, simple_loss=0.1008, pruned_loss=0.02125, audio_tagging_loss=0.01089, over 14851.00 frames. ], tot_loss[loss=0.07897, simple_loss=0.1003, pruned_loss=0.01895, audio_tagging_loss=0.009844, over 3054624.44 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 19:00:20,426 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.07 vs. limit=15.0 2023-11-20 19:00:26,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1181933.3333333333, ans=0.0 2023-11-20 19:00:33,298 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177300 2023-11-20 19:00:51,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1182066.6666666667, ans=0.125 2023-11-20 19:01:04,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1182133.3333333333, ans=0.2 2023-11-20 19:01:04,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1182133.3333333333, ans=0.2 2023-11-20 19:01:13,864 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9000, loss[loss=0.06882, simple_loss=0.07687, pruned_loss=0.01575, audio_tagging_loss=0.01464, over 15302.00 frames. ], tot_loss[loss=0.07951, simple_loss=0.1013, pruned_loss=0.0191, audio_tagging_loss=0.009776, over 3058739.49 frames. ], batch size: 60, lr: 4.56e-03, grad_scale: 32.0 2023-11-20 19:01:13,865 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 19:01:41,844 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.0055, 5.9127, 5.7330, 5.5958], device='cuda:3') 2023-11-20 19:01:56,957 INFO [train_asr.py:1253] (3/4) Epoch 15, validation: loss=0.0619, simple_loss=0.05318, pruned_loss=0.005552, audio_tagging_loss=0.02975, over 4681554.00 frames. 2023-11-20 19:01:56,958 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 19:02:00,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.97 vs. limit=15.0 2023-11-20 19:02:08,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.41 vs. limit=15.0 2023-11-20 19:02:09,910 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=5.27 vs. limit=15.0 2023-11-20 19:02:20,423 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177350 2023-11-20 19:02:25,236 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.224e+01 8.889e+01 9.368e+01 1.234e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-20 19:02:27,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1182333.3333333333, ans=0.1 2023-11-20 19:02:29,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1182333.3333333333, ans=0.0 2023-11-20 19:02:42,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1182400.0, ans=0.1 2023-11-20 19:02:46,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.42 vs. limit=15.0 2023-11-20 19:02:58,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1182466.6666666667, ans=0.125 2023-11-20 19:03:02,037 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9050, loss[loss=0.08811, simple_loss=0.1143, pruned_loss=0.0218, audio_tagging_loss=0.009157, over 14911.00 frames. ], tot_loss[loss=0.07926, simple_loss=0.1009, pruned_loss=0.01906, audio_tagging_loss=0.009763, over 3053405.60 frames. ], batch size: 54, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:03:22,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1182600.0, ans=0.125 2023-11-20 19:03:25,563 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177400 2023-11-20 19:03:41,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.66 vs. limit=15.0 2023-11-20 19:04:01,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1182800.0, ans=0.125 2023-11-20 19:04:07,291 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9100, loss[loss=0.06937, simple_loss=0.09033, pruned_loss=0.01532, audio_tagging_loss=0.008883, over 14667.00 frames. ], tot_loss[loss=0.07909, simple_loss=0.1004, pruned_loss=0.01906, audio_tagging_loss=0.009837, over 3058735.68 frames. ], batch size: 55, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:04:11,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1182866.6666666667, ans=0.1 2023-11-20 19:04:30,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177450 2023-11-20 19:04:34,930 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.170e+01 8.840e+01 9.659e+01 1.289e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-20 19:04:56,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1183066.6666666667, ans=0.125 2023-11-20 19:05:12,043 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9150, loss[loss=0.07559, simple_loss=0.09576, pruned_loss=0.0175, audio_tagging_loss=0.01021, over 16262.00 frames. ], tot_loss[loss=0.07885, simple_loss=0.1003, pruned_loss=0.01882, audio_tagging_loss=0.009868, over 3060492.63 frames. ], batch size: 62, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:05:35,355 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177500 2023-11-20 19:05:44,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1183333.3333333333, ans=0.0 2023-11-20 19:06:04,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.20 vs. limit=15.0 2023-11-20 19:06:06,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1183466.6666666667, ans=0.2 2023-11-20 19:06:15,564 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9200, loss[loss=0.08022, simple_loss=0.09782, pruned_loss=0.02084, audio_tagging_loss=0.01047, over 16419.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.1002, pruned_loss=0.01869, audio_tagging_loss=0.009814, over 3059403.89 frames. ], batch size: 63, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:06:39,212 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177550 2023-11-20 19:06:44,016 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.006e+01 8.354e+01 8.939e+01 9.701e+01 1.171e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-20 19:06:45,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1183666.6666666667, ans=0.0 2023-11-20 19:06:51,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1183666.6666666667, ans=0.0 2023-11-20 19:07:00,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1183733.3333333333, ans=0.5 2023-11-20 19:07:20,270 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9250, loss[loss=0.09013, simple_loss=0.1135, pruned_loss=0.02513, audio_tagging_loss=0.008235, over 14235.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.09945, pruned_loss=0.01866, audio_tagging_loss=0.009814, over 3061476.64 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:07:27,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1183866.6666666667, ans=0.1 2023-11-20 19:07:39,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1183933.3333333333, ans=0.125 2023-11-20 19:07:43,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177600 2023-11-20 19:07:46,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1184000.0, ans=0.125 2023-11-20 19:07:47,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1184000.0, ans=0.125 2023-11-20 19:08:05,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.72 vs. limit=22.5 2023-11-20 19:08:18,792 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:08:19,251 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.69 vs. limit=15.0 2023-11-20 19:08:24,595 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9300, loss[loss=0.08801, simple_loss=0.1111, pruned_loss=0.02402, audio_tagging_loss=0.00846, over 15482.00 frames. ], tot_loss[loss=0.07796, simple_loss=0.09887, pruned_loss=0.0187, audio_tagging_loss=0.009814, over 3064676.19 frames. ], batch size: 57, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:08:35,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1184200.0, ans=0.0 2023-11-20 19:08:48,433 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177650 2023-11-20 19:08:53,135 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 8.090e+01 8.777e+01 9.713e+01 1.132e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-20 19:08:56,319 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.20 vs. limit=15.0 2023-11-20 19:09:24,414 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.60 vs. limit=15.0 2023-11-20 19:09:25,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.25 vs. limit=15.0 2023-11-20 19:09:28,666 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9350, loss[loss=0.09554, simple_loss=0.1217, pruned_loss=0.02406, audio_tagging_loss=0.01065, over 15494.00 frames. ], tot_loss[loss=0.07852, simple_loss=0.09952, pruned_loss=0.01893, audio_tagging_loss=0.009833, over 3063259.65 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:09:30,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1184533.3333333333, ans=0.2 2023-11-20 19:09:50,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1184600.0, ans=0.125 2023-11-20 19:09:52,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177700 2023-11-20 19:10:10,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1184733.3333333333, ans=0.2 2023-11-20 19:10:33,265 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9400, loss[loss=0.07664, simple_loss=0.1044, pruned_loss=0.01641, audio_tagging_loss=0.008034, over 14721.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09935, pruned_loss=0.01885, audio_tagging_loss=0.009849, over 3057174.72 frames. ], batch size: 53, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:10:56,066 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177750 2023-11-20 19:11:01,382 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.184e+01 8.847e+01 9.642e+01 1.225e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-20 19:11:11,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1185066.6666666667, ans=0.125 2023-11-20 19:11:26,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.85 vs. limit=10.0 2023-11-20 19:11:33,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1185133.3333333333, ans=0.1 2023-11-20 19:11:33,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1185133.3333333333, ans=0.0 2023-11-20 19:11:35,959 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:11:37,158 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9450, loss[loss=0.06936, simple_loss=0.09548, pruned_loss=0.01223, audio_tagging_loss=0.009387, over 14974.00 frames. ], tot_loss[loss=0.07911, simple_loss=0.1005, pruned_loss=0.01895, audio_tagging_loss=0.009919, over 3056437.21 frames. ], batch size: 55, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:11:43,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1185200.0, ans=0.0 2023-11-20 19:11:43,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1185200.0, ans=15.0 2023-11-20 19:12:00,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1185266.6666666667, ans=0.04949747468305833 2023-11-20 19:12:00,881 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177800 2023-11-20 19:12:19,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1185400.0, ans=0.0 2023-11-20 19:12:22,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1185400.0, ans=0.0 2023-11-20 19:12:24,871 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.59 vs. limit=22.5 2023-11-20 19:12:37,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1185466.6666666667, ans=0.1 2023-11-20 19:12:41,815 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9500, loss[loss=0.1078, simple_loss=0.147, pruned_loss=0.02826, audio_tagging_loss=0.006067, over 15145.00 frames. ], tot_loss[loss=0.07904, simple_loss=0.1002, pruned_loss=0.01895, audio_tagging_loss=0.01002, over 3053630.69 frames. ], batch size: 55, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:12:51,160 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.63 vs. limit=15.0 2023-11-20 19:12:58,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1185600.0, ans=0.2 2023-11-20 19:13:05,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177850 2023-11-20 19:13:10,720 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.422e+01 8.141e+01 8.754e+01 9.457e+01 1.227e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-20 19:13:45,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1185800.0, ans=0.2 2023-11-20 19:13:47,316 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9550, loss[loss=0.06513, simple_loss=0.08206, pruned_loss=0.01244, audio_tagging_loss=0.01166, over 14323.00 frames. ], tot_loss[loss=0.07938, simple_loss=0.1004, pruned_loss=0.0191, audio_tagging_loss=0.01009, over 3053917.33 frames. ], batch size: 55, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:13:48,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1185866.6666666667, ans=0.0 2023-11-20 19:13:58,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1185866.6666666667, ans=0.0 2023-11-20 19:14:04,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1185933.3333333333, ans=0.0 2023-11-20 19:14:05,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1185933.3333333333, ans=0.1 2023-11-20 19:14:05,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1185933.3333333333, ans=0.95 2023-11-20 19:14:10,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177900 2023-11-20 19:14:30,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=1186066.6666666667, ans=0.02 2023-11-20 19:14:32,451 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.12 vs. limit=15.0 2023-11-20 19:14:38,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1186133.3333333333, ans=0.0 2023-11-20 19:14:47,447 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:14:51,969 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9600, loss[loss=0.06408, simple_loss=0.07682, pruned_loss=0.01228, audio_tagging_loss=0.01339, over 15521.00 frames. ], tot_loss[loss=0.07876, simple_loss=0.09956, pruned_loss=0.0188, audio_tagging_loss=0.01018, over 3049537.25 frames. ], batch size: 59, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:15:04,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1186266.6666666667, ans=0.2 2023-11-20 19:15:15,783 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 177950 2023-11-20 19:15:16,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.89 vs. limit=12.0 2023-11-20 19:15:17,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1186333.3333333333, ans=0.125 2023-11-20 19:15:21,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1186333.3333333333, ans=0.2 2023-11-20 19:15:22,419 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.290e+01 9.064e+01 1.000e+02 1.775e+02, threshold=1.813e+02, percent-clipped=1.0 2023-11-20 19:15:43,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1186466.6666666667, ans=0.0 2023-11-20 19:15:56,305 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9650, loss[loss=0.05443, simple_loss=0.06753, pruned_loss=0.01208, audio_tagging_loss=0.008583, over 15383.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.09875, pruned_loss=0.01864, audio_tagging_loss=0.01019, over 3048505.52 frames. ], batch size: 58, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:16:05,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.54 vs. limit=15.0 2023-11-20 19:16:08,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.98 vs. limit=15.0 2023-11-20 19:16:12,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1186600.0, ans=0.125 2023-11-20 19:16:20,393 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178000 2023-11-20 19:16:25,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1186666.6666666667, ans=0.125 2023-11-20 19:16:26,602 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.98 vs. limit=15.0 2023-11-20 19:17:02,205 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9700, loss[loss=0.05761, simple_loss=0.07902, pruned_loss=0.01099, audio_tagging_loss=0.007111, over 14629.00 frames. ], tot_loss[loss=0.07825, simple_loss=0.09941, pruned_loss=0.01861, audio_tagging_loss=0.009929, over 3045131.14 frames. ], batch size: 54, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:17:09,473 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.26 vs. limit=10.0 2023-11-20 19:17:19,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1186933.3333333333, ans=0.125 2023-11-20 19:17:25,262 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178050 2023-11-20 19:17:28,467 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.28 vs. limit=15.0 2023-11-20 19:17:31,280 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.180e+01 8.975e+01 9.665e+01 1.279e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-20 19:17:31,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1187000.0, ans=0.1 2023-11-20 19:17:53,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1187133.3333333333, ans=0.2 2023-11-20 19:17:59,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1187133.3333333333, ans=0.2 2023-11-20 19:18:01,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1187133.3333333333, ans=0.2 2023-11-20 19:18:05,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1187200.0, ans=0.2 2023-11-20 19:18:06,465 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9750, loss[loss=0.07679, simple_loss=0.0989, pruned_loss=0.01718, audio_tagging_loss=0.01016, over 14484.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.09881, pruned_loss=0.01838, audio_tagging_loss=0.009897, over 3041015.34 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:18:10,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.82 vs. limit=15.0 2023-11-20 19:18:15,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1187200.0, ans=0.125 2023-11-20 19:18:28,239 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178100 2023-11-20 19:18:42,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1187333.3333333333, ans=0.125 2023-11-20 19:18:53,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1187400.0, ans=0.0 2023-11-20 19:18:53,870 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:18:57,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1187466.6666666667, ans=0.2 2023-11-20 19:19:01,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.38 vs. limit=22.5 2023-11-20 19:19:09,557 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9800, loss[loss=0.07165, simple_loss=0.09564, pruned_loss=0.01615, audio_tagging_loss=0.007679, over 15431.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09921, pruned_loss=0.01828, audio_tagging_loss=0.009876, over 3041861.25 frames. ], batch size: 60, lr: 4.55e-03, grad_scale: 32.0 2023-11-20 19:19:11,193 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:19:26,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1187600.0, ans=0.125 2023-11-20 19:19:33,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178150 2023-11-20 19:19:39,613 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.101e+01 8.620e+01 9.316e+01 1.225e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-20 19:20:05,622 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:20:05,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1187800.0, ans=0.2 2023-11-20 19:20:12,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1187800.0, ans=0.2 2023-11-20 19:20:14,294 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9850, loss[loss=0.07453, simple_loss=0.0939, pruned_loss=0.01579, audio_tagging_loss=0.01179, over 16351.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.0999, pruned_loss=0.01853, audio_tagging_loss=0.009717, over 3045798.68 frames. ], batch size: 64, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:20:19,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1187866.6666666667, ans=0.015 2023-11-20 19:20:20,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.82 vs. limit=10.0 2023-11-20 19:20:35,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1187933.3333333333, ans=0.05 2023-11-20 19:20:37,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178200 2023-11-20 19:20:42,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1188000.0, ans=0.2 2023-11-20 19:20:57,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1188066.6666666667, ans=0.0 2023-11-20 19:21:06,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1188133.3333333333, ans=0.125 2023-11-20 19:21:19,009 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9900, loss[loss=0.06711, simple_loss=0.08614, pruned_loss=0.01594, audio_tagging_loss=0.008092, over 13876.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09984, pruned_loss=0.01863, audio_tagging_loss=0.009603, over 3043425.83 frames. ], batch size: 55, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:21:29,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1188200.0, ans=0.0 2023-11-20 19:21:41,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178250 2023-11-20 19:21:47,808 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.012e+01 8.821e+01 9.655e+01 1.193e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-20 19:22:22,894 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 9950, loss[loss=0.07273, simple_loss=0.08641, pruned_loss=0.01712, audio_tagging_loss=0.01241, over 15267.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09896, pruned_loss=0.01851, audio_tagging_loss=0.009735, over 3038085.40 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:22:24,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.43 vs. limit=15.0 2023-11-20 19:22:32,341 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.77 vs. limit=15.0 2023-11-20 19:22:45,956 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178300 2023-11-20 19:22:58,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1188666.6666666667, ans=0.125 2023-11-20 19:23:02,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1188733.3333333333, ans=0.1 2023-11-20 19:23:04,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1188733.3333333333, ans=0.125 2023-11-20 19:23:07,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1188733.3333333333, ans=0.125 2023-11-20 19:23:09,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1188733.3333333333, ans=0.2 2023-11-20 19:23:15,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.52 vs. limit=15.0 2023-11-20 19:23:19,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1188800.0, ans=0.05 2023-11-20 19:23:22,403 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.62 vs. limit=15.0 2023-11-20 19:23:23,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1188800.0, ans=0.1 2023-11-20 19:23:27,643 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10000, loss[loss=0.08971, simple_loss=0.1124, pruned_loss=0.02302, audio_tagging_loss=0.01049, over 15379.00 frames. ], tot_loss[loss=0.07735, simple_loss=0.09831, pruned_loss=0.01848, audio_tagging_loss=0.00972, over 3042561.48 frames. ], batch size: 58, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:23:50,558 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178350 2023-11-20 19:23:51,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1189000.0, ans=0.125 2023-11-20 19:23:51,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1189000.0, ans=0.0 2023-11-20 19:23:55,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1189000.0, ans=0.0 2023-11-20 19:23:56,453 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 7.984e+01 8.693e+01 9.319e+01 1.323e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-20 19:24:03,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1189000.0, ans=15.0 2023-11-20 19:24:12,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1189066.6666666667, ans=10.0 2023-11-20 19:24:21,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1189133.3333333333, ans=0.0 2023-11-20 19:24:30,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1189200.0, ans=0.1 2023-11-20 19:24:31,283 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10050, loss[loss=0.08209, simple_loss=0.1067, pruned_loss=0.01899, audio_tagging_loss=0.00973, over 15654.00 frames. ], tot_loss[loss=0.07785, simple_loss=0.09903, pruned_loss=0.01858, audio_tagging_loss=0.009755, over 3050852.06 frames. ], batch size: 58, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:24:37,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1189200.0, ans=0.0 2023-11-20 19:24:48,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1189266.6666666667, ans=0.1 2023-11-20 19:24:51,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1189266.6666666667, ans=0.0 2023-11-20 19:24:53,396 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178400 2023-11-20 19:25:08,024 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.44 vs. limit=15.0 2023-11-20 19:25:11,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1189400.0, ans=10.0 2023-11-20 19:25:21,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1189466.6666666667, ans=0.125 2023-11-20 19:25:23,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1189466.6666666667, ans=0.0 2023-11-20 19:25:30,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1189466.6666666667, ans=0.125 2023-11-20 19:25:35,261 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10100, loss[loss=0.08703, simple_loss=0.1097, pruned_loss=0.02258, audio_tagging_loss=0.009624, over 15502.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09993, pruned_loss=0.01866, audio_tagging_loss=0.00975, over 3050973.91 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:25:36,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1189533.3333333333, ans=0.0 2023-11-20 19:25:39,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.23 vs. limit=15.0 2023-11-20 19:25:58,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178450 2023-11-20 19:26:04,390 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.521e+01 8.112e+01 8.895e+01 9.698e+01 1.490e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-20 19:26:06,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1189666.6666666667, ans=0.125 2023-11-20 19:26:18,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1189733.3333333333, ans=0.125 2023-11-20 19:26:21,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1189733.3333333333, ans=0.0 2023-11-20 19:26:26,335 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:26:35,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1189800.0, ans=0.04949747468305833 2023-11-20 19:26:38,696 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10150, loss[loss=0.07156, simple_loss=0.09211, pruned_loss=0.01373, audio_tagging_loss=0.01178, over 15287.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09894, pruned_loss=0.01835, audio_tagging_loss=0.009932, over 3045414.49 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:26:46,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1189866.6666666667, ans=0.125 2023-11-20 19:26:58,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1189933.3333333333, ans=0.1 2023-11-20 19:27:03,189 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178500 2023-11-20 19:27:07,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1190000.0, ans=0.0 2023-11-20 19:27:09,265 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:27:19,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1190066.6666666667, ans=0.125 2023-11-20 19:27:31,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1190133.3333333333, ans=0.0 2023-11-20 19:27:35,467 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.96 vs. limit=12.0 2023-11-20 19:27:37,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1190133.3333333333, ans=0.125 2023-11-20 19:27:38,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1190133.3333333333, ans=0.0 2023-11-20 19:27:43,573 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10200, loss[loss=0.05298, simple_loss=0.06422, pruned_loss=0.01192, audio_tagging_loss=0.008948, over 14687.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09877, pruned_loss=0.01839, audio_tagging_loss=0.009969, over 3045423.21 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:27:49,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1190200.0, ans=0.125 2023-11-20 19:28:07,007 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178550 2023-11-20 19:28:08,192 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:28:12,620 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.65 vs. limit=15.0 2023-11-20 19:28:13,018 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.453e+01 8.265e+01 8.993e+01 9.730e+01 1.182e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-20 19:28:16,899 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:28:21,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1190400.0, ans=0.0 2023-11-20 19:28:29,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1190400.0, ans=0.125 2023-11-20 19:28:48,085 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10250, loss[loss=0.08793, simple_loss=0.1096, pruned_loss=0.02022, audio_tagging_loss=0.0129, over 14463.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09925, pruned_loss=0.01858, audio_tagging_loss=0.009942, over 3049402.14 frames. ], batch size: 52, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:28:48,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1190533.3333333333, ans=0.125 2023-11-20 19:29:11,256 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178600 2023-11-20 19:29:17,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1190666.6666666667, ans=0.125 2023-11-20 19:29:43,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1190800.0, ans=0.0 2023-11-20 19:29:51,528 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10300, loss[loss=0.08093, simple_loss=0.1045, pruned_loss=0.01724, audio_tagging_loss=0.01142, over 14872.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.0988, pruned_loss=0.01847, audio_tagging_loss=0.01008, over 3049395.08 frames. ], batch size: 59, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:30:06,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1190933.3333333333, ans=0.0 2023-11-20 19:30:07,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1190933.3333333333, ans=0.04949747468305833 2023-11-20 19:30:15,390 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178650 2023-11-20 19:30:21,243 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.948e+01 8.606e+01 9.117e+01 1.225e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-20 19:30:23,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1191000.0, ans=0.0 2023-11-20 19:30:39,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1191066.6666666667, ans=0.125 2023-11-20 19:30:42,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1191133.3333333333, ans=0.125 2023-11-20 19:30:55,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1191200.0, ans=0.125 2023-11-20 19:30:55,971 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10350, loss[loss=0.08011, simple_loss=0.09986, pruned_loss=0.01762, audio_tagging_loss=0.01256, over 14861.00 frames. ], tot_loss[loss=0.07845, simple_loss=0.09924, pruned_loss=0.0187, audio_tagging_loss=0.01014, over 3047578.72 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 16.0 2023-11-20 19:31:08,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1191266.6666666667, ans=6.0 2023-11-20 19:31:10,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1191266.6666666667, ans=0.0 2023-11-20 19:31:19,178 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178700 2023-11-20 19:31:20,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1191333.3333333333, ans=0.125 2023-11-20 19:31:21,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.04 vs. limit=10.0 2023-11-20 19:31:43,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1191400.0, ans=0.2 2023-11-20 19:31:59,608 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10400, loss[loss=0.0967, simple_loss=0.1185, pruned_loss=0.0284, audio_tagging_loss=0.009063, over 15823.00 frames. ], tot_loss[loss=0.07852, simple_loss=0.09921, pruned_loss=0.01876, audio_tagging_loss=0.01016, over 3045767.86 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:32:02,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1191533.3333333333, ans=0.0 2023-11-20 19:32:23,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178750 2023-11-20 19:32:30,360 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.343e+01 9.051e+01 9.727e+01 1.230e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-20 19:32:39,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=1191733.3333333333, ans=15.0 2023-11-20 19:32:40,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1191733.3333333333, ans=0.04949747468305833 2023-11-20 19:32:55,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1191800.0, ans=0.125 2023-11-20 19:33:03,412 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10450, loss[loss=0.1063, simple_loss=0.1322, pruned_loss=0.03308, audio_tagging_loss=0.007087, over 15855.00 frames. ], tot_loss[loss=0.07854, simple_loss=0.09913, pruned_loss=0.0189, audio_tagging_loss=0.01008, over 3042742.21 frames. ], batch size: 58, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:33:17,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.10 vs. limit=15.0 2023-11-20 19:33:27,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178800 2023-11-20 19:33:34,352 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=15.0 2023-11-20 19:33:51,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1192066.6666666667, ans=0.0 2023-11-20 19:34:02,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1192133.3333333333, ans=0.025 2023-11-20 19:34:08,051 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10500, loss[loss=0.0717, simple_loss=0.09704, pruned_loss=0.01261, audio_tagging_loss=0.01057, over 15162.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09879, pruned_loss=0.01876, audio_tagging_loss=0.01, over 3036147.09 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:34:30,410 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178850 2023-11-20 19:34:30,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1192266.6666666667, ans=10.0 2023-11-20 19:34:37,988 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.449e+01 8.858e+01 9.853e+01 1.185e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-20 19:34:43,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1192333.3333333333, ans=0.05 2023-11-20 19:34:45,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1192400.0, ans=0.125 2023-11-20 19:34:48,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1192400.0, ans=0.035 2023-11-20 19:34:54,383 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.29 vs. limit=15.0 2023-11-20 19:35:07,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1192466.6666666667, ans=0.1 2023-11-20 19:35:07,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1192466.6666666667, ans=0.0 2023-11-20 19:35:11,107 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10550, loss[loss=0.0822, simple_loss=0.1027, pruned_loss=0.02088, audio_tagging_loss=0.009976, over 14905.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09855, pruned_loss=0.01862, audio_tagging_loss=0.009891, over 3040220.98 frames. ], batch size: 54, lr: 4.54e-03, grad_scale: 32.0 2023-11-20 19:35:33,718 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178900 2023-11-20 19:35:41,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1192666.6666666667, ans=0.2 2023-11-20 19:35:53,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1192733.3333333333, ans=0.07 2023-11-20 19:36:14,504 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10600, loss[loss=0.08401, simple_loss=0.103, pruned_loss=0.02294, audio_tagging_loss=0.009584, over 15456.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09844, pruned_loss=0.01862, audio_tagging_loss=0.009911, over 3045843.11 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 16.0 2023-11-20 19:36:25,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1192866.6666666667, ans=0.125 2023-11-20 19:36:30,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.77 vs. limit=15.0 2023-11-20 19:36:34,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1192933.3333333333, ans=0.125 2023-11-20 19:36:38,424 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 178950 2023-11-20 19:36:41,401 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0 2023-11-20 19:36:46,939 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.146e+01 8.630e+01 9.635e+01 1.232e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-20 19:37:14,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1193133.3333333333, ans=0.0 2023-11-20 19:37:18,824 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10650, loss[loss=0.06116, simple_loss=0.08068, pruned_loss=0.01088, audio_tagging_loss=0.009938, over 14278.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.09759, pruned_loss=0.01836, audio_tagging_loss=0.009911, over 3039221.44 frames. ], batch size: 53, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:37:22,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1193200.0, ans=0.125 2023-11-20 19:37:25,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0 2023-11-20 19:37:31,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1193266.6666666667, ans=0.0 2023-11-20 19:37:38,356 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.64 vs. limit=6.0 2023-11-20 19:37:41,206 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179000 2023-11-20 19:37:41,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1193266.6666666667, ans=0.1 2023-11-20 19:37:44,478 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.96 vs. limit=15.0 2023-11-20 19:37:46,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.22 vs. limit=22.5 2023-11-20 19:37:54,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1193333.3333333333, ans=0.125 2023-11-20 19:38:08,012 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.13 vs. limit=22.5 2023-11-20 19:38:14,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1193466.6666666667, ans=0.0 2023-11-20 19:38:22,588 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10700, loss[loss=0.09162, simple_loss=0.1222, pruned_loss=0.01974, audio_tagging_loss=0.01077, over 16173.00 frames. ], tot_loss[loss=0.07707, simple_loss=0.09759, pruned_loss=0.01831, audio_tagging_loss=0.009965, over 3042866.45 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:38:32,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1193533.3333333333, ans=0.1 2023-11-20 19:38:39,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1193600.0, ans=0.1 2023-11-20 19:38:44,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1193600.0, ans=0.125 2023-11-20 19:38:45,331 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179050 2023-11-20 19:38:54,225 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.026e+01 8.604e+01 9.471e+01 1.335e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-20 19:38:54,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1193666.6666666667, ans=0.125 2023-11-20 19:39:00,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0 2023-11-20 19:39:06,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1193733.3333333333, ans=0.125 2023-11-20 19:39:25,050 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10750, loss[loss=0.05831, simple_loss=0.07817, pruned_loss=0.01195, audio_tagging_loss=0.007273, over 15810.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09602, pruned_loss=0.01789, audio_tagging_loss=0.01003, over 3050657.47 frames. ], batch size: 59, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:39:43,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1193933.3333333333, ans=0.125 2023-11-20 19:39:47,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179100 2023-11-20 19:40:10,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.21 vs. limit=12.0 2023-11-20 19:40:28,488 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10800, loss[loss=0.08748, simple_loss=0.1188, pruned_loss=0.01932, audio_tagging_loss=0.00874, over 15206.00 frames. ], tot_loss[loss=0.07682, simple_loss=0.09731, pruned_loss=0.0182, audio_tagging_loss=0.009964, over 3049021.12 frames. ], batch size: 55, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:40:29,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1194200.0, ans=0.1 2023-11-20 19:40:37,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1194200.0, ans=0.125 2023-11-20 19:40:47,704 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.91 vs. limit=15.0 2023-11-20 19:40:50,807 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179150 2023-11-20 19:41:00,341 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.337e+01 8.834e+01 9.923e+01 1.321e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-20 19:41:15,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1194400.0, ans=10.0 2023-11-20 19:41:16,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1194400.0, ans=0.1 2023-11-20 19:41:31,483 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10850, loss[loss=0.08039, simple_loss=0.1074, pruned_loss=0.01947, audio_tagging_loss=0.00723, over 15811.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09775, pruned_loss=0.01817, audio_tagging_loss=0.009875, over 3051915.36 frames. ], batch size: 59, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:41:36,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1194533.3333333333, ans=0.0 2023-11-20 19:41:41,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1194533.3333333333, ans=0.025 2023-11-20 19:41:42,197 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.11 vs. limit=10.0 2023-11-20 19:41:53,636 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179200 2023-11-20 19:41:54,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.51 vs. limit=10.0 2023-11-20 19:42:26,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1194800.0, ans=0.125 2023-11-20 19:42:30,668 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:42:34,256 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10900, loss[loss=0.07206, simple_loss=0.09373, pruned_loss=0.0176, audio_tagging_loss=0.007586, over 15403.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09825, pruned_loss=0.01845, audio_tagging_loss=0.009871, over 3052537.27 frames. ], batch size: 55, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:42:34,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1194866.6666666667, ans=0.1 2023-11-20 19:42:58,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179250 2023-11-20 19:43:08,255 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.081e+01 8.823e+01 9.357e+01 1.279e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-20 19:43:08,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1195000.0, ans=0.125 2023-11-20 19:43:21,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1195066.6666666667, ans=0.125 2023-11-20 19:43:25,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1195133.3333333333, ans=0.125 2023-11-20 19:43:38,315 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 10950, loss[loss=0.06696, simple_loss=0.08202, pruned_loss=0.0142, audio_tagging_loss=0.01174, over 15054.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.0981, pruned_loss=0.01832, audio_tagging_loss=0.009932, over 3052219.02 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:43:38,802 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:43:57,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.21 vs. limit=22.5 2023-11-20 19:44:01,765 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179300 2023-11-20 19:44:09,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1195333.3333333333, ans=0.0 2023-11-20 19:44:16,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1195400.0, ans=0.125 2023-11-20 19:44:17,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1195400.0, ans=0.1 2023-11-20 19:44:25,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1195400.0, ans=0.5 2023-11-20 19:44:32,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1195466.6666666667, ans=0.2 2023-11-20 19:44:41,968 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=4.116e-01 2023-11-20 19:44:42,830 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11000, loss[loss=0.05594, simple_loss=0.06885, pruned_loss=0.008563, audio_tagging_loss=0.01295, over 15245.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09887, pruned_loss=0.0185, audio_tagging_loss=0.009963, over 3055653.04 frames. ], batch size: 58, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:44:46,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1195533.3333333333, ans=0.0 2023-11-20 19:44:51,290 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:45:04,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179350 2023-11-20 19:45:15,401 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.295e+01 8.189e+01 8.881e+01 9.768e+01 1.271e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-20 19:45:18,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1195666.6666666667, ans=0.2 2023-11-20 19:45:23,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1195733.3333333333, ans=0.125 2023-11-20 19:45:31,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1195733.3333333333, ans=0.0 2023-11-20 19:45:35,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1195800.0, ans=0.1 2023-11-20 19:45:45,437 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11050, loss[loss=0.08772, simple_loss=0.1112, pruned_loss=0.02257, audio_tagging_loss=0.009532, over 16723.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09915, pruned_loss=0.01867, audio_tagging_loss=0.009977, over 3057638.70 frames. ], batch size: 60, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:46:06,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1195933.3333333333, ans=0.125 2023-11-20 19:46:06,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1195933.3333333333, ans=0.125 2023-11-20 19:46:07,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179400 2023-11-20 19:46:12,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1196000.0, ans=0.125 2023-11-20 19:46:47,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1196200.0, ans=0.1 2023-11-20 19:46:48,002 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11100, loss[loss=0.06817, simple_loss=0.09222, pruned_loss=0.01026, audio_tagging_loss=0.0118, over 15445.00 frames. ], tot_loss[loss=0.07821, simple_loss=0.09915, pruned_loss=0.01864, audio_tagging_loss=0.009992, over 3055428.74 frames. ], batch size: 59, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:46:52,564 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:47:11,878 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179450 2023-11-20 19:47:20,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1196333.3333333333, ans=0.1 2023-11-20 19:47:21,549 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.315e+01 8.689e+01 9.550e+01 1.347e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-20 19:47:27,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.35 vs. limit=15.0 2023-11-20 19:47:43,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1196466.6666666667, ans=0.5 2023-11-20 19:47:52,013 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11150, loss[loss=0.07668, simple_loss=0.108, pruned_loss=0.0139, audio_tagging_loss=0.00879, over 14540.00 frames. ], tot_loss[loss=0.07839, simple_loss=0.09934, pruned_loss=0.0186, audio_tagging_loss=0.01011, over 3051123.57 frames. ], batch size: 54, lr: 4.53e-03, grad_scale: 16.0 2023-11-20 19:48:13,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179500 2023-11-20 19:48:14,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1196666.6666666667, ans=0.125 2023-11-20 19:48:53,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.77 vs. limit=22.5 2023-11-20 19:48:54,080 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11200, loss[loss=0.1147, simple_loss=0.1508, pruned_loss=0.0332, audio_tagging_loss=0.006132, over 15945.00 frames. ], tot_loss[loss=0.07849, simple_loss=0.09935, pruned_loss=0.01861, audio_tagging_loss=0.01021, over 3049890.06 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 32.0 2023-11-20 19:49:01,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1196866.6666666667, ans=0.125 2023-11-20 19:49:16,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179550 2023-11-20 19:49:19,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1197000.0, ans=0.125 2023-11-20 19:49:25,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1197000.0, ans=0.125 2023-11-20 19:49:26,748 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.291e+01 7.963e+01 8.549e+01 9.018e+01 1.277e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-20 19:49:36,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1197066.6666666667, ans=0.0 2023-11-20 19:49:47,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1197133.3333333333, ans=0.125 2023-11-20 19:49:53,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1197133.3333333333, ans=0.1 2023-11-20 19:49:56,614 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11250, loss[loss=0.1037, simple_loss=0.1422, pruned_loss=0.02428, audio_tagging_loss=0.00831, over 15924.00 frames. ], tot_loss[loss=0.07778, simple_loss=0.09841, pruned_loss=0.01845, audio_tagging_loss=0.01013, over 3051096.05 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 32.0 2023-11-20 19:49:58,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1197200.0, ans=0.1 2023-11-20 19:50:02,831 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.99 vs. limit=15.0 2023-11-20 19:50:04,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1197200.0, ans=0.0 2023-11-20 19:50:13,855 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0 2023-11-20 19:50:20,356 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179600 2023-11-20 19:50:33,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1197333.3333333333, ans=0.125 2023-11-20 19:50:35,462 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:50:47,585 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:50:52,023 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.46 vs. limit=6.0 2023-11-20 19:50:59,724 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11300, loss[loss=0.07893, simple_loss=0.101, pruned_loss=0.02016, audio_tagging_loss=0.008284, over 14300.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09868, pruned_loss=0.01853, audio_tagging_loss=0.009942, over 3050760.38 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 32.0 2023-11-20 19:51:15,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1197600.0, ans=0.125 2023-11-20 19:51:22,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179650 2023-11-20 19:51:24,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1197666.6666666667, ans=0.0 2023-11-20 19:51:26,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1197666.6666666667, ans=0.1 2023-11-20 19:51:28,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1197666.6666666667, ans=0.0 2023-11-20 19:51:32,421 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.320e+01 8.034e+01 8.460e+01 9.162e+01 1.233e+02, threshold=1.692e+02, percent-clipped=0.0 2023-11-20 19:51:40,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1197733.3333333333, ans=0.1 2023-11-20 19:52:00,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1197800.0, ans=0.04949747468305833 2023-11-20 19:52:03,243 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11350, loss[loss=0.06341, simple_loss=0.07716, pruned_loss=0.01525, audio_tagging_loss=0.009577, over 15650.00 frames. ], tot_loss[loss=0.07871, simple_loss=0.1001, pruned_loss=0.01885, audio_tagging_loss=0.009796, over 3053704.70 frames. ], batch size: 61, lr: 4.53e-03, grad_scale: 32.0 2023-11-20 19:52:05,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1197866.6666666667, ans=0.2 2023-11-20 19:52:08,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1197866.6666666667, ans=0.125 2023-11-20 19:52:26,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179700 2023-11-20 19:52:27,977 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.13 vs. limit=12.0 2023-11-20 19:52:35,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.71 vs. limit=15.0 2023-11-20 19:52:54,600 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.37 vs. limit=22.5 2023-11-20 19:53:04,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1198200.0, ans=0.1 2023-11-20 19:53:05,856 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11400, loss[loss=0.0528, simple_loss=0.06071, pruned_loss=0.01437, audio_tagging_loss=0.008068, over 15100.00 frames. ], tot_loss[loss=0.07826, simple_loss=0.09934, pruned_loss=0.01873, audio_tagging_loss=0.009849, over 3051841.75 frames. ], batch size: 58, lr: 4.53e-03, grad_scale: 32.0 2023-11-20 19:53:10,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1198200.0, ans=0.05 2023-11-20 19:53:17,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1198200.0, ans=0.0 2023-11-20 19:53:17,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1198200.0, ans=0.125 2023-11-20 19:53:29,511 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179750 2023-11-20 19:53:37,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1198333.3333333333, ans=0.125 2023-11-20 19:53:39,664 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.357e+01 7.998e+01 8.618e+01 9.401e+01 1.167e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-20 19:53:44,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1198400.0, ans=0.125 2023-11-20 19:53:45,358 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.31 vs. limit=15.0 2023-11-20 19:54:09,902 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11450, loss[loss=0.06647, simple_loss=0.08207, pruned_loss=0.01556, audio_tagging_loss=0.009874, over 14990.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09868, pruned_loss=0.01851, audio_tagging_loss=0.009795, over 3042376.34 frames. ], batch size: 55, lr: 4.52e-03, grad_scale: 32.0 2023-11-20 19:54:16,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1198533.3333333333, ans=0.125 2023-11-20 19:54:25,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=1198600.0, ans=0.02 2023-11-20 19:54:32,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179800 2023-11-20 19:54:38,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.18 vs. limit=10.0 2023-11-20 19:54:57,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1198733.3333333333, ans=0.2 2023-11-20 19:55:07,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1198800.0, ans=0.0 2023-11-20 19:55:13,637 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11500, loss[loss=0.07901, simple_loss=0.1009, pruned_loss=0.02057, audio_tagging_loss=0.007976, over 14932.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.09813, pruned_loss=0.01836, audio_tagging_loss=0.009772, over 3044692.64 frames. ], batch size: 56, lr: 4.52e-03, grad_scale: 32.0 2023-11-20 19:55:23,302 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.39 vs. limit=22.5 2023-11-20 19:55:36,259 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179850 2023-11-20 19:55:43,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1199000.0, ans=0.04949747468305833 2023-11-20 19:55:45,014 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 19:55:47,196 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.150e+01 8.270e+01 9.064e+01 9.843e+01 1.286e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-20 19:56:06,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1199133.3333333333, ans=0.125 2023-11-20 19:56:17,609 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11550, loss[loss=0.06294, simple_loss=0.0749, pruned_loss=0.01499, audio_tagging_loss=0.01051, over 15060.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.09909, pruned_loss=0.01864, audio_tagging_loss=0.009755, over 3049769.62 frames. ], batch size: 56, lr: 4.52e-03, grad_scale: 32.0 2023-11-20 19:56:20,751 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.82 vs. limit=12.0 2023-11-20 19:56:24,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1199200.0, ans=0.0 2023-11-20 19:56:37,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1199266.6666666667, ans=0.2 2023-11-20 19:56:41,390 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179900 2023-11-20 19:56:44,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1199333.3333333333, ans=0.09899494936611666 2023-11-20 19:56:44,326 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.90 vs. limit=22.5 2023-11-20 19:56:56,142 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 19:57:00,585 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.84 vs. limit=22.5 2023-11-20 19:57:04,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1199400.0, ans=0.125 2023-11-20 19:57:06,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1199400.0, ans=0.0 2023-11-20 19:57:10,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1199466.6666666667, ans=0.125 2023-11-20 19:57:11,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1199466.6666666667, ans=0.09899494936611666 2023-11-20 19:57:19,530 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.47 vs. limit=10.0 2023-11-20 19:57:22,573 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11600, loss[loss=0.07628, simple_loss=0.08628, pruned_loss=0.01902, audio_tagging_loss=0.01412, over 14135.00 frames. ], tot_loss[loss=0.0784, simple_loss=0.09982, pruned_loss=0.01879, audio_tagging_loss=0.009693, over 3048096.94 frames. ], batch size: 54, lr: 4.52e-03, grad_scale: 32.0 2023-11-20 19:57:37,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1199600.0, ans=0.0 2023-11-20 19:57:40,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1199600.0, ans=0.125 2023-11-20 19:57:44,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 179950 2023-11-20 19:57:54,838 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.42 vs. limit=22.5 2023-11-20 19:57:55,284 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.413e+01 9.073e+01 1.016e+02 1.405e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-20 19:57:55,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1199666.6666666667, ans=0.0 2023-11-20 19:58:00,937 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=23.62 vs. limit=22.5 2023-11-20 19:58:02,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.53 vs. limit=22.5 2023-11-20 19:58:20,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1199800.0, ans=0.1 2023-11-20 19:58:26,088 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11650, loss[loss=0.08082, simple_loss=0.106, pruned_loss=0.01662, audio_tagging_loss=0.0112, over 15651.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09868, pruned_loss=0.01855, audio_tagging_loss=0.0098, over 3047476.19 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 32.0 2023-11-20 19:58:32,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1199866.6666666667, ans=0.1 2023-11-20 19:58:38,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.03 vs. limit=10.0 2023-11-20 19:58:48,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180000 2023-11-20 19:59:02,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1200000.0, ans=0.2 2023-11-20 19:59:03,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1200000.0, ans=0.0 2023-11-20 19:59:24,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1200133.3333333333, ans=0.0 2023-11-20 19:59:26,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1200133.3333333333, ans=0.0 2023-11-20 19:59:28,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1200133.3333333333, ans=0.0 2023-11-20 19:59:32,099 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11700, loss[loss=0.06239, simple_loss=0.07965, pruned_loss=0.01362, audio_tagging_loss=0.008948, over 13927.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.09849, pruned_loss=0.01833, audio_tagging_loss=0.009788, over 3050474.09 frames. ], batch size: 55, lr: 4.52e-03, grad_scale: 32.0 2023-11-20 19:59:51,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1200266.6666666667, ans=0.1 2023-11-20 19:59:55,370 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180050 2023-11-20 20:00:06,809 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.544e+01 8.114e+01 8.755e+01 9.534e+01 1.280e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-20 20:00:29,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1200466.6666666667, ans=0.0 2023-11-20 20:00:35,967 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11750, loss[loss=0.1092, simple_loss=0.1425, pruned_loss=0.02898, audio_tagging_loss=0.009011, over 15507.00 frames. ], tot_loss[loss=0.07829, simple_loss=0.09996, pruned_loss=0.01861, audio_tagging_loss=0.009695, over 3054889.33 frames. ], batch size: 55, lr: 4.52e-03, grad_scale: 16.0 2023-11-20 20:00:51,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1200600.0, ans=0.125 2023-11-20 20:00:58,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180100 2023-11-20 20:01:12,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1200666.6666666667, ans=0.125 2023-11-20 20:01:37,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.whiten.whitening_limit, batch_count=1200800.0, ans=12.0 2023-11-20 20:01:40,521 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11800, loss[loss=0.1008, simple_loss=0.1208, pruned_loss=0.03042, audio_tagging_loss=0.009955, over 15085.00 frames. ], tot_loss[loss=0.07814, simple_loss=0.09962, pruned_loss=0.01857, audio_tagging_loss=0.009753, over 3050437.35 frames. ], batch size: 57, lr: 4.52e-03, grad_scale: 16.0 2023-11-20 20:02:03,362 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180150 2023-11-20 20:02:15,383 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.209e+01 9.058e+01 9.835e+01 1.362e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-20 20:02:43,902 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11850, loss[loss=0.0807, simple_loss=0.1024, pruned_loss=0.01983, audio_tagging_loss=0.009687, over 14152.00 frames. ], tot_loss[loss=0.07801, simple_loss=0.09936, pruned_loss=0.01849, audio_tagging_loss=0.009846, over 3049579.57 frames. ], batch size: 54, lr: 4.52e-03, grad_scale: 16.0 2023-11-20 20:02:49,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1201200.0, ans=0.2 2023-11-20 20:02:50,712 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0 2023-11-20 20:02:53,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1201200.0, ans=0.125 2023-11-20 20:03:07,776 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180200 2023-11-20 20:03:18,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.85 vs. limit=15.0 2023-11-20 20:03:22,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.60 vs. limit=22.5 2023-11-20 20:03:44,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1201466.6666666667, ans=0.1 2023-11-20 20:03:48,549 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11900, loss[loss=0.07152, simple_loss=0.08033, pruned_loss=0.01804, audio_tagging_loss=0.01331, over 14403.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09932, pruned_loss=0.01851, audio_tagging_loss=0.009938, over 3048440.57 frames. ], batch size: 56, lr: 4.52e-03, grad_scale: 16.0 2023-11-20 20:03:48,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1201533.3333333333, ans=0.125 2023-11-20 20:04:11,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180250 2023-11-20 20:04:22,706 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.009e+01 8.683e+01 9.562e+01 2.232e+02, threshold=1.737e+02, percent-clipped=1.0 2023-11-20 20:04:25,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1201733.3333333333, ans=0.125 2023-11-20 20:04:32,465 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.48 vs. limit=6.0 2023-11-20 20:04:40,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1201800.0, ans=0.125 2023-11-20 20:04:40,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.74 vs. limit=12.0 2023-11-20 20:04:43,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1201800.0, ans=0.125 2023-11-20 20:04:52,339 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 11950, loss[loss=0.06282, simple_loss=0.08361, pruned_loss=0.01225, audio_tagging_loss=0.00876, over 14067.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.09894, pruned_loss=0.01834, audio_tagging_loss=0.01008, over 3046541.57 frames. ], batch size: 54, lr: 4.52e-03, grad_scale: 16.0 2023-11-20 20:05:14,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180300 2023-11-20 20:05:52,621 INFO [train_asr.py:1221] (3/4) Epoch 15, batch 12000, loss[loss=0.06119, simple_loss=0.07954, pruned_loss=0.0132, audio_tagging_loss=0.008221, over 15039.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09843, pruned_loss=0.01827, audio_tagging_loss=0.01011, over 3045989.33 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 32.0 2023-11-20 20:05:52,621 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 20:06:18,689 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.3190, 3.1111, 2.8086, 2.7600, 3.4769, 3.5484, 3.1427, 3.6134], device='cuda:3') 2023-11-20 20:06:26,051 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4146, 3.6619, 2.6397, 3.4437], device='cuda:3') 2023-11-20 20:06:33,117 INFO [train_asr.py:1253] (3/4) Epoch 15, validation: loss=0.06134, simple_loss=0.05315, pruned_loss=0.00551, audio_tagging_loss=0.02926, over 4681554.00 frames. 2023-11-20 20:06:33,118 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 20:06:33,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.20 vs. limit=15.0 2023-11-20 20:06:53,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1202266.6666666667, ans=0.0 2023-11-20 20:06:54,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0 2023-11-20 20:06:54,594 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180350 2023-11-20 20:06:59,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1202333.3333333333, ans=0.125 2023-11-20 20:07:37,549 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 0, loss[loss=0.09226, simple_loss=0.1137, pruned_loss=0.01491, audio_tagging_loss=0.02049, over 15729.00 frames. ], tot_loss[loss=0.09226, simple_loss=0.1137, pruned_loss=0.01491, audio_tagging_loss=0.02049, over 15729.00 frames. ], batch size: 59, lr: 4.37e-03, grad_scale: 32.0 2023-11-20 20:07:37,550 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 20:08:13,619 INFO [train_asr.py:1253] (3/4) Epoch 16, validation: loss=0.06129, simple_loss=0.0532, pruned_loss=0.005566, audio_tagging_loss=0.02913, over 4681554.00 frames. 2023-11-20 20:08:13,620 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 20:08:17,303 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.282e+01 8.389e+01 9.121e+01 1.002e+02 1.446e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-20 20:08:25,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1202426.6666666667, ans=0.125 2023-11-20 20:09:04,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1202626.6666666667, ans=0.0 2023-11-20 20:09:11,330 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180400 2023-11-20 20:09:19,015 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 50, loss[loss=0.08608, simple_loss=0.09235, pruned_loss=0.01855, audio_tagging_loss=0.02135, over 13844.00 frames. ], tot_loss[loss=0.08685, simple_loss=0.0993, pruned_loss=0.0184, audio_tagging_loss=0.0188, over 688453.59 frames. ], batch size: 53, lr: 4.37e-03, grad_scale: 16.0 2023-11-20 20:10:16,043 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180450 2023-11-20 20:10:23,097 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.22 vs. limit=15.0 2023-11-20 20:10:23,394 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 100, loss[loss=0.07143, simple_loss=0.09127, pruned_loss=0.01254, audio_tagging_loss=0.01326, over 15753.00 frames. ], tot_loss[loss=0.08612, simple_loss=0.0984, pruned_loss=0.01873, audio_tagging_loss=0.0182, over 1211751.64 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 16.0 2023-11-20 20:10:29,504 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.348e+01 8.922e+01 9.752e+01 1.060e+02 1.405e+02, threshold=1.950e+02, percent-clipped=0.0 2023-11-20 20:10:33,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1203026.6666666667, ans=0.125 2023-11-20 20:10:34,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1203026.6666666667, ans=0.0 2023-11-20 20:10:43,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1203093.3333333333, ans=0.2 2023-11-20 20:11:08,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.75 vs. limit=22.5 2023-11-20 20:11:21,280 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180500 2023-11-20 20:11:29,071 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 150, loss[loss=0.06576, simple_loss=0.07377, pruned_loss=0.01334, audio_tagging_loss=0.01553, over 14657.00 frames. ], tot_loss[loss=0.08418, simple_loss=0.09814, pruned_loss=0.0187, audio_tagging_loss=0.01641, over 1621790.55 frames. ], batch size: 55, lr: 4.37e-03, grad_scale: 8.0 2023-11-20 20:11:33,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1203360.0, ans=0.2 2023-11-20 20:11:36,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1203360.0, ans=0.0 2023-11-20 20:11:41,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.08 vs. limit=15.0 2023-11-20 20:11:49,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1203426.6666666667, ans=0.5 2023-11-20 20:12:04,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1203493.3333333333, ans=0.125 2023-11-20 20:12:13,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1203560.0, ans=0.1 2023-11-20 20:12:14,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=15.0 2023-11-20 20:12:24,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1203626.6666666667, ans=0.125 2023-11-20 20:12:25,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180550 2023-11-20 20:12:27,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.06 vs. limit=8.0 2023-11-20 20:12:31,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1203626.6666666667, ans=0.125 2023-11-20 20:12:33,422 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 200, loss[loss=0.07256, simple_loss=0.09396, pruned_loss=0.01789, audio_tagging_loss=0.007694, over 15768.00 frames. ], tot_loss[loss=0.08218, simple_loss=0.09831, pruned_loss=0.01851, audio_tagging_loss=0.01451, over 1943572.67 frames. ], batch size: 61, lr: 4.37e-03, grad_scale: 8.0 2023-11-20 20:12:39,630 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.954e+01 8.158e+01 8.825e+01 9.398e+01 1.176e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-20 20:12:51,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1203760.0, ans=0.125 2023-11-20 20:13:17,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1203893.3333333333, ans=0.1 2023-11-20 20:13:21,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1203893.3333333333, ans=0.125 2023-11-20 20:13:29,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180600 2023-11-20 20:13:31,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1203960.0, ans=0.0 2023-11-20 20:13:35,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1203960.0, ans=0.125 2023-11-20 20:13:37,508 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 250, loss[loss=0.07536, simple_loss=0.08392, pruned_loss=0.01544, audio_tagging_loss=0.01796, over 15369.00 frames. ], tot_loss[loss=0.08134, simple_loss=0.09898, pruned_loss=0.01871, audio_tagging_loss=0.01313, over 2191540.18 frames. ], batch size: 58, lr: 4.37e-03, grad_scale: 8.0 2023-11-20 20:13:45,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1204026.6666666667, ans=0.125 2023-11-20 20:13:52,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1204093.3333333333, ans=0.125 2023-11-20 20:13:53,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1204093.3333333333, ans=0.125 2023-11-20 20:14:04,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1204160.0, ans=0.125 2023-11-20 20:14:22,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1204226.6666666667, ans=0.0 2023-11-20 20:14:27,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1204293.3333333333, ans=0.1 2023-11-20 20:14:35,077 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180650 2023-11-20 20:14:42,349 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 300, loss[loss=0.08433, simple_loss=0.1087, pruned_loss=0.02034, audio_tagging_loss=0.009639, over 14520.00 frames. ], tot_loss[loss=0.08116, simple_loss=0.09998, pruned_loss=0.0189, audio_tagging_loss=0.01227, over 2379771.11 frames. ], batch size: 56, lr: 4.37e-03, grad_scale: 8.0 2023-11-20 20:14:45,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=12.0 2023-11-20 20:14:48,972 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.454e+01 9.235e+01 1.018e+02 1.447e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-20 20:15:08,321 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:15:25,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1204560.0, ans=0.125 2023-11-20 20:15:39,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180700 2023-11-20 20:15:45,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=1204693.3333333333, ans=0.02 2023-11-20 20:15:46,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.30 vs. limit=15.0 2023-11-20 20:15:46,882 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 350, loss[loss=0.06297, simple_loss=0.08047, pruned_loss=0.01276, audio_tagging_loss=0.009977, over 15282.00 frames. ], tot_loss[loss=0.08026, simple_loss=0.09975, pruned_loss=0.01875, audio_tagging_loss=0.01164, over 2531998.59 frames. ], batch size: 58, lr: 4.37e-03, grad_scale: 8.0 2023-11-20 20:16:20,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1204826.6666666667, ans=0.125 2023-11-20 20:16:36,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1204893.3333333333, ans=0.125 2023-11-20 20:16:39,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1204960.0, ans=0.5 2023-11-20 20:16:44,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180750 2023-11-20 20:16:51,457 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 400, loss[loss=0.09682, simple_loss=0.1275, pruned_loss=0.02694, audio_tagging_loss=0.006141, over 15264.00 frames. ], tot_loss[loss=0.07949, simple_loss=0.09964, pruned_loss=0.01848, audio_tagging_loss=0.01119, over 2652143.27 frames. ], batch size: 55, lr: 4.37e-03, grad_scale: 16.0 2023-11-20 20:16:58,138 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.022e+01 8.670e+01 9.315e+01 1.277e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-20 20:17:05,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=15.0 2023-11-20 20:17:09,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1205093.3333333333, ans=0.2 2023-11-20 20:17:09,952 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:17:13,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1205093.3333333333, ans=0.0 2023-11-20 20:17:14,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1205093.3333333333, ans=0.125 2023-11-20 20:17:21,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1205160.0, ans=0.125 2023-11-20 20:17:23,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1205160.0, ans=0.1 2023-11-20 20:17:48,895 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180800 2023-11-20 20:17:56,393 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 450, loss[loss=0.0816, simple_loss=0.09824, pruned_loss=0.02115, audio_tagging_loss=0.01133, over 14969.00 frames. ], tot_loss[loss=0.07929, simple_loss=0.09992, pruned_loss=0.01854, audio_tagging_loss=0.01079, over 2745243.59 frames. ], batch size: 56, lr: 4.37e-03, grad_scale: 16.0 2023-11-20 20:18:14,233 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.45 vs. limit=22.5 2023-11-20 20:18:33,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1205560.0, ans=0.125 2023-11-20 20:18:52,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180850 2023-11-20 20:18:53,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0 2023-11-20 20:18:58,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1205693.3333333333, ans=0.125 2023-11-20 20:18:59,874 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 500, loss[loss=0.08009, simple_loss=0.104, pruned_loss=0.01816, audio_tagging_loss=0.009914, over 16044.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.09816, pruned_loss=0.01817, audio_tagging_loss=0.01062, over 2812755.84 frames. ], batch size: 59, lr: 4.37e-03, grad_scale: 16.0 2023-11-20 20:19:07,188 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.820e+01 8.156e+01 8.896e+01 9.725e+01 1.312e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-20 20:19:12,813 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.55 vs. limit=15.0 2023-11-20 20:19:54,431 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:19:56,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180900 2023-11-20 20:20:03,754 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 550, loss[loss=0.07196, simple_loss=0.0889, pruned_loss=0.01791, audio_tagging_loss=0.009603, over 14293.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09813, pruned_loss=0.01814, audio_tagging_loss=0.01042, over 2859793.22 frames. ], batch size: 53, lr: 4.37e-03, grad_scale: 16.0 2023-11-20 20:20:03,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1206026.6666666667, ans=0.125 2023-11-20 20:20:17,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1206093.3333333333, ans=0.125 2023-11-20 20:20:22,340 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.68 vs. limit=15.0 2023-11-20 20:20:29,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1206160.0, ans=0.2 2023-11-20 20:20:37,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1206160.0, ans=0.2 2023-11-20 20:20:39,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1206160.0, ans=10.0 2023-11-20 20:20:46,800 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.59 vs. limit=15.0 2023-11-20 20:20:50,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1206226.6666666667, ans=0.125 2023-11-20 20:20:59,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 180950 2023-11-20 20:21:04,249 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.05 vs. limit=22.5 2023-11-20 20:21:07,313 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 600, loss[loss=0.07363, simple_loss=0.09727, pruned_loss=0.01729, audio_tagging_loss=0.007705, over 15320.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09829, pruned_loss=0.01821, audio_tagging_loss=0.01034, over 2900494.20 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 16.0 2023-11-20 20:21:11,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1206360.0, ans=0.035 2023-11-20 20:21:13,340 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.725e+01 8.009e+01 8.820e+01 9.311e+01 1.105e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-20 20:21:41,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1206493.3333333333, ans=0.125 2023-11-20 20:21:50,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1206560.0, ans=0.0 2023-11-20 20:21:55,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1206560.0, ans=0.125 2023-11-20 20:22:02,627 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181000 2023-11-20 20:22:02,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1206626.6666666667, ans=0.125 2023-11-20 20:22:10,125 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 650, loss[loss=0.06361, simple_loss=0.07755, pruned_loss=0.01253, audio_tagging_loss=0.01231, over 14939.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.09765, pruned_loss=0.01812, audio_tagging_loss=0.01024, over 2921605.11 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:22:46,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1206826.6666666667, ans=0.1 2023-11-20 20:23:06,208 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181050 2023-11-20 20:23:08,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1206960.0, ans=0.125 2023-11-20 20:23:14,271 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 700, loss[loss=0.09248, simple_loss=0.1233, pruned_loss=0.02369, audio_tagging_loss=0.007157, over 15120.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09824, pruned_loss=0.01815, audio_tagging_loss=0.01025, over 2947242.08 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:23:14,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1207026.6666666667, ans=0.125 2023-11-20 20:23:15,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1207026.6666666667, ans=0.125 2023-11-20 20:23:20,226 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.218e+01 8.797e+01 9.608e+01 1.489e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-20 20:23:29,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1207093.3333333333, ans=0.1 2023-11-20 20:23:35,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1207093.3333333333, ans=0.0 2023-11-20 20:23:57,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1207226.6666666667, ans=0.125 2023-11-20 20:24:01,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1207226.6666666667, ans=0.1 2023-11-20 20:24:06,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1207293.3333333333, ans=0.125 2023-11-20 20:24:06,696 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.57 vs. limit=15.0 2023-11-20 20:24:09,693 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181100 2023-11-20 20:24:17,484 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 750, loss[loss=0.09974, simple_loss=0.1298, pruned_loss=0.02499, audio_tagging_loss=0.009865, over 16146.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09878, pruned_loss=0.01819, audio_tagging_loss=0.01012, over 2971352.37 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:24:19,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1207360.0, ans=0.2 2023-11-20 20:24:40,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1207426.6666666667, ans=0.125 2023-11-20 20:24:56,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1207560.0, ans=0.2 2023-11-20 20:25:13,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181150 2023-11-20 20:25:20,313 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 800, loss[loss=0.07846, simple_loss=0.09699, pruned_loss=0.01938, audio_tagging_loss=0.01059, over 15023.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09844, pruned_loss=0.0182, audio_tagging_loss=0.01009, over 2985933.65 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 32.0 2023-11-20 20:25:26,956 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.116e+01 8.797e+01 9.654e+01 1.312e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-20 20:25:35,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1207760.0, ans=0.125 2023-11-20 20:25:42,367 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:25:45,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1207826.6666666667, ans=0.0 2023-11-20 20:25:47,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1207826.6666666667, ans=0.1 2023-11-20 20:25:54,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=23.43 vs. limit=22.5 2023-11-20 20:26:12,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1207960.0, ans=0.125 2023-11-20 20:26:16,411 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181200 2023-11-20 20:26:23,962 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 850, loss[loss=0.09419, simple_loss=0.1242, pruned_loss=0.01874, audio_tagging_loss=0.01336, over 16227.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.09748, pruned_loss=0.01815, audio_tagging_loss=0.01016, over 2999786.80 frames. ], batch size: 60, lr: 4.36e-03, grad_scale: 32.0 2023-11-20 20:27:15,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1208293.3333333333, ans=0.125 2023-11-20 20:27:20,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181250 2023-11-20 20:27:28,687 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 900, loss[loss=0.06964, simple_loss=0.07684, pruned_loss=0.01845, audio_tagging_loss=0.01277, over 15511.00 frames. ], tot_loss[loss=0.07693, simple_loss=0.09714, pruned_loss=0.0181, audio_tagging_loss=0.01026, over 3016219.30 frames. ], batch size: 61, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:27:35,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.50 vs. limit=15.0 2023-11-20 20:27:35,949 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.347e+01 8.999e+01 9.634e+01 1.332e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-20 20:27:57,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1208493.3333333333, ans=0.1 2023-11-20 20:28:22,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1208626.6666666667, ans=0.125 2023-11-20 20:28:23,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1208626.6666666667, ans=0.125 2023-11-20 20:28:24,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181300 2023-11-20 20:28:31,772 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 950, loss[loss=0.0915, simple_loss=0.1204, pruned_loss=0.02543, audio_tagging_loss=0.005882, over 14804.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.09803, pruned_loss=0.01831, audio_tagging_loss=0.01009, over 3028737.95 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:28:32,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1208693.3333333333, ans=0.2 2023-11-20 20:28:34,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1208693.3333333333, ans=0.125 2023-11-20 20:28:38,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1208693.3333333333, ans=0.125 2023-11-20 20:29:27,800 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181350 2023-11-20 20:29:35,686 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1000, loss[loss=0.05123, simple_loss=0.06139, pruned_loss=0.00849, audio_tagging_loss=0.01205, over 14172.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09821, pruned_loss=0.01841, audio_tagging_loss=0.009951, over 3025437.86 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:29:42,946 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.425e+01 8.384e+01 9.027e+01 9.924e+01 1.224e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-20 20:29:43,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0 2023-11-20 20:29:55,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1209093.3333333333, ans=0.0 2023-11-20 20:30:02,968 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 20:30:03,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1209160.0, ans=0.2 2023-11-20 20:30:04,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0 2023-11-20 20:30:18,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1209226.6666666667, ans=0.125 2023-11-20 20:30:21,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1209226.6666666667, ans=0.0 2023-11-20 20:30:22,848 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.11 vs. limit=22.5 2023-11-20 20:30:24,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.07 vs. limit=15.0 2023-11-20 20:30:31,948 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181400 2023-11-20 20:30:40,294 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1050, loss[loss=0.09016, simple_loss=0.1209, pruned_loss=0.02247, audio_tagging_loss=0.007225, over 15981.00 frames. ], tot_loss[loss=0.07725, simple_loss=0.09798, pruned_loss=0.01837, audio_tagging_loss=0.009888, over 3032487.95 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:30:47,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1209360.0, ans=0.0 2023-11-20 20:30:48,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1209360.0, ans=0.125 2023-11-20 20:30:48,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1209360.0, ans=0.2 2023-11-20 20:31:10,309 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0 2023-11-20 20:31:20,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1209560.0, ans=0.125 2023-11-20 20:31:25,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1209560.0, ans=0.07 2023-11-20 20:31:32,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1209626.6666666667, ans=0.07 2023-11-20 20:31:32,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.73 vs. limit=15.0 2023-11-20 20:31:35,691 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181450 2023-11-20 20:31:42,804 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1100, loss[loss=0.07659, simple_loss=0.09345, pruned_loss=0.01919, audio_tagging_loss=0.01067, over 14748.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09763, pruned_loss=0.01818, audio_tagging_loss=0.009811, over 3037527.14 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:31:45,251 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 20:31:49,904 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.927e+01 7.892e+01 8.572e+01 9.142e+01 1.110e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-20 20:32:17,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1209826.6666666667, ans=0.125 2023-11-20 20:32:20,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1209893.3333333333, ans=0.125 2023-11-20 20:32:23,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1209893.3333333333, ans=0.0 2023-11-20 20:32:36,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1209960.0, ans=0.125 2023-11-20 20:32:38,159 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181500 2023-11-20 20:32:38,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1209960.0, ans=0.05 2023-11-20 20:32:46,233 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1150, loss[loss=0.07784, simple_loss=0.09795, pruned_loss=0.01961, audio_tagging_loss=0.009257, over 15699.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09719, pruned_loss=0.01798, audio_tagging_loss=0.009733, over 3042700.75 frames. ], batch size: 59, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:33:01,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1210093.3333333333, ans=0.05 2023-11-20 20:33:12,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1210160.0, ans=0.125 2023-11-20 20:33:14,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1210160.0, ans=0.125 2023-11-20 20:33:17,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1210160.0, ans=0.1 2023-11-20 20:33:20,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1210160.0, ans=0.0 2023-11-20 20:33:25,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1210226.6666666667, ans=0.125 2023-11-20 20:33:28,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1210226.6666666667, ans=0.125 2023-11-20 20:33:34,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1210226.6666666667, ans=0.1 2023-11-20 20:33:36,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1210293.3333333333, ans=0.1 2023-11-20 20:33:42,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181550 2023-11-20 20:33:46,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1210293.3333333333, ans=0.125 2023-11-20 20:33:49,348 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1200, loss[loss=0.0501, simple_loss=0.05683, pruned_loss=0.0101, audio_tagging_loss=0.01158, over 13900.00 frames. ], tot_loss[loss=0.07657, simple_loss=0.09741, pruned_loss=0.01809, audio_tagging_loss=0.009767, over 3045600.26 frames. ], batch size: 54, lr: 4.36e-03, grad_scale: 32.0 2023-11-20 20:33:50,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1210360.0, ans=0.1 2023-11-20 20:33:55,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1210360.0, ans=0.125 2023-11-20 20:33:57,407 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.318e+01 9.034e+01 9.990e+01 1.243e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-20 20:34:14,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1210493.3333333333, ans=0.2 2023-11-20 20:34:17,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.48 vs. limit=22.5 2023-11-20 20:34:26,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1210493.3333333333, ans=0.0 2023-11-20 20:34:26,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.66 vs. limit=15.0 2023-11-20 20:34:45,952 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181600 2023-11-20 20:34:48,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1210626.6666666667, ans=0.125 2023-11-20 20:34:54,253 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1250, loss[loss=0.07903, simple_loss=0.09592, pruned_loss=0.02127, audio_tagging_loss=0.009803, over 15449.00 frames. ], tot_loss[loss=0.07693, simple_loss=0.09766, pruned_loss=0.01829, audio_tagging_loss=0.009807, over 3043755.01 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 32.0 2023-11-20 20:35:21,414 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.12 vs. limit=6.0 2023-11-20 20:35:45,983 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.96 vs. limit=15.0 2023-11-20 20:35:50,134 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181650 2023-11-20 20:35:52,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1210960.0, ans=0.0 2023-11-20 20:35:55,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1210960.0, ans=15.0 2023-11-20 20:35:57,736 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1300, loss[loss=0.0874, simple_loss=0.1254, pruned_loss=0.01687, audio_tagging_loss=0.007819, over 15923.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09693, pruned_loss=0.01799, audio_tagging_loss=0.009814, over 3044701.75 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:35:58,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1211026.6666666667, ans=0.125 2023-11-20 20:36:06,280 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.079e+01 9.042e+01 9.644e+01 1.533e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-20 20:36:07,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1211026.6666666667, ans=0.0 2023-11-20 20:36:33,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1211160.0, ans=0.2 2023-11-20 20:36:38,707 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.48 vs. limit=10.0 2023-11-20 20:36:47,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1211293.3333333333, ans=0.1 2023-11-20 20:36:49,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1211293.3333333333, ans=0.1 2023-11-20 20:36:53,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181700 2023-11-20 20:37:00,283 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1350, loss[loss=0.07778, simple_loss=0.1024, pruned_loss=0.01896, audio_tagging_loss=0.007625, over 15183.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09739, pruned_loss=0.01806, audio_tagging_loss=0.009748, over 3047694.54 frames. ], batch size: 55, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:37:05,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1211360.0, ans=0.1 2023-11-20 20:37:15,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1211426.6666666667, ans=0.0 2023-11-20 20:37:46,293 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 20:37:56,931 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181750 2023-11-20 20:37:58,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1211626.6666666667, ans=0.125 2023-11-20 20:38:02,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5 2023-11-20 20:38:04,310 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1400, loss[loss=0.07769, simple_loss=0.09531, pruned_loss=0.02009, audio_tagging_loss=0.009941, over 15397.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09782, pruned_loss=0.01795, audio_tagging_loss=0.009823, over 3043411.60 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0 2023-11-20 20:38:13,597 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.529e+01 8.254e+01 8.824e+01 9.596e+01 1.357e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-20 20:38:19,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1211760.0, ans=0.125 2023-11-20 20:38:24,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=1211760.0, ans=0.02 2023-11-20 20:38:25,601 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:38:34,627 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.82 vs. limit=15.0 2023-11-20 20:38:46,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1211893.3333333333, ans=0.0 2023-11-20 20:39:01,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181800 2023-11-20 20:39:04,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1211960.0, ans=0.125 2023-11-20 20:39:07,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1211960.0, ans=0.025 2023-11-20 20:39:09,393 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1450, loss[loss=0.08872, simple_loss=0.112, pruned_loss=0.02468, audio_tagging_loss=0.008063, over 15865.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09908, pruned_loss=0.01836, audio_tagging_loss=0.009725, over 3040709.13 frames. ], batch size: 57, lr: 4.35e-03, grad_scale: 16.0 2023-11-20 20:39:15,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1212026.6666666667, ans=0.0 2023-11-20 20:39:19,231 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.69 vs. limit=10.0 2023-11-20 20:39:33,358 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.09 vs. limit=12.0 2023-11-20 20:39:39,464 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.51 vs. limit=22.5 2023-11-20 20:39:50,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1212226.6666666667, ans=0.0 2023-11-20 20:40:05,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181850 2023-11-20 20:40:06,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1212293.3333333333, ans=0.125 2023-11-20 20:40:13,044 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1500, loss[loss=0.0773, simple_loss=0.094, pruned_loss=0.01703, audio_tagging_loss=0.01327, over 17594.00 frames. ], tot_loss[loss=0.07841, simple_loss=0.1, pruned_loss=0.01864, audio_tagging_loss=0.009765, over 3050334.43 frames. ], batch size: 66, lr: 4.35e-03, grad_scale: 16.0 2023-11-20 20:40:22,243 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.058e+01 8.717e+01 9.373e+01 1.477e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-20 20:40:39,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1212493.3333333333, ans=0.0 2023-11-20 20:40:43,032 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.33 vs. limit=15.0 2023-11-20 20:40:48,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1212493.3333333333, ans=0.1 2023-11-20 20:40:49,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1212493.3333333333, ans=0.125 2023-11-20 20:41:08,817 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181900 2023-11-20 20:41:16,549 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1550, loss[loss=0.07248, simple_loss=0.1016, pruned_loss=0.01529, audio_tagging_loss=0.006404, over 15460.00 frames. ], tot_loss[loss=0.07845, simple_loss=0.09995, pruned_loss=0.01854, audio_tagging_loss=0.009932, over 3049796.02 frames. ], batch size: 57, lr: 4.35e-03, grad_scale: 16.0 2023-11-20 20:42:03,778 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.58 vs. limit=15.0 2023-11-20 20:42:13,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 181950 2023-11-20 20:42:20,299 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1600, loss[loss=0.0745, simple_loss=0.08802, pruned_loss=0.01806, audio_tagging_loss=0.01243, over 15653.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.09906, pruned_loss=0.01837, audio_tagging_loss=0.01001, over 3046966.00 frames. ], batch size: 62, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:42:20,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1213026.6666666667, ans=0.125 2023-11-20 20:42:30,051 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.255e+01 8.909e+01 9.457e+01 1.581e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-20 20:42:44,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1213093.3333333333, ans=0.0 2023-11-20 20:43:17,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182000 2023-11-20 20:43:24,935 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1650, loss[loss=0.08233, simple_loss=0.1063, pruned_loss=0.02044, audio_tagging_loss=0.008756, over 14489.00 frames. ], tot_loss[loss=0.07835, simple_loss=0.09952, pruned_loss=0.01853, audio_tagging_loss=0.01007, over 3045234.38 frames. ], batch size: 54, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:43:57,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1213493.3333333333, ans=0.0 2023-11-20 20:44:00,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1213493.3333333333, ans=0.5 2023-11-20 20:44:13,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1213560.0, ans=0.09899494936611666 2023-11-20 20:44:15,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=1213626.6666666667, ans=0.2 2023-11-20 20:44:18,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1213626.6666666667, ans=0.125 2023-11-20 20:44:20,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182050 2023-11-20 20:44:21,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.74 vs. limit=15.0 2023-11-20 20:44:28,445 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1700, loss[loss=0.06898, simple_loss=0.09404, pruned_loss=0.01281, audio_tagging_loss=0.009151, over 15890.00 frames. ], tot_loss[loss=0.07824, simple_loss=0.0994, pruned_loss=0.01846, audio_tagging_loss=0.01008, over 3053437.45 frames. ], batch size: 60, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:44:36,999 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.816e+01 8.092e+01 8.985e+01 9.523e+01 1.287e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-20 20:44:37,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1213693.3333333333, ans=0.0 2023-11-20 20:44:42,519 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.16 vs. limit=22.5 2023-11-20 20:44:45,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1213760.0, ans=0.2 2023-11-20 20:44:52,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1213826.6666666667, ans=0.2 2023-11-20 20:45:23,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182100 2023-11-20 20:45:25,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1213960.0, ans=0.125 2023-11-20 20:45:31,267 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1750, loss[loss=0.08074, simple_loss=0.09662, pruned_loss=0.02131, audio_tagging_loss=0.01113, over 15064.00 frames. ], tot_loss[loss=0.07796, simple_loss=0.09906, pruned_loss=0.01838, audio_tagging_loss=0.01005, over 3052284.12 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:45:43,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1214093.3333333333, ans=0.0 2023-11-20 20:46:17,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1214226.6666666667, ans=0.0 2023-11-20 20:46:22,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1214293.3333333333, ans=0.125 2023-11-20 20:46:26,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1214293.3333333333, ans=0.125 2023-11-20 20:46:27,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182150 2023-11-20 20:46:29,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1214293.3333333333, ans=0.125 2023-11-20 20:46:35,464 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1800, loss[loss=0.08669, simple_loss=0.1212, pruned_loss=0.01801, audio_tagging_loss=0.008106, over 15964.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09906, pruned_loss=0.01829, audio_tagging_loss=0.009881, over 3043377.54 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:46:43,831 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 8.260e+01 8.761e+01 9.578e+01 1.185e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-20 20:46:49,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=15.0 2023-11-20 20:47:11,196 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.38 vs. limit=22.5 2023-11-20 20:47:15,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.60 vs. limit=15.0 2023-11-20 20:47:31,377 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182200 2023-11-20 20:47:35,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1214626.6666666667, ans=0.5 2023-11-20 20:47:40,224 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1850, loss[loss=0.07457, simple_loss=0.09179, pruned_loss=0.01641, audio_tagging_loss=0.01227, over 14770.00 frames. ], tot_loss[loss=0.07725, simple_loss=0.0982, pruned_loss=0.01826, audio_tagging_loss=0.009887, over 3041685.28 frames. ], batch size: 54, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:47:42,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.82 vs. limit=22.5 2023-11-20 20:47:43,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1214693.3333333333, ans=0.2 2023-11-20 20:47:50,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1214693.3333333333, ans=0.125 2023-11-20 20:48:24,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.58 vs. limit=15.0 2023-11-20 20:48:29,967 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.78 vs. limit=22.5 2023-11-20 20:48:37,000 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182250 2023-11-20 20:48:44,182 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1900, loss[loss=0.08377, simple_loss=0.1119, pruned_loss=0.0192, audio_tagging_loss=0.008634, over 15053.00 frames. ], tot_loss[loss=0.07716, simple_loss=0.09822, pruned_loss=0.01824, audio_tagging_loss=0.009807, over 3046507.39 frames. ], batch size: 57, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:48:53,247 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 7.917e+01 8.580e+01 9.433e+01 1.165e+02, threshold=1.716e+02, percent-clipped=0.0 2023-11-20 20:49:09,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1215160.0, ans=0.0 2023-11-20 20:49:11,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1215160.0, ans=0.1 2023-11-20 20:49:12,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1215160.0, ans=0.125 2023-11-20 20:49:34,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1215293.3333333333, ans=0.0 2023-11-20 20:49:40,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182300 2023-11-20 20:49:47,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1215360.0, ans=0.1 2023-11-20 20:49:48,456 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 1950, loss[loss=0.07589, simple_loss=0.08893, pruned_loss=0.01891, audio_tagging_loss=0.01252, over 15498.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09735, pruned_loss=0.01813, audio_tagging_loss=0.009859, over 3045866.56 frames. ], batch size: 59, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:49:55,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1215360.0, ans=0.1 2023-11-20 20:50:07,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1215426.6666666667, ans=0.1 2023-11-20 20:50:10,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1215426.6666666667, ans=0.0 2023-11-20 20:50:16,970 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.36 vs. limit=22.5 2023-11-20 20:50:19,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1215493.3333333333, ans=0.5 2023-11-20 20:50:37,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1215560.0, ans=0.1 2023-11-20 20:50:45,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182350 2023-11-20 20:50:53,430 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2000, loss[loss=0.09952, simple_loss=0.1275, pruned_loss=0.02664, audio_tagging_loss=0.009122, over 15929.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09712, pruned_loss=0.01806, audio_tagging_loss=0.009891, over 3047370.69 frames. ], batch size: 59, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:50:56,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1215693.3333333333, ans=0.125 2023-11-20 20:51:00,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1215693.3333333333, ans=0.1 2023-11-20 20:51:02,097 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.27 vs. limit=15.0 2023-11-20 20:51:02,675 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.938e+01 8.733e+01 9.578e+01 1.409e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-20 20:51:02,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1215693.3333333333, ans=0.1 2023-11-20 20:51:06,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1215760.0, ans=0.0 2023-11-20 20:51:11,610 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:51:19,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.13 vs. limit=15.0 2023-11-20 20:51:48,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1215960.0, ans=0.125 2023-11-20 20:51:49,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1215960.0, ans=0.125 2023-11-20 20:51:49,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1215960.0, ans=0.07 2023-11-20 20:51:50,569 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182400 2023-11-20 20:51:58,154 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2050, loss[loss=0.08329, simple_loss=0.1102, pruned_loss=0.02121, audio_tagging_loss=0.00698, over 15776.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.09634, pruned_loss=0.01798, audio_tagging_loss=0.009939, over 3041013.17 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:51:59,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1216026.6666666667, ans=0.125 2023-11-20 20:52:40,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1216226.6666666667, ans=0.0 2023-11-20 20:52:43,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1216226.6666666667, ans=0.125 2023-11-20 20:52:44,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1216226.6666666667, ans=0.0 2023-11-20 20:52:54,275 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182450 2023-11-20 20:53:02,341 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2100, loss[loss=0.08825, simple_loss=0.1134, pruned_loss=0.02422, audio_tagging_loss=0.007337, over 16099.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09793, pruned_loss=0.01855, audio_tagging_loss=0.009863, over 3034694.53 frames. ], batch size: 60, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:53:11,020 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.190e+01 8.920e+01 9.750e+01 1.833e+02, threshold=1.784e+02, percent-clipped=1.0 2023-11-20 20:53:21,522 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0 2023-11-20 20:53:24,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.04 vs. limit=15.0 2023-11-20 20:53:30,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.83 vs. limit=6.0 2023-11-20 20:53:43,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1216560.0, ans=0.0 2023-11-20 20:53:51,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1216560.0, ans=0.0 2023-11-20 20:53:53,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1216626.6666666667, ans=0.2 2023-11-20 20:53:57,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1216626.6666666667, ans=0.2 2023-11-20 20:53:58,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182500 2023-11-20 20:54:05,962 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.06 vs. limit=15.0 2023-11-20 20:54:06,755 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2150, loss[loss=0.07272, simple_loss=0.09933, pruned_loss=0.01305, audio_tagging_loss=0.01001, over 15699.00 frames. ], tot_loss[loss=0.07696, simple_loss=0.09766, pruned_loss=0.01835, audio_tagging_loss=0.009777, over 3040277.12 frames. ], batch size: 57, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:54:09,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.80 vs. limit=15.0 2023-11-20 20:54:16,083 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.60 vs. limit=15.0 2023-11-20 20:54:26,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1216760.0, ans=0.125 2023-11-20 20:54:34,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1216826.6666666667, ans=0.125 2023-11-20 20:54:39,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1216826.6666666667, ans=0.5 2023-11-20 20:54:43,817 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 20:55:02,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1216960.0, ans=0.125 2023-11-20 20:55:03,998 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182550 2023-11-20 20:55:11,332 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2200, loss[loss=0.08307, simple_loss=0.1098, pruned_loss=0.02063, audio_tagging_loss=0.007557, over 14827.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09837, pruned_loss=0.01844, audio_tagging_loss=0.009766, over 3046334.98 frames. ], batch size: 54, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:55:19,777 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.514e+01 9.084e+01 9.823e+01 1.256e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-20 20:55:28,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1217093.3333333333, ans=0.125 2023-11-20 20:55:29,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1217093.3333333333, ans=0.125 2023-11-20 20:56:07,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182600 2023-11-20 20:56:12,251 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:56:16,429 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2250, loss[loss=0.07577, simple_loss=0.1011, pruned_loss=0.01759, audio_tagging_loss=0.007645, over 14600.00 frames. ], tot_loss[loss=0.07672, simple_loss=0.09728, pruned_loss=0.01817, audio_tagging_loss=0.009907, over 3045211.14 frames. ], batch size: 57, lr: 4.35e-03, grad_scale: 32.0 2023-11-20 20:56:19,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1217360.0, ans=0.0 2023-11-20 20:56:19,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1217360.0, ans=0.125 2023-11-20 20:56:24,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1217360.0, ans=0.125 2023-11-20 20:56:33,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.97 vs. limit=10.0 2023-11-20 20:56:55,800 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 20:57:00,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1217560.0, ans=0.125 2023-11-20 20:57:11,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182650 2023-11-20 20:57:19,142 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2300, loss[loss=0.1069, simple_loss=0.1515, pruned_loss=0.02511, audio_tagging_loss=0.006005, over 15222.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.0983, pruned_loss=0.01842, audio_tagging_loss=0.009936, over 3041837.39 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 20:57:29,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.519e+01 8.084e+01 8.941e+01 9.693e+01 1.110e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-20 20:57:29,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1217693.3333333333, ans=0.125 2023-11-20 20:57:56,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1217826.6666666667, ans=0.0 2023-11-20 20:58:03,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1217893.3333333333, ans=0.1 2023-11-20 20:58:15,665 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 20:58:15,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1217960.0, ans=0.125 2023-11-20 20:58:16,918 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182700 2023-11-20 20:58:18,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1217960.0, ans=0.125 2023-11-20 20:58:24,659 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2350, loss[loss=0.08789, simple_loss=0.1166, pruned_loss=0.02163, audio_tagging_loss=0.007968, over 15715.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.0983, pruned_loss=0.01822, audio_tagging_loss=0.009983, over 3044341.04 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 20:58:53,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1218160.0, ans=0.1 2023-11-20 20:58:57,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1218160.0, ans=0.125 2023-11-20 20:59:15,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1218293.3333333333, ans=0.125 2023-11-20 20:59:21,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182750 2023-11-20 20:59:28,323 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2400, loss[loss=0.0675, simple_loss=0.08358, pruned_loss=0.01611, audio_tagging_loss=0.009594, over 14143.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.09736, pruned_loss=0.01808, audio_tagging_loss=0.01022, over 3045570.71 frames. ], batch size: 53, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 20:59:29,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1218360.0, ans=0.125 2023-11-20 20:59:37,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1218360.0, ans=0.125 2023-11-20 20:59:38,639 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.066e+01 8.791e+01 9.748e+01 1.239e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-20 21:00:18,574 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:00:25,003 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182800 2023-11-20 21:00:25,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1218626.6666666667, ans=0.0 2023-11-20 21:00:32,674 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2450, loss[loss=0.06858, simple_loss=0.09009, pruned_loss=0.01454, audio_tagging_loss=0.008993, over 16139.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09691, pruned_loss=0.01806, audio_tagging_loss=0.01029, over 3046228.58 frames. ], batch size: 63, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 21:00:35,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1218693.3333333333, ans=0.125 2023-11-20 21:00:43,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1218693.3333333333, ans=0.0 2023-11-20 21:01:09,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1218826.6666666667, ans=0.1 2023-11-20 21:01:27,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1218960.0, ans=0.1 2023-11-20 21:01:30,022 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182850 2023-11-20 21:01:37,755 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2500, loss[loss=0.1091, simple_loss=0.1447, pruned_loss=0.02919, audio_tagging_loss=0.007596, over 15555.00 frames. ], tot_loss[loss=0.07682, simple_loss=0.09708, pruned_loss=0.01809, audio_tagging_loss=0.01019, over 3040800.53 frames. ], batch size: 54, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 21:01:48,262 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 7.891e+01 8.728e+01 9.490e+01 1.260e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-20 21:01:55,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1219093.3333333333, ans=0.0 2023-11-20 21:02:15,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=1219226.6666666667, ans=15.0 2023-11-20 21:02:25,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1219226.6666666667, ans=0.125 2023-11-20 21:02:26,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1219226.6666666667, ans=0.0 2023-11-20 21:02:29,322 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.85 vs. limit=15.0 2023-11-20 21:02:35,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182900 2023-11-20 21:02:36,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.51 vs. limit=10.0 2023-11-20 21:02:39,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1219293.3333333333, ans=0.125 2023-11-20 21:02:42,996 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2550, loss[loss=0.08747, simple_loss=0.115, pruned_loss=0.02464, audio_tagging_loss=0.005333, over 15529.00 frames. ], tot_loss[loss=0.0767, simple_loss=0.09703, pruned_loss=0.01811, audio_tagging_loss=0.01007, over 3041478.65 frames. ], batch size: 54, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 21:02:44,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1219360.0, ans=0.0 2023-11-20 21:02:48,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1219360.0, ans=0.0 2023-11-20 21:03:14,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1219493.3333333333, ans=0.125 2023-11-20 21:03:28,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1219560.0, ans=0.0 2023-11-20 21:03:40,338 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 182950 2023-11-20 21:03:47,485 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2600, loss[loss=0.07937, simple_loss=0.1007, pruned_loss=0.02147, audio_tagging_loss=0.00755, over 15511.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.09716, pruned_loss=0.01804, audio_tagging_loss=0.009916, over 3038395.82 frames. ], batch size: 58, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 21:03:57,807 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.668e+01 8.265e+01 9.445e+01 1.045e+02 1.287e+02, threshold=1.889e+02, percent-clipped=0.0 2023-11-20 21:04:01,497 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.04 vs. limit=22.5 2023-11-20 21:04:04,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1219760.0, ans=0.2 2023-11-20 21:04:22,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1219826.6666666667, ans=0.1 2023-11-20 21:04:43,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183000 2023-11-20 21:04:52,363 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2650, loss[loss=0.06559, simple_loss=0.08369, pruned_loss=0.01647, audio_tagging_loss=0.007277, over 15851.00 frames. ], tot_loss[loss=0.07712, simple_loss=0.09822, pruned_loss=0.01828, audio_tagging_loss=0.009731, over 3043494.10 frames. ], batch size: 59, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 21:05:02,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1220026.6666666667, ans=0.125 2023-11-20 21:05:10,883 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.40 vs. limit=15.0 2023-11-20 21:05:18,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1220160.0, ans=0.04949747468305833 2023-11-20 21:05:21,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1220160.0, ans=0.2 2023-11-20 21:05:48,551 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183050 2023-11-20 21:05:51,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1220293.3333333333, ans=0.1 2023-11-20 21:05:56,272 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2700, loss[loss=0.0707, simple_loss=0.08877, pruned_loss=0.01771, audio_tagging_loss=0.008601, over 15160.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09823, pruned_loss=0.01814, audio_tagging_loss=0.00967, over 3040799.38 frames. ], batch size: 57, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 21:05:58,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1220360.0, ans=0.125 2023-11-20 21:06:00,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.28 vs. limit=15.0 2023-11-20 21:06:06,370 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.355e+01 9.034e+01 9.882e+01 1.379e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-20 21:06:11,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.04 vs. limit=12.0 2023-11-20 21:06:16,592 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.52 vs. limit=15.0 2023-11-20 21:06:35,296 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.57 vs. limit=8.0 2023-11-20 21:06:51,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.55 vs. limit=15.0 2023-11-20 21:06:52,681 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183100 2023-11-20 21:06:55,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1220626.6666666667, ans=0.0 2023-11-20 21:06:59,895 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2750, loss[loss=0.06464, simple_loss=0.07489, pruned_loss=0.01279, audio_tagging_loss=0.0144, over 15069.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09787, pruned_loss=0.01814, audio_tagging_loss=0.009689, over 3048593.40 frames. ], batch size: 57, lr: 4.34e-03, grad_scale: 16.0 2023-11-20 21:07:17,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1220760.0, ans=0.125 2023-11-20 21:07:42,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.60 vs. limit=15.0 2023-11-20 21:07:52,657 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 21:07:56,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183150 2023-11-20 21:07:56,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1220960.0, ans=0.125 2023-11-20 21:08:04,603 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2800, loss[loss=0.0638, simple_loss=0.08614, pruned_loss=0.01269, audio_tagging_loss=0.008038, over 15671.00 frames. ], tot_loss[loss=0.07682, simple_loss=0.0981, pruned_loss=0.01814, audio_tagging_loss=0.009628, over 3048737.09 frames. ], batch size: 61, lr: 4.34e-03, grad_scale: 32.0 2023-11-20 21:08:07,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1221026.6666666667, ans=0.1 2023-11-20 21:08:09,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1221026.6666666667, ans=0.125 2023-11-20 21:08:15,657 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.121e+01 8.798e+01 9.627e+01 1.849e+02, threshold=1.760e+02, percent-clipped=1.0 2023-11-20 21:08:15,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1221093.3333333333, ans=0.1 2023-11-20 21:08:20,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1221093.3333333333, ans=0.1 2023-11-20 21:08:35,996 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.48 vs. limit=15.0 2023-11-20 21:08:46,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1221226.6666666667, ans=0.2 2023-11-20 21:09:01,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183200 2023-11-20 21:09:01,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1221293.3333333333, ans=0.125 2023-11-20 21:09:08,971 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2850, loss[loss=0.08013, simple_loss=0.1062, pruned_loss=0.01703, audio_tagging_loss=0.009983, over 15035.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09938, pruned_loss=0.01836, audio_tagging_loss=0.00957, over 3050100.16 frames. ], batch size: 57, lr: 4.34e-03, grad_scale: 16.0 2023-11-20 21:09:35,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1221493.3333333333, ans=0.1 2023-11-20 21:09:44,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.41 vs. limit=15.0 2023-11-20 21:09:46,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.50 vs. limit=22.5 2023-11-20 21:09:59,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1221626.6666666667, ans=0.125 2023-11-20 21:10:05,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183250 2023-11-20 21:10:13,750 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2900, loss[loss=0.06436, simple_loss=0.07445, pruned_loss=0.01393, audio_tagging_loss=0.0132, over 14942.00 frames. ], tot_loss[loss=0.07748, simple_loss=0.09939, pruned_loss=0.01826, audio_tagging_loss=0.009526, over 3052823.01 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 16.0 2023-11-20 21:10:21,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1221693.3333333333, ans=0.2 2023-11-20 21:10:26,046 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.821e+01 8.647e+01 9.509e+01 1.495e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-20 21:10:42,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1221826.6666666667, ans=0.5 2023-11-20 21:10:55,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=15.0 2023-11-20 21:11:09,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183300 2023-11-20 21:11:17,176 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 2950, loss[loss=0.1116, simple_loss=0.1464, pruned_loss=0.03216, audio_tagging_loss=0.006248, over 14822.00 frames. ], tot_loss[loss=0.07814, simple_loss=0.1002, pruned_loss=0.01846, audio_tagging_loss=0.009585, over 3052367.67 frames. ], batch size: 55, lr: 4.34e-03, grad_scale: 16.0 2023-11-20 21:11:30,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.30 vs. limit=22.5 2023-11-20 21:11:40,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1222093.3333333333, ans=0.0 2023-11-20 21:11:55,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.19 vs. limit=15.0 2023-11-20 21:11:56,079 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:12:05,011 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=12.0 2023-11-20 21:12:11,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1222293.3333333333, ans=0.125 2023-11-20 21:12:13,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183350 2023-11-20 21:12:15,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1222293.3333333333, ans=0.1 2023-11-20 21:12:21,099 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3000, loss[loss=0.08529, simple_loss=0.1147, pruned_loss=0.01955, audio_tagging_loss=0.008411, over 15419.00 frames. ], tot_loss[loss=0.07809, simple_loss=0.09978, pruned_loss=0.01844, audio_tagging_loss=0.009757, over 3054054.90 frames. ], batch size: 58, lr: 4.34e-03, grad_scale: 16.0 2023-11-20 21:12:21,099 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 21:12:45,958 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.4758, 0.8023, 3.5537, 2.8503, 2.7500, 3.1804, 2.9065, 3.0074], device='cuda:3') 2023-11-20 21:12:59,440 INFO [train_asr.py:1253] (3/4) Epoch 16, validation: loss=0.06057, simple_loss=0.053, pruned_loss=0.005481, audio_tagging_loss=0.02859, over 4681554.00 frames. 2023-11-20 21:12:59,442 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 21:13:11,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1222426.6666666667, ans=0.125 2023-11-20 21:13:12,785 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.173e+01 8.695e+01 9.591e+01 1.296e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-20 21:13:20,395 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:13:26,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1222493.3333333333, ans=0.2 2023-11-20 21:13:30,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1222493.3333333333, ans=0.2 2023-11-20 21:13:48,667 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0 2023-11-20 21:13:49,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1222626.6666666667, ans=0.125 2023-11-20 21:13:55,675 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183400 2023-11-20 21:14:04,218 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3050, loss[loss=0.0663, simple_loss=0.08695, pruned_loss=0.01068, audio_tagging_loss=0.01214, over 14626.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.09939, pruned_loss=0.01841, audio_tagging_loss=0.009886, over 3056799.37 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 16.0 2023-11-20 21:14:16,855 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:14:20,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1222760.0, ans=0.05 2023-11-20 21:14:24,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1222760.0, ans=0.125 2023-11-20 21:14:25,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1222760.0, ans=0.0 2023-11-20 21:14:39,816 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 21:14:58,732 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.53 vs. limit=15.0 2023-11-20 21:15:00,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183450 2023-11-20 21:15:00,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1222960.0, ans=0.035 2023-11-20 21:15:00,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1222960.0, ans=0.125 2023-11-20 21:15:07,765 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3100, loss[loss=0.08325, simple_loss=0.1104, pruned_loss=0.01908, audio_tagging_loss=0.008972, over 14974.00 frames. ], tot_loss[loss=0.0783, simple_loss=0.1002, pruned_loss=0.01842, audio_tagging_loss=0.009781, over 3051228.10 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 16.0 2023-11-20 21:15:12,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1223026.6666666667, ans=0.125 2023-11-20 21:15:20,363 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.321e+01 8.878e+01 9.736e+01 1.416e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-20 21:15:33,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1223160.0, ans=0.0 2023-11-20 21:15:41,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1223160.0, ans=0.1 2023-11-20 21:15:53,608 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0 2023-11-20 21:16:03,242 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.42 vs. limit=15.0 2023-11-20 21:16:03,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183500 2023-11-20 21:16:11,731 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3150, loss[loss=0.07554, simple_loss=0.09906, pruned_loss=0.0146, audio_tagging_loss=0.01141, over 15610.00 frames. ], tot_loss[loss=0.07833, simple_loss=0.1, pruned_loss=0.01836, audio_tagging_loss=0.009963, over 3045845.07 frames. ], batch size: 58, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:16:12,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1223360.0, ans=0.125 2023-11-20 21:16:14,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1223360.0, ans=10.0 2023-11-20 21:16:18,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1223360.0, ans=0.0 2023-11-20 21:16:25,609 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=15.0 2023-11-20 21:16:26,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1223426.6666666667, ans=0.0 2023-11-20 21:16:46,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1223493.3333333333, ans=0.035 2023-11-20 21:17:07,890 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183550 2023-11-20 21:17:12,300 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:17:16,815 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3200, loss[loss=0.09519, simple_loss=0.1111, pruned_loss=0.02809, audio_tagging_loss=0.01156, over 14909.00 frames. ], tot_loss[loss=0.07812, simple_loss=0.09951, pruned_loss=0.01836, audio_tagging_loss=0.01001, over 3042410.50 frames. ], batch size: 60, lr: 4.33e-03, grad_scale: 32.0 2023-11-20 21:17:28,829 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.173e+01 8.835e+01 9.713e+01 1.308e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-20 21:17:32,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1223760.0, ans=10.0 2023-11-20 21:17:54,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1223893.3333333333, ans=0.125 2023-11-20 21:17:56,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.76 vs. limit=15.0 2023-11-20 21:18:07,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1223960.0, ans=0.0 2023-11-20 21:18:12,575 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183600 2023-11-20 21:18:20,306 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3250, loss[loss=0.08206, simple_loss=0.09831, pruned_loss=0.0199, audio_tagging_loss=0.01301, over 15226.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.09858, pruned_loss=0.01826, audio_tagging_loss=0.01016, over 3041007.02 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:18:44,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1224160.0, ans=0.1 2023-11-20 21:18:46,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1224160.0, ans=10.0 2023-11-20 21:19:16,938 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183650 2023-11-20 21:19:24,572 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3300, loss[loss=0.08191, simple_loss=0.1071, pruned_loss=0.02087, audio_tagging_loss=0.00749, over 15623.00 frames. ], tot_loss[loss=0.07821, simple_loss=0.09919, pruned_loss=0.01838, audio_tagging_loss=0.01024, over 3044649.23 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:19:27,693 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.63 vs. limit=15.0 2023-11-20 21:19:38,005 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.036e+01 8.648e+01 9.640e+01 1.721e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-20 21:19:50,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1224493.3333333333, ans=0.1 2023-11-20 21:19:59,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1224493.3333333333, ans=0.2 2023-11-20 21:20:02,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1224560.0, ans=0.0 2023-11-20 21:20:20,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183700 2023-11-20 21:20:25,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1224626.6666666667, ans=0.1 2023-11-20 21:20:28,152 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3350, loss[loss=0.09189, simple_loss=0.1234, pruned_loss=0.02172, audio_tagging_loss=0.008474, over 15922.00 frames. ], tot_loss[loss=0.07816, simple_loss=0.09931, pruned_loss=0.01839, audio_tagging_loss=0.01012, over 3046192.70 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:20:35,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1224693.3333333333, ans=0.125 2023-11-20 21:20:50,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1224760.0, ans=0.2 2023-11-20 21:21:21,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1224960.0, ans=0.125 2023-11-20 21:21:25,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1224960.0, ans=0.125 2023-11-20 21:21:26,404 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183750 2023-11-20 21:21:33,801 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3400, loss[loss=0.05361, simple_loss=0.05872, pruned_loss=0.009037, audio_tagging_loss=0.01522, over 15216.00 frames. ], tot_loss[loss=0.07807, simple_loss=0.09929, pruned_loss=0.01845, audio_tagging_loss=0.009977, over 3046585.28 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:21:34,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.79 vs. limit=15.0 2023-11-20 21:21:38,943 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:21:47,180 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.496e+01 8.162e+01 8.769e+01 9.534e+01 1.233e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-20 21:21:49,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1225093.3333333333, ans=0.0 2023-11-20 21:21:53,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1225093.3333333333, ans=0.025 2023-11-20 21:21:56,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1225093.3333333333, ans=0.125 2023-11-20 21:22:16,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1225226.6666666667, ans=0.125 2023-11-20 21:22:20,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1225226.6666666667, ans=15.0 2023-11-20 21:22:29,940 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183800 2023-11-20 21:22:30,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1225293.3333333333, ans=0.0 2023-11-20 21:22:38,209 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3450, loss[loss=0.09368, simple_loss=0.1276, pruned_loss=0.02211, audio_tagging_loss=0.00775, over 15445.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09954, pruned_loss=0.01843, audio_tagging_loss=0.009901, over 3051093.22 frames. ], batch size: 55, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:22:49,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1225426.6666666667, ans=0.1 2023-11-20 21:23:15,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1225560.0, ans=0.0 2023-11-20 21:23:30,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1225626.6666666667, ans=0.125 2023-11-20 21:23:34,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183850 2023-11-20 21:23:41,697 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3500, loss[loss=0.0862, simple_loss=0.1115, pruned_loss=0.02009, audio_tagging_loss=0.01036, over 15141.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.09977, pruned_loss=0.01854, audio_tagging_loss=0.009778, over 3047732.87 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:23:48,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1225693.3333333333, ans=0.025 2023-11-20 21:23:57,064 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.353e+01 9.150e+01 1.014e+02 1.535e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-20 21:24:08,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1225826.6666666667, ans=0.09899494936611666 2023-11-20 21:24:09,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1225826.6666666667, ans=0.0 2023-11-20 21:24:13,353 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 21:24:17,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1225826.6666666667, ans=0.125 2023-11-20 21:24:34,625 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:24:37,602 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:24:39,959 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183900 2023-11-20 21:24:47,820 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3550, loss[loss=0.06952, simple_loss=0.09062, pruned_loss=0.01637, audio_tagging_loss=0.007841, over 15061.00 frames. ], tot_loss[loss=0.07851, simple_loss=0.1005, pruned_loss=0.0186, audio_tagging_loss=0.009686, over 3054418.74 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:25:24,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.22 vs. limit=15.0 2023-11-20 21:25:36,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.06 vs. limit=22.5 2023-11-20 21:25:44,362 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 183950 2023-11-20 21:25:51,792 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3600, loss[loss=0.05889, simple_loss=0.06862, pruned_loss=0.01186, audio_tagging_loss=0.01272, over 16250.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09898, pruned_loss=0.01838, audio_tagging_loss=0.00973, over 3052502.51 frames. ], batch size: 62, lr: 4.33e-03, grad_scale: 32.0 2023-11-20 21:25:52,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.31 vs. limit=22.5 2023-11-20 21:26:02,996 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.89 vs. limit=22.5 2023-11-20 21:26:05,869 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.147e+01 8.022e+01 8.880e+01 9.578e+01 1.162e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-20 21:26:09,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1226426.6666666667, ans=0.1 2023-11-20 21:26:17,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1226493.3333333333, ans=0.125 2023-11-20 21:26:48,805 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184000 2023-11-20 21:27:00,016 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3650, loss[loss=0.1021, simple_loss=0.1266, pruned_loss=0.02889, audio_tagging_loss=0.009891, over 13677.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.09936, pruned_loss=0.01846, audio_tagging_loss=0.009754, over 3048916.01 frames. ], batch size: 53, lr: 4.33e-03, grad_scale: 32.0 2023-11-20 21:27:25,101 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.75 vs. limit=15.0 2023-11-20 21:27:43,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1226893.3333333333, ans=0.0 2023-11-20 21:27:43,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1226893.3333333333, ans=0.2 2023-11-20 21:27:57,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1226960.0, ans=0.125 2023-11-20 21:27:57,979 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184050 2023-11-20 21:27:58,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1226960.0, ans=0.125 2023-11-20 21:28:05,152 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3700, loss[loss=0.08387, simple_loss=0.1129, pruned_loss=0.02028, audio_tagging_loss=0.007129, over 15977.00 frames. ], tot_loss[loss=0.078, simple_loss=0.09948, pruned_loss=0.0185, audio_tagging_loss=0.009759, over 3045841.14 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 32.0 2023-11-20 21:28:08,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1227026.6666666667, ans=0.2 2023-11-20 21:28:11,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1227026.6666666667, ans=0.1 2023-11-20 21:28:15,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1227026.6666666667, ans=0.0 2023-11-20 21:28:16,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.09 vs. limit=10.0 2023-11-20 21:28:19,419 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.009e+01 8.576e+01 9.316e+01 1.305e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-20 21:28:23,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1227093.3333333333, ans=0.2 2023-11-20 21:28:31,791 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.90 vs. limit=12.0 2023-11-20 21:28:38,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1227160.0, ans=0.2 2023-11-20 21:28:43,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1227226.6666666667, ans=0.125 2023-11-20 21:28:53,472 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:28:57,751 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.77 vs. limit=12.0 2023-11-20 21:28:58,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1227293.3333333333, ans=0.125 2023-11-20 21:29:02,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184100 2023-11-20 21:29:08,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1227293.3333333333, ans=0.1 2023-11-20 21:29:10,247 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3750, loss[loss=0.0965, simple_loss=0.1305, pruned_loss=0.02493, audio_tagging_loss=0.006324, over 16382.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09959, pruned_loss=0.01855, audio_tagging_loss=0.009812, over 3047245.78 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 32.0 2023-11-20 21:29:27,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1227426.6666666667, ans=0.125 2023-11-20 21:29:52,880 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 21:30:02,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.32 vs. limit=5.0 2023-11-20 21:30:06,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184150 2023-11-20 21:30:12,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1227693.3333333333, ans=0.125 2023-11-20 21:30:13,954 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3800, loss[loss=0.05282, simple_loss=0.06276, pruned_loss=0.01369, audio_tagging_loss=0.007751, over 16496.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09914, pruned_loss=0.01841, audio_tagging_loss=0.009855, over 3055654.39 frames. ], batch size: 63, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:30:19,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1227693.3333333333, ans=0.0 2023-11-20 21:30:29,924 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.193e+01 8.281e+01 9.046e+01 9.670e+01 1.407e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-20 21:30:56,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.53 vs. limit=22.5 2023-11-20 21:30:58,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1227893.3333333333, ans=0.125 2023-11-20 21:31:11,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184200 2023-11-20 21:31:13,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1227960.0, ans=0.1 2023-11-20 21:31:14,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1227960.0, ans=0.0 2023-11-20 21:31:18,621 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3850, loss[loss=0.07665, simple_loss=0.09738, pruned_loss=0.01887, audio_tagging_loss=0.009093, over 15360.00 frames. ], tot_loss[loss=0.07791, simple_loss=0.0994, pruned_loss=0.01838, audio_tagging_loss=0.009828, over 3053149.82 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:31:43,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1228160.0, ans=0.2 2023-11-20 21:31:44,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1228160.0, ans=0.125 2023-11-20 21:31:53,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1228160.0, ans=0.05 2023-11-20 21:32:11,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1228293.3333333333, ans=0.0 2023-11-20 21:32:15,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184250 2023-11-20 21:32:19,636 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.88 vs. limit=6.0 2023-11-20 21:32:23,136 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3900, loss[loss=0.07614, simple_loss=0.09888, pruned_loss=0.0184, audio_tagging_loss=0.008293, over 16006.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09883, pruned_loss=0.01838, audio_tagging_loss=0.009972, over 3042604.76 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:32:31,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.67 vs. limit=15.0 2023-11-20 21:32:37,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1228426.6666666667, ans=0.0 2023-11-20 21:32:38,152 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.176e+01 8.890e+01 9.815e+01 1.146e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-20 21:32:38,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1228426.6666666667, ans=0.125 2023-11-20 21:32:38,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0 2023-11-20 21:32:53,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1228493.3333333333, ans=0.125 2023-11-20 21:33:13,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1228626.6666666667, ans=0.125 2023-11-20 21:33:14,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1228626.6666666667, ans=0.0 2023-11-20 21:33:20,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184300 2023-11-20 21:33:23,054 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.10 vs. limit=15.0 2023-11-20 21:33:27,626 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 3950, loss[loss=0.07335, simple_loss=0.09729, pruned_loss=0.01304, audio_tagging_loss=0.01166, over 16149.00 frames. ], tot_loss[loss=0.0774, simple_loss=0.09831, pruned_loss=0.01822, audio_tagging_loss=0.01002, over 3047424.57 frames. ], batch size: 61, lr: 4.33e-03, grad_scale: 16.0 2023-11-20 21:33:31,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1228693.3333333333, ans=0.0 2023-11-20 21:33:40,942 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.55 vs. limit=10.0 2023-11-20 21:33:58,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1228826.6666666667, ans=0.125 2023-11-20 21:34:08,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1228893.3333333333, ans=0.125 2023-11-20 21:34:09,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1228893.3333333333, ans=0.125 2023-11-20 21:34:11,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1228893.3333333333, ans=0.04949747468305833 2023-11-20 21:34:11,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.58 vs. limit=6.0 2023-11-20 21:34:18,791 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.86 vs. limit=15.0 2023-11-20 21:34:19,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1228960.0, ans=0.125 2023-11-20 21:34:24,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184350 2023-11-20 21:34:32,588 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4000, loss[loss=0.06958, simple_loss=0.08686, pruned_loss=0.01359, audio_tagging_loss=0.01257, over 15390.00 frames. ], tot_loss[loss=0.07761, simple_loss=0.09822, pruned_loss=0.01831, audio_tagging_loss=0.01019, over 3050413.61 frames. ], batch size: 60, lr: 4.32e-03, grad_scale: 32.0 2023-11-20 21:34:36,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1229026.6666666667, ans=0.2 2023-11-20 21:34:39,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1229026.6666666667, ans=0.0 2023-11-20 21:34:47,104 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.174e+01 8.766e+01 9.644e+01 1.258e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-20 21:35:08,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.25 vs. limit=22.5 2023-11-20 21:35:26,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1229293.3333333333, ans=0.025 2023-11-20 21:35:28,650 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184400 2023-11-20 21:35:36,162 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4050, loss[loss=0.1066, simple_loss=0.1445, pruned_loss=0.0295, audio_tagging_loss=0.004916, over 15374.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09817, pruned_loss=0.01833, audio_tagging_loss=0.0102, over 3046958.23 frames. ], batch size: 54, lr: 4.32e-03, grad_scale: 32.0 2023-11-20 21:35:37,496 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 21:35:50,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1229426.6666666667, ans=0.0 2023-11-20 21:36:09,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1229493.3333333333, ans=0.0 2023-11-20 21:36:22,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1229560.0, ans=0.0 2023-11-20 21:36:27,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1229626.6666666667, ans=0.0 2023-11-20 21:36:32,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184450 2023-11-20 21:36:34,440 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.42 vs. limit=6.0 2023-11-20 21:36:38,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1229626.6666666667, ans=0.125 2023-11-20 21:36:40,474 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4100, loss[loss=0.09522, simple_loss=0.1247, pruned_loss=0.02507, audio_tagging_loss=0.00778, over 15847.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.09888, pruned_loss=0.01836, audio_tagging_loss=0.01012, over 3047708.57 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:36:40,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1229693.3333333333, ans=0.125 2023-11-20 21:36:53,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1229760.0, ans=0.0 2023-11-20 21:36:57,519 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.177e+01 8.908e+01 9.687e+01 1.316e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-20 21:37:04,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1229760.0, ans=0.1 2023-11-20 21:37:10,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1229826.6666666667, ans=0.125 2023-11-20 21:37:15,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1229826.6666666667, ans=0.2 2023-11-20 21:37:21,991 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.47 vs. limit=22.5 2023-11-20 21:37:36,954 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184500 2023-11-20 21:37:44,736 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4150, loss[loss=0.06071, simple_loss=0.07541, pruned_loss=0.01499, audio_tagging_loss=0.008022, over 17024.00 frames. ], tot_loss[loss=0.07807, simple_loss=0.09924, pruned_loss=0.01849, audio_tagging_loss=0.009963, over 3046729.07 frames. ], batch size: 66, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:37:48,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1230026.6666666667, ans=0.0 2023-11-20 21:37:50,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1230026.6666666667, ans=0.2 2023-11-20 21:37:51,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1230026.6666666667, ans=0.125 2023-11-20 21:37:53,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1230026.6666666667, ans=0.125 2023-11-20 21:37:55,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1230026.6666666667, ans=0.0 2023-11-20 21:37:59,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1230093.3333333333, ans=0.125 2023-11-20 21:38:03,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1230093.3333333333, ans=0.0 2023-11-20 21:38:03,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1230093.3333333333, ans=0.125 2023-11-20 21:38:28,762 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 21:38:36,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1230293.3333333333, ans=0.125 2023-11-20 21:38:42,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184550 2023-11-20 21:38:44,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1230293.3333333333, ans=0.125 2023-11-20 21:38:47,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1230293.3333333333, ans=0.2 2023-11-20 21:38:49,319 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4200, loss[loss=0.06605, simple_loss=0.08454, pruned_loss=0.0148, audio_tagging_loss=0.008983, over 14881.00 frames. ], tot_loss[loss=0.07797, simple_loss=0.09927, pruned_loss=0.01851, audio_tagging_loss=0.009829, over 3043147.53 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:38:58,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1230360.0, ans=0.1 2023-11-20 21:39:07,583 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.428e+01 8.014e+01 8.553e+01 9.363e+01 1.203e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-20 21:39:41,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1230626.6666666667, ans=0.0 2023-11-20 21:39:45,468 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184600 2023-11-20 21:39:45,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1230626.6666666667, ans=0.0 2023-11-20 21:39:52,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1230693.3333333333, ans=0.125 2023-11-20 21:39:53,579 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4250, loss[loss=0.06624, simple_loss=0.0791, pruned_loss=0.01517, audio_tagging_loss=0.01151, over 14559.00 frames. ], tot_loss[loss=0.07752, simple_loss=0.09872, pruned_loss=0.0183, audio_tagging_loss=0.00986, over 3041916.31 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 8.0 2023-11-20 21:39:56,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1230693.3333333333, ans=0.125 2023-11-20 21:40:02,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1230693.3333333333, ans=0.0 2023-11-20 21:40:27,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1230826.6666666667, ans=0.0 2023-11-20 21:40:40,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1230893.3333333333, ans=0.1 2023-11-20 21:40:50,036 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184650 2023-11-20 21:40:53,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1230960.0, ans=0.125 2023-11-20 21:40:57,893 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4300, loss[loss=0.07357, simple_loss=0.09681, pruned_loss=0.01497, audio_tagging_loss=0.0102, over 15451.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09943, pruned_loss=0.01841, audio_tagging_loss=0.009748, over 3043327.97 frames. ], batch size: 63, lr: 4.32e-03, grad_scale: 8.0 2023-11-20 21:41:09,884 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.13 vs. limit=15.0 2023-11-20 21:41:15,299 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.115e+01 8.273e+01 8.996e+01 9.641e+01 1.140e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-20 21:41:33,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1231160.0, ans=0.09899494936611666 2023-11-20 21:41:54,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184700 2023-11-20 21:42:01,353 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4350, loss[loss=0.07746, simple_loss=0.09348, pruned_loss=0.02037, audio_tagging_loss=0.01035, over 15100.00 frames. ], tot_loss[loss=0.07808, simple_loss=0.1004, pruned_loss=0.01832, audio_tagging_loss=0.009578, over 3054037.26 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 8.0 2023-11-20 21:42:18,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.12 vs. limit=22.5 2023-11-20 21:42:26,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1231493.3333333333, ans=0.0 2023-11-20 21:42:28,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1231493.3333333333, ans=0.1 2023-11-20 21:42:34,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1231493.3333333333, ans=0.125 2023-11-20 21:42:45,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1231560.0, ans=0.07 2023-11-20 21:42:52,824 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:42:57,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184750 2023-11-20 21:43:05,107 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4400, loss[loss=0.05082, simple_loss=0.06301, pruned_loss=0.01168, audio_tagging_loss=0.00763, over 14971.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09951, pruned_loss=0.01821, audio_tagging_loss=0.009604, over 3051683.03 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:43:05,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1231693.3333333333, ans=0.125 2023-11-20 21:43:22,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1231760.0, ans=0.0 2023-11-20 21:43:23,435 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.140e+01 8.871e+01 9.490e+01 1.181e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-20 21:43:48,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1231893.3333333333, ans=0.125 2023-11-20 21:44:01,699 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184800 2023-11-20 21:44:09,925 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4450, loss[loss=0.08706, simple_loss=0.1126, pruned_loss=0.02276, audio_tagging_loss=0.008018, over 15018.00 frames. ], tot_loss[loss=0.07854, simple_loss=0.1009, pruned_loss=0.01861, audio_tagging_loss=0.009503, over 3054552.00 frames. ], batch size: 57, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:44:10,624 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.77 vs. limit=15.0 2023-11-20 21:44:17,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1232026.6666666667, ans=0.1 2023-11-20 21:44:34,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1232160.0, ans=0.125 2023-11-20 21:44:40,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1232160.0, ans=0.0 2023-11-20 21:44:49,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1232226.6666666667, ans=0.125 2023-11-20 21:44:57,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1232226.6666666667, ans=0.0 2023-11-20 21:45:02,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1232293.3333333333, ans=0.125 2023-11-20 21:45:07,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184850 2023-11-20 21:45:15,030 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4500, loss[loss=0.08532, simple_loss=0.1074, pruned_loss=0.02092, audio_tagging_loss=0.01069, over 15057.00 frames. ], tot_loss[loss=0.07835, simple_loss=0.1004, pruned_loss=0.01855, audio_tagging_loss=0.009618, over 3050054.17 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:45:32,804 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.142e+01 8.902e+01 9.690e+01 1.395e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-20 21:46:02,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1232560.0, ans=0.125 2023-11-20 21:46:06,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1232626.6666666667, ans=0.125 2023-11-20 21:46:11,412 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184900 2023-11-20 21:46:14,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1232626.6666666667, ans=0.09899494936611666 2023-11-20 21:46:18,632 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4550, loss[loss=0.07732, simple_loss=0.09422, pruned_loss=0.01831, audio_tagging_loss=0.01189, over 15152.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09918, pruned_loss=0.01837, audio_tagging_loss=0.009743, over 3045755.48 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:46:28,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1232693.3333333333, ans=0.0 2023-11-20 21:46:40,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1232760.0, ans=0.125 2023-11-20 21:47:02,124 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.19 vs. limit=15.0 2023-11-20 21:47:05,287 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 21:47:08,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1232893.3333333333, ans=0.2 2023-11-20 21:47:09,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1232960.0, ans=0.125 2023-11-20 21:47:15,659 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 184950 2023-11-20 21:47:22,848 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4600, loss[loss=0.08433, simple_loss=0.1011, pruned_loss=0.02436, audio_tagging_loss=0.00942, over 14792.00 frames. ], tot_loss[loss=0.07703, simple_loss=0.0979, pruned_loss=0.01822, audio_tagging_loss=0.009855, over 3041344.31 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:47:36,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1233093.3333333333, ans=0.125 2023-11-20 21:47:40,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.813e+01 7.895e+01 8.619e+01 9.471e+01 1.250e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-20 21:47:42,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1233093.3333333333, ans=0.0 2023-11-20 21:47:53,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1233160.0, ans=0.0 2023-11-20 21:47:53,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.37 vs. limit=22.5 2023-11-20 21:48:05,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1233226.6666666667, ans=0.0 2023-11-20 21:48:19,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185000 2023-11-20 21:48:27,578 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4650, loss[loss=0.07504, simple_loss=0.1019, pruned_loss=0.01674, audio_tagging_loss=0.007373, over 15296.00 frames. ], tot_loss[loss=0.0769, simple_loss=0.09804, pruned_loss=0.01807, audio_tagging_loss=0.009817, over 3045497.70 frames. ], batch size: 57, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:48:40,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1233426.6666666667, ans=0.1 2023-11-20 21:48:53,323 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.85 vs. limit=10.0 2023-11-20 21:49:23,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185050 2023-11-20 21:49:27,804 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.41 vs. limit=6.0 2023-11-20 21:49:30,672 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4700, loss[loss=0.06883, simple_loss=0.08328, pruned_loss=0.01339, audio_tagging_loss=0.0138, over 15222.00 frames. ], tot_loss[loss=0.07667, simple_loss=0.09757, pruned_loss=0.01794, audio_tagging_loss=0.00995, over 3046578.19 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:49:41,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1233693.3333333333, ans=0.0 2023-11-20 21:49:48,166 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0 2023-11-20 21:49:48,750 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.027e+01 8.804e+01 9.916e+01 1.555e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-20 21:50:14,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1233893.3333333333, ans=0.125 2023-11-20 21:50:22,620 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.32 vs. limit=5.0 2023-11-20 21:50:28,585 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185100 2023-11-20 21:50:35,788 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4750, loss[loss=0.0741, simple_loss=0.09801, pruned_loss=0.0146, audio_tagging_loss=0.0105, over 14494.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09826, pruned_loss=0.01818, audio_tagging_loss=0.00986, over 3049723.63 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 16.0 2023-11-20 21:50:38,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1234026.6666666667, ans=0.125 2023-11-20 21:50:53,408 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.83 vs. limit=22.5 2023-11-20 21:51:00,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1234160.0, ans=0.5 2023-11-20 21:51:23,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.27 vs. limit=15.0 2023-11-20 21:51:29,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1234293.3333333333, ans=0.1 2023-11-20 21:51:32,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185150 2023-11-20 21:51:37,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1234293.3333333333, ans=0.025 2023-11-20 21:51:39,959 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4800, loss[loss=0.08351, simple_loss=0.1071, pruned_loss=0.02145, audio_tagging_loss=0.008509, over 14873.00 frames. ], tot_loss[loss=0.07734, simple_loss=0.09796, pruned_loss=0.0182, audio_tagging_loss=0.01015, over 3055332.65 frames. ], batch size: 54, lr: 4.32e-03, grad_scale: 32.0 2023-11-20 21:51:41,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1234360.0, ans=0.125 2023-11-20 21:51:42,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1234360.0, ans=0.2 2023-11-20 21:51:55,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1234426.6666666667, ans=0.0 2023-11-20 21:51:58,550 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.531e+01 9.078e+01 1.027e+02 1.282e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-20 21:52:09,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1234493.3333333333, ans=0.125 2023-11-20 21:52:26,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1234560.0, ans=0.07 2023-11-20 21:52:35,879 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185200 2023-11-20 21:52:44,114 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4850, loss[loss=0.07491, simple_loss=0.08598, pruned_loss=0.01859, audio_tagging_loss=0.01333, over 14624.00 frames. ], tot_loss[loss=0.0774, simple_loss=0.09809, pruned_loss=0.01814, audio_tagging_loss=0.01022, over 3054621.87 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 21:52:58,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1234760.0, ans=0.125 2023-11-20 21:52:59,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1234760.0, ans=0.1 2023-11-20 21:53:06,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1234760.0, ans=0.125 2023-11-20 21:53:17,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1234826.6666666667, ans=0.1 2023-11-20 21:53:31,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1234893.3333333333, ans=0.0 2023-11-20 21:53:37,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1234960.0, ans=0.2 2023-11-20 21:53:40,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185250 2023-11-20 21:53:43,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1234960.0, ans=0.125 2023-11-20 21:53:47,771 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4900, loss[loss=0.07614, simple_loss=0.1015, pruned_loss=0.01596, audio_tagging_loss=0.009423, over 15559.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09813, pruned_loss=0.0181, audio_tagging_loss=0.01014, over 3044404.78 frames. ], batch size: 55, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 21:54:06,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1235093.3333333333, ans=0.125 2023-11-20 21:54:06,864 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.089e+01 8.842e+01 9.821e+01 1.319e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-20 21:54:08,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1235093.3333333333, ans=0.05 2023-11-20 21:54:30,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1235226.6666666667, ans=0.1 2023-11-20 21:54:31,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1235226.6666666667, ans=0.0 2023-11-20 21:54:37,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1235293.3333333333, ans=0.125 2023-11-20 21:54:43,758 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185300 2023-11-20 21:54:52,112 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 4950, loss[loss=0.05697, simple_loss=0.06758, pruned_loss=0.01236, audio_tagging_loss=0.01083, over 14593.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09716, pruned_loss=0.01797, audio_tagging_loss=0.01001, over 3035311.37 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 21:54:58,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1235360.0, ans=0.1 2023-11-20 21:55:48,257 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185350 2023-11-20 21:55:49,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1235626.6666666667, ans=0.125 2023-11-20 21:55:55,732 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5000, loss[loss=0.06481, simple_loss=0.08047, pruned_loss=0.01317, audio_tagging_loss=0.01141, over 15904.00 frames. ], tot_loss[loss=0.07676, simple_loss=0.09783, pruned_loss=0.01809, audio_tagging_loss=0.009754, over 3037953.04 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 21:56:16,467 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.618e+01 7.784e+01 8.478e+01 9.107e+01 1.087e+02, threshold=1.696e+02, percent-clipped=0.0 2023-11-20 21:56:27,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1235826.6666666667, ans=0.0 2023-11-20 21:56:42,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1235893.3333333333, ans=0.0 2023-11-20 21:56:52,803 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185400 2023-11-20 21:56:54,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1235960.0, ans=0.2 2023-11-20 21:57:01,017 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5050, loss[loss=0.06561, simple_loss=0.08321, pruned_loss=0.01514, audio_tagging_loss=0.008861, over 15181.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09828, pruned_loss=0.01819, audio_tagging_loss=0.009693, over 3037999.90 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 8.0 2023-11-20 21:57:28,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1236160.0, ans=0.1 2023-11-20 21:57:32,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.56 vs. limit=15.0 2023-11-20 21:57:49,415 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.82 vs. limit=15.0 2023-11-20 21:57:57,609 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185450 2023-11-20 21:58:05,627 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5100, loss[loss=0.07601, simple_loss=0.08597, pruned_loss=0.02104, audio_tagging_loss=0.01198, over 16375.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09708, pruned_loss=0.01796, audio_tagging_loss=0.009666, over 3038623.61 frames. ], batch size: 63, lr: 4.31e-03, grad_scale: 8.0 2023-11-20 21:58:25,679 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.354e+01 9.032e+01 1.014e+02 3.240e+02, threshold=1.806e+02, percent-clipped=1.0 2023-11-20 21:58:31,938 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.06 vs. limit=10.0 2023-11-20 21:58:53,220 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 21:59:01,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1236626.6666666667, ans=0.1 2023-11-20 21:59:02,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185500 2023-11-20 21:59:09,305 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5150, loss[loss=0.07615, simple_loss=0.09665, pruned_loss=0.01799, audio_tagging_loss=0.009831, over 14376.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09744, pruned_loss=0.0181, audio_tagging_loss=0.00971, over 3042842.54 frames. ], batch size: 56, lr: 4.31e-03, grad_scale: 8.0 2023-11-20 21:59:25,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.49 vs. limit=15.0 2023-11-20 21:59:29,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1236760.0, ans=0.0 2023-11-20 21:59:33,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1236760.0, ans=0.0 2023-11-20 21:59:33,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1236760.0, ans=0.05 2023-11-20 21:59:42,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1236826.6666666667, ans=0.1 2023-11-20 21:59:54,101 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.68 vs. limit=15.0 2023-11-20 22:00:02,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1236960.0, ans=0.125 2023-11-20 22:00:05,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185550 2023-11-20 22:00:14,336 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5200, loss[loss=0.05049, simple_loss=0.04911, pruned_loss=0.01304, audio_tagging_loss=0.01289, over 15479.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09861, pruned_loss=0.01826, audio_tagging_loss=0.009645, over 3046012.60 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:00:14,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=1237026.6666666667, ans=0.2 2023-11-20 22:00:21,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.76 vs. limit=15.0 2023-11-20 22:00:28,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1237093.3333333333, ans=0.0 2023-11-20 22:00:34,949 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.910e+01 8.252e+01 8.979e+01 9.707e+01 1.443e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-20 22:00:46,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1237160.0, ans=0.0 2023-11-20 22:00:51,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1237226.6666666667, ans=0.125 2023-11-20 22:01:04,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1237293.3333333333, ans=0.2 2023-11-20 22:01:10,285 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185600 2023-11-20 22:01:18,566 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5250, loss[loss=0.1038, simple_loss=0.1278, pruned_loss=0.03067, audio_tagging_loss=0.009304, over 15947.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.1001, pruned_loss=0.01864, audio_tagging_loss=0.009602, over 3049429.68 frames. ], batch size: 56, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:01:27,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1237360.0, ans=0.0 2023-11-20 22:01:38,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1237426.6666666667, ans=0.125 2023-11-20 22:02:15,567 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185650 2023-11-20 22:02:20,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1237626.6666666667, ans=10.0 2023-11-20 22:02:22,622 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5300, loss[loss=0.07711, simple_loss=0.1006, pruned_loss=0.01755, audio_tagging_loss=0.009269, over 15731.00 frames. ], tot_loss[loss=0.0783, simple_loss=0.09996, pruned_loss=0.01865, audio_tagging_loss=0.009677, over 3048523.77 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:02:39,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1237760.0, ans=0.125 2023-11-20 22:02:42,487 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.578e+01 8.164e+01 8.663e+01 9.355e+01 1.569e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-20 22:03:18,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185700 2023-11-20 22:03:25,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1238026.6666666667, ans=0.125 2023-11-20 22:03:26,272 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5350, loss[loss=0.05324, simple_loss=0.05943, pruned_loss=0.01207, audio_tagging_loss=0.01145, over 14894.00 frames. ], tot_loss[loss=0.07834, simple_loss=0.1003, pruned_loss=0.01864, audio_tagging_loss=0.009559, over 3042000.77 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:03:28,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.whiten.whitening_limit, batch_count=1238026.6666666667, ans=15.0 2023-11-20 22:03:51,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1238160.0, ans=0.0 2023-11-20 22:03:55,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1238160.0, ans=0.125 2023-11-20 22:03:56,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1238160.0, ans=0.125 2023-11-20 22:03:59,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1238160.0, ans=0.2 2023-11-20 22:04:17,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.97 vs. limit=15.0 2023-11-20 22:04:22,999 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185750 2023-11-20 22:04:31,035 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5400, loss[loss=0.08153, simple_loss=0.104, pruned_loss=0.01786, audio_tagging_loss=0.01164, over 14456.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.09933, pruned_loss=0.01833, audio_tagging_loss=0.009558, over 3044703.63 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:04:42,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1238360.0, ans=0.1 2023-11-20 22:04:51,611 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.664e+01 7.937e+01 8.475e+01 9.191e+01 1.261e+02, threshold=1.695e+02, percent-clipped=0.0 2023-11-20 22:04:54,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1238426.6666666667, ans=0.125 2023-11-20 22:05:08,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.76 vs. limit=15.0 2023-11-20 22:05:28,231 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185800 2023-11-20 22:05:35,787 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5450, loss[loss=0.081, simple_loss=0.108, pruned_loss=0.01764, audio_tagging_loss=0.009356, over 14829.00 frames. ], tot_loss[loss=0.07821, simple_loss=0.09987, pruned_loss=0.01857, audio_tagging_loss=0.009707, over 3046684.89 frames. ], batch size: 58, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:05:39,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1238693.3333333333, ans=0.0 2023-11-20 22:05:43,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1238693.3333333333, ans=0.09899494936611666 2023-11-20 22:05:57,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1238760.0, ans=0.0 2023-11-20 22:06:06,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1238826.6666666667, ans=0.0 2023-11-20 22:06:17,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1238893.3333333333, ans=0.0 2023-11-20 22:06:22,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1238893.3333333333, ans=0.125 2023-11-20 22:06:28,499 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.58 vs. limit=12.0 2023-11-20 22:06:31,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185850 2023-11-20 22:06:38,729 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5500, loss[loss=0.08944, simple_loss=0.1243, pruned_loss=0.01928, audio_tagging_loss=0.007999, over 14900.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09934, pruned_loss=0.01832, audio_tagging_loss=0.009766, over 3050663.44 frames. ], batch size: 53, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:06:42,119 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 22:06:44,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.19 vs. limit=15.0 2023-11-20 22:06:47,413 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.01 vs. limit=10.0 2023-11-20 22:07:00,056 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.304e+01 8.468e+01 9.055e+01 9.983e+01 1.429e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-20 22:07:19,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1239226.6666666667, ans=0.1 2023-11-20 22:07:21,111 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.84 vs. limit=15.0 2023-11-20 22:07:21,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1239226.6666666667, ans=0.1 2023-11-20 22:07:35,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185900 2023-11-20 22:07:42,626 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5550, loss[loss=0.1031, simple_loss=0.1203, pruned_loss=0.03403, audio_tagging_loss=0.008937, over 14352.00 frames. ], tot_loss[loss=0.07851, simple_loss=0.1001, pruned_loss=0.01862, audio_tagging_loss=0.009863, over 3047073.39 frames. ], batch size: 54, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:08:05,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1239426.6666666667, ans=0.1 2023-11-20 22:08:05,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.89 vs. limit=12.0 2023-11-20 22:08:06,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1239426.6666666667, ans=0.1 2023-11-20 22:08:16,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1239493.3333333333, ans=0.125 2023-11-20 22:08:34,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1239626.6666666667, ans=0.125 2023-11-20 22:08:40,163 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 185950 2023-11-20 22:08:48,139 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5600, loss[loss=0.07131, simple_loss=0.08971, pruned_loss=0.0155, audio_tagging_loss=0.01095, over 15044.00 frames. ], tot_loss[loss=0.0789, simple_loss=0.1008, pruned_loss=0.01859, audio_tagging_loss=0.009915, over 3047848.48 frames. ], batch size: 54, lr: 4.31e-03, grad_scale: 32.0 2023-11-20 22:08:56,065 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.42 vs. limit=15.0 2023-11-20 22:09:08,126 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.915e+01 8.506e+01 9.381e+01 1.103e+02, threshold=1.701e+02, percent-clipped=0.0 2023-11-20 22:09:14,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1239826.6666666667, ans=0.04949747468305833 2023-11-20 22:09:31,252 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 22:09:43,273 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186000 2023-11-20 22:09:50,706 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5650, loss[loss=0.08163, simple_loss=0.108, pruned_loss=0.01827, audio_tagging_loss=0.009384, over 16223.00 frames. ], tot_loss[loss=0.07894, simple_loss=0.1007, pruned_loss=0.01867, audio_tagging_loss=0.009936, over 3042459.35 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 32.0 2023-11-20 22:10:06,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.31 vs. limit=15.0 2023-11-20 22:10:32,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1240226.6666666667, ans=0.125 2023-11-20 22:10:46,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1240293.3333333333, ans=0.07 2023-11-20 22:10:47,223 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186050 2023-11-20 22:10:54,513 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5700, loss[loss=0.06598, simple_loss=0.08498, pruned_loss=0.01527, audio_tagging_loss=0.00822, over 15024.00 frames. ], tot_loss[loss=0.07824, simple_loss=0.0995, pruned_loss=0.01853, audio_tagging_loss=0.009959, over 3048194.60 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0 2023-11-20 22:11:14,668 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 22:11:16,790 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.091e+01 8.750e+01 9.672e+01 1.332e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-20 22:11:17,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1240426.6666666667, ans=0.125 2023-11-20 22:11:20,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1240493.3333333333, ans=0.0 2023-11-20 22:11:44,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1240626.6666666667, ans=0.2 2023-11-20 22:11:51,020 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186100 2023-11-20 22:11:53,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1240626.6666666667, ans=0.2 2023-11-20 22:11:53,891 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.83 vs. limit=22.5 2023-11-20 22:11:58,994 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5750, loss[loss=0.08856, simple_loss=0.1163, pruned_loss=0.02098, audio_tagging_loss=0.009409, over 15621.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09914, pruned_loss=0.0184, audio_tagging_loss=0.009912, over 3045091.55 frames. ], batch size: 57, lr: 4.30e-03, grad_scale: 16.0 2023-11-20 22:12:01,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1240693.3333333333, ans=0.07 2023-11-20 22:12:16,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1240760.0, ans=0.07 2023-11-20 22:12:29,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1240826.6666666667, ans=0.125 2023-11-20 22:12:33,055 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 22:12:42,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1240893.3333333333, ans=0.125 2023-11-20 22:12:47,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1240893.3333333333, ans=0.2 2023-11-20 22:12:49,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1240960.0, ans=0.07 2023-11-20 22:12:51,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1240960.0, ans=0.1 2023-11-20 22:12:55,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186150 2023-11-20 22:12:55,772 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.44 vs. limit=15.0 2023-11-20 22:13:02,399 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5800, loss[loss=0.06078, simple_loss=0.07767, pruned_loss=0.01323, audio_tagging_loss=0.008719, over 15775.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09866, pruned_loss=0.01824, audio_tagging_loss=0.009807, over 3047699.57 frames. ], batch size: 60, lr: 4.30e-03, grad_scale: 16.0 2023-11-20 22:13:02,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1241026.6666666667, ans=0.0 2023-11-20 22:13:09,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1241026.6666666667, ans=0.05 2023-11-20 22:13:19,710 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.59 vs. limit=15.0 2023-11-20 22:13:22,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1241093.3333333333, ans=0.125 2023-11-20 22:13:23,633 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.344e+01 8.039e+01 8.914e+01 9.519e+01 1.369e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-20 22:13:29,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1241160.0, ans=0.0 2023-11-20 22:13:35,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1241160.0, ans=0.0 2023-11-20 22:13:40,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1241226.6666666667, ans=0.2 2023-11-20 22:13:41,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1241226.6666666667, ans=0.125 2023-11-20 22:13:51,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1241226.6666666667, ans=0.125 2023-11-20 22:13:58,805 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186200 2023-11-20 22:14:06,327 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5850, loss[loss=0.09153, simple_loss=0.1163, pruned_loss=0.02356, audio_tagging_loss=0.009818, over 16096.00 frames. ], tot_loss[loss=0.07683, simple_loss=0.0978, pruned_loss=0.01806, audio_tagging_loss=0.009864, over 3047177.20 frames. ], batch size: 60, lr: 4.30e-03, grad_scale: 16.0 2023-11-20 22:14:12,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1241360.0, ans=0.1 2023-11-20 22:14:23,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1241426.6666666667, ans=0.125 2023-11-20 22:14:34,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1241493.3333333333, ans=0.2 2023-11-20 22:14:41,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1241493.3333333333, ans=0.2 2023-11-20 22:14:43,137 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.41 vs. limit=12.0 2023-11-20 22:14:49,756 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.78 vs. limit=15.0 2023-11-20 22:14:50,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1241560.0, ans=0.1 2023-11-20 22:14:59,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.51 vs. limit=15.0 2023-11-20 22:15:02,757 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186250 2023-11-20 22:15:11,180 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5900, loss[loss=0.07546, simple_loss=0.0862, pruned_loss=0.02197, audio_tagging_loss=0.0104, over 15005.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09788, pruned_loss=0.01798, audio_tagging_loss=0.009767, over 3043775.58 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 16.0 2023-11-20 22:15:13,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1241693.3333333333, ans=0.0 2023-11-20 22:15:28,449 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.74 vs. limit=5.0 2023-11-20 22:15:32,174 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.032e+01 8.653e+01 9.726e+01 1.327e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-20 22:15:34,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.05 vs. limit=10.0 2023-11-20 22:15:34,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1241826.6666666667, ans=0.09899494936611666 2023-11-20 22:15:45,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1241826.6666666667, ans=0.125 2023-11-20 22:16:06,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186300 2023-11-20 22:16:14,548 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 5950, loss[loss=0.08093, simple_loss=0.1056, pruned_loss=0.02047, audio_tagging_loss=0.007677, over 15723.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.09746, pruned_loss=0.01785, audio_tagging_loss=0.009707, over 3040729.26 frames. ], batch size: 59, lr: 4.30e-03, grad_scale: 16.0 2023-11-20 22:16:47,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1242160.0, ans=0.125 2023-11-20 22:16:53,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1242226.6666666667, ans=0.125 2023-11-20 22:17:01,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1242226.6666666667, ans=0.0 2023-11-20 22:17:01,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1242226.6666666667, ans=0.035 2023-11-20 22:17:11,497 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186350 2023-11-20 22:17:14,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1242293.3333333333, ans=0.0 2023-11-20 22:17:19,222 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6000, loss[loss=0.08702, simple_loss=0.1167, pruned_loss=0.01884, audio_tagging_loss=0.009813, over 15754.00 frames. ], tot_loss[loss=0.07661, simple_loss=0.09775, pruned_loss=0.01804, audio_tagging_loss=0.009699, over 3043332.30 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:17:19,222 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 22:18:03,606 INFO [train_asr.py:1253] (3/4) Epoch 16, validation: loss=0.06177, simple_loss=0.05296, pruned_loss=0.005445, audio_tagging_loss=0.02985, over 4681554.00 frames. 2023-11-20 22:18:03,607 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 22:18:15,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1242426.6666666667, ans=0.1 2023-11-20 22:18:16,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1242426.6666666667, ans=0.0 2023-11-20 22:18:25,411 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.385e+01 8.060e+01 8.625e+01 9.633e+01 1.979e+02, threshold=1.725e+02, percent-clipped=1.0 2023-11-20 22:18:29,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1242493.3333333333, ans=0.0 2023-11-20 22:18:32,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1242493.3333333333, ans=0.1 2023-11-20 22:18:45,332 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.44 vs. limit=15.0 2023-11-20 22:18:47,831 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 22:18:53,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1242626.6666666667, ans=0.125 2023-11-20 22:18:59,514 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186400 2023-11-20 22:19:07,779 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6050, loss[loss=0.08789, simple_loss=0.1031, pruned_loss=0.02544, audio_tagging_loss=0.01091, over 16119.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09798, pruned_loss=0.018, audio_tagging_loss=0.009706, over 3045486.11 frames. ], batch size: 61, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:19:10,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1242693.3333333333, ans=0.125 2023-11-20 22:19:10,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1242693.3333333333, ans=0.2 2023-11-20 22:19:32,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1242826.6666666667, ans=0.0 2023-11-20 22:19:49,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1242893.3333333333, ans=0.125 2023-11-20 22:19:57,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1242893.3333333333, ans=0.0 2023-11-20 22:19:59,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.38 vs. limit=15.0 2023-11-20 22:20:04,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186450 2023-11-20 22:20:07,936 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.03 vs. limit=15.0 2023-11-20 22:20:12,056 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6100, loss[loss=0.091, simple_loss=0.1143, pruned_loss=0.02391, audio_tagging_loss=0.009948, over 16026.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09825, pruned_loss=0.01818, audio_tagging_loss=0.009615, over 3046139.22 frames. ], batch size: 61, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:20:23,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1243093.3333333333, ans=0.05 2023-11-20 22:20:29,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1243093.3333333333, ans=0.125 2023-11-20 22:20:34,065 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.005e+01 8.728e+01 9.355e+01 1.254e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-20 22:20:52,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1243226.6666666667, ans=0.04949747468305833 2023-11-20 22:21:06,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1243293.3333333333, ans=0.2 2023-11-20 22:21:08,393 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186500 2023-11-20 22:21:16,643 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6150, loss[loss=0.09312, simple_loss=0.1211, pruned_loss=0.02446, audio_tagging_loss=0.008113, over 14171.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09699, pruned_loss=0.01789, audio_tagging_loss=0.009809, over 3046110.26 frames. ], batch size: 54, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:22:12,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186550 2023-11-20 22:22:19,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1243693.3333333333, ans=0.04949747468305833 2023-11-20 22:22:20,054 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6200, loss[loss=0.05138, simple_loss=0.05303, pruned_loss=0.009948, audio_tagging_loss=0.01492, over 15053.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.0966, pruned_loss=0.0176, audio_tagging_loss=0.009907, over 3048139.32 frames. ], batch size: 59, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:22:42,088 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.113e+01 8.838e+01 9.519e+01 1.384e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-20 22:22:48,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.90 vs. limit=15.0 2023-11-20 22:23:17,810 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186600 2023-11-20 22:23:25,502 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6250, loss[loss=0.05712, simple_loss=0.07136, pruned_loss=0.01069, audio_tagging_loss=0.01076, over 13709.00 frames. ], tot_loss[loss=0.07574, simple_loss=0.09639, pruned_loss=0.01757, audio_tagging_loss=0.009975, over 3042993.01 frames. ], batch size: 53, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:23:37,095 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.24 vs. limit=22.5 2023-11-20 22:23:41,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1244093.3333333333, ans=0.0 2023-11-20 22:23:42,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1244093.3333333333, ans=0.025 2023-11-20 22:23:51,724 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.66 vs. limit=22.5 2023-11-20 22:24:04,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1244226.6666666667, ans=0.0 2023-11-20 22:24:05,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1244226.6666666667, ans=0.125 2023-11-20 22:24:08,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1244226.6666666667, ans=0.125 2023-11-20 22:24:09,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1244226.6666666667, ans=0.0 2023-11-20 22:24:20,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1244293.3333333333, ans=0.125 2023-11-20 22:24:21,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186650 2023-11-20 22:24:21,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1244293.3333333333, ans=0.2 2023-11-20 22:24:25,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1244293.3333333333, ans=0.0 2023-11-20 22:24:29,276 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6300, loss[loss=0.07225, simple_loss=0.08226, pruned_loss=0.01815, audio_tagging_loss=0.01298, over 16254.00 frames. ], tot_loss[loss=0.07667, simple_loss=0.09775, pruned_loss=0.01785, audio_tagging_loss=0.009936, over 3046714.32 frames. ], batch size: 62, lr: 4.30e-03, grad_scale: 16.0 2023-11-20 22:24:36,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1244360.0, ans=0.0 2023-11-20 22:24:49,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1244426.6666666667, ans=0.0 2023-11-20 22:24:51,660 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.323e+01 8.066e+01 8.583e+01 9.288e+01 1.245e+02, threshold=1.717e+02, percent-clipped=0.0 2023-11-20 22:25:25,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186700 2023-11-20 22:25:32,755 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6350, loss[loss=0.06939, simple_loss=0.08583, pruned_loss=0.0159, audio_tagging_loss=0.01057, over 14678.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09722, pruned_loss=0.01797, audio_tagging_loss=0.01008, over 3038853.38 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 16.0 2023-11-20 22:25:40,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1244693.3333333333, ans=0.1 2023-11-20 22:25:41,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1244693.3333333333, ans=0.0 2023-11-20 22:26:09,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1244826.6666666667, ans=0.125 2023-11-20 22:26:20,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1244893.3333333333, ans=0.1 2023-11-20 22:26:22,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.55 vs. limit=15.0 2023-11-20 22:26:29,158 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186750 2023-11-20 22:26:32,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1244960.0, ans=0.125 2023-11-20 22:26:37,569 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6400, loss[loss=0.08029, simple_loss=0.1046, pruned_loss=0.01836, audio_tagging_loss=0.009644, over 14659.00 frames. ], tot_loss[loss=0.07754, simple_loss=0.09831, pruned_loss=0.01832, audio_tagging_loss=0.01007, over 3038973.08 frames. ], batch size: 54, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:26:37,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1245026.6666666667, ans=0.125 2023-11-20 22:26:42,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1245026.6666666667, ans=0.0 2023-11-20 22:26:53,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1245093.3333333333, ans=0.125 2023-11-20 22:26:54,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1245093.3333333333, ans=0.2 2023-11-20 22:26:55,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1245093.3333333333, ans=0.1 2023-11-20 22:27:00,829 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.359e+01 8.103e+01 8.559e+01 9.205e+01 1.089e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-20 22:27:25,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1245226.6666666667, ans=0.125 2023-11-20 22:27:34,042 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186800 2023-11-20 22:27:41,525 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6450, loss[loss=0.07604, simple_loss=0.1037, pruned_loss=0.01583, audio_tagging_loss=0.008338, over 16624.00 frames. ], tot_loss[loss=0.07806, simple_loss=0.09897, pruned_loss=0.01857, audio_tagging_loss=0.01001, over 3040256.98 frames. ], batch size: 63, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:27:50,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1245360.0, ans=0.125 2023-11-20 22:27:58,897 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 22:28:10,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1245493.3333333333, ans=0.1 2023-11-20 22:28:30,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1245560.0, ans=0.125 2023-11-20 22:28:39,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186850 2023-11-20 22:28:45,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1245693.3333333333, ans=0.1 2023-11-20 22:28:45,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1245693.3333333333, ans=10.0 2023-11-20 22:28:46,231 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6500, loss[loss=0.0661, simple_loss=0.08037, pruned_loss=0.01653, audio_tagging_loss=0.009384, over 15294.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.0996, pruned_loss=0.0185, audio_tagging_loss=0.00991, over 3040854.39 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:28:57,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1245760.0, ans=0.125 2023-11-20 22:29:09,944 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.106e+01 8.759e+01 9.505e+01 1.385e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-20 22:29:32,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1245893.3333333333, ans=0.05 2023-11-20 22:29:37,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1245960.0, ans=0.125 2023-11-20 22:29:38,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.16 vs. limit=15.0 2023-11-20 22:29:42,945 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186900 2023-11-20 22:29:50,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1246026.6666666667, ans=0.2 2023-11-20 22:29:50,896 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6550, loss[loss=0.07215, simple_loss=0.08904, pruned_loss=0.01502, audio_tagging_loss=0.01261, over 14328.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09901, pruned_loss=0.01848, audio_tagging_loss=0.009945, over 3041300.98 frames. ], batch size: 55, lr: 4.30e-03, grad_scale: 32.0 2023-11-20 22:30:20,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.63 vs. limit=15.0 2023-11-20 22:30:26,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1246160.0, ans=0.125 2023-11-20 22:30:37,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1246226.6666666667, ans=0.0 2023-11-20 22:30:40,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.13 vs. limit=10.0 2023-11-20 22:30:48,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 186950 2023-11-20 22:30:55,609 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6600, loss[loss=0.09475, simple_loss=0.1249, pruned_loss=0.02527, audio_tagging_loss=0.00702, over 15989.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09848, pruned_loss=0.01823, audio_tagging_loss=0.009785, over 3036079.48 frames. ], batch size: 60, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:30:58,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1246360.0, ans=0.125 2023-11-20 22:31:17,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1246426.6666666667, ans=0.125 2023-11-20 22:31:18,838 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.367e+01 8.854e+01 9.653e+01 1.222e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-20 22:31:52,862 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187000 2023-11-20 22:32:00,542 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6650, loss[loss=0.07756, simple_loss=0.1039, pruned_loss=0.01773, audio_tagging_loss=0.007882, over 15557.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09841, pruned_loss=0.01814, audio_tagging_loss=0.009744, over 3030165.88 frames. ], batch size: 57, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:32:09,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.50 vs. limit=6.0 2023-11-20 22:32:24,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1246826.6666666667, ans=0.125 2023-11-20 22:32:45,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1246893.3333333333, ans=0.1 2023-11-20 22:32:50,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1246960.0, ans=0.125 2023-11-20 22:32:56,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187050 2023-11-20 22:33:04,405 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6700, loss[loss=0.07011, simple_loss=0.08885, pruned_loss=0.01814, audio_tagging_loss=0.007538, over 15782.00 frames. ], tot_loss[loss=0.07725, simple_loss=0.09866, pruned_loss=0.01819, audio_tagging_loss=0.009727, over 3028326.73 frames. ], batch size: 58, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:33:15,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1247093.3333333333, ans=0.2 2023-11-20 22:33:23,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=1247093.3333333333, ans=0.05 2023-11-20 22:33:27,691 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.055e+01 8.769e+01 9.493e+01 1.327e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-20 22:33:40,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1247160.0, ans=0.0 2023-11-20 22:33:56,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1247293.3333333333, ans=0.125 2023-11-20 22:34:00,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187100 2023-11-20 22:34:05,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1247293.3333333333, ans=0.2 2023-11-20 22:34:08,177 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6750, loss[loss=0.08409, simple_loss=0.1056, pruned_loss=0.01936, audio_tagging_loss=0.01195, over 15502.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09861, pruned_loss=0.01826, audio_tagging_loss=0.009727, over 3026270.95 frames. ], batch size: 57, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:34:09,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1247360.0, ans=0.0 2023-11-20 22:34:11,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1247360.0, ans=0.125 2023-11-20 22:34:19,071 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0 2023-11-20 22:35:05,412 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187150 2023-11-20 22:35:13,208 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6800, loss[loss=0.08263, simple_loss=0.1114, pruned_loss=0.02047, audio_tagging_loss=0.006448, over 14946.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.0994, pruned_loss=0.01851, audio_tagging_loss=0.009657, over 3026025.18 frames. ], batch size: 54, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:35:16,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1247693.3333333333, ans=10.0 2023-11-20 22:35:29,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1247760.0, ans=0.0 2023-11-20 22:35:34,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1247760.0, ans=0.125 2023-11-20 22:35:35,482 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.835e+01 8.011e+01 8.725e+01 9.590e+01 1.326e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-20 22:35:52,931 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.48 vs. limit=15.0 2023-11-20 22:36:08,981 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187200 2023-11-20 22:36:16,478 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6850, loss[loss=0.06715, simple_loss=0.09064, pruned_loss=0.01316, audio_tagging_loss=0.00868, over 15648.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.0986, pruned_loss=0.01819, audio_tagging_loss=0.009621, over 3035356.51 frames. ], batch size: 59, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:36:21,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1248026.6666666667, ans=0.125 2023-11-20 22:36:40,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1248160.0, ans=0.1 2023-11-20 22:36:48,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.98 vs. limit=15.0 2023-11-20 22:37:12,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187250 2023-11-20 22:37:13,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1248293.3333333333, ans=0.0 2023-11-20 22:37:20,304 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6900, loss[loss=0.07193, simple_loss=0.09634, pruned_loss=0.01524, audio_tagging_loss=0.008521, over 15872.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.09917, pruned_loss=0.01817, audio_tagging_loss=0.009609, over 3040119.46 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:37:29,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1248360.0, ans=0.0 2023-11-20 22:37:44,215 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.341e+01 8.921e+01 9.872e+01 1.364e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-20 22:38:06,841 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 22:38:16,572 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187300 2023-11-20 22:38:25,001 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 6950, loss[loss=0.08516, simple_loss=0.1024, pruned_loss=0.02587, audio_tagging_loss=0.008079, over 14253.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09954, pruned_loss=0.0184, audio_tagging_loss=0.009531, over 3039684.57 frames. ], batch size: 54, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:38:51,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1248826.6666666667, ans=0.125 2023-11-20 22:39:01,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1248893.3333333333, ans=0.125 2023-11-20 22:39:04,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1248893.3333333333, ans=0.09899494936611666 2023-11-20 22:39:16,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.60 vs. limit=15.0 2023-11-20 22:39:17,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1248960.0, ans=0.0 2023-11-20 22:39:21,196 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187350 2023-11-20 22:39:23,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1248960.0, ans=0.0 2023-11-20 22:39:27,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1249026.6666666667, ans=0.125 2023-11-20 22:39:28,597 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7000, loss[loss=0.06334, simple_loss=0.07865, pruned_loss=0.01177, audio_tagging_loss=0.01225, over 14760.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.0986, pruned_loss=0.01821, audio_tagging_loss=0.009659, over 3035392.81 frames. ], batch size: 55, lr: 4.29e-03, grad_scale: 16.0 2023-11-20 22:39:39,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1249026.6666666667, ans=0.125 2023-11-20 22:39:46,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1249093.3333333333, ans=0.125 2023-11-20 22:39:47,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1249093.3333333333, ans=0.125 2023-11-20 22:39:49,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1249093.3333333333, ans=0.1 2023-11-20 22:39:52,393 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.148e+01 8.784e+01 9.512e+01 1.267e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-20 22:39:55,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1249160.0, ans=0.0 2023-11-20 22:39:55,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.45 vs. limit=15.0 2023-11-20 22:40:25,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187400 2023-11-20 22:40:32,866 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.95 vs. limit=10.0 2023-11-20 22:40:33,236 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7050, loss[loss=0.07206, simple_loss=0.09703, pruned_loss=0.01606, audio_tagging_loss=0.00748, over 15260.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09827, pruned_loss=0.01818, audio_tagging_loss=0.009699, over 3034550.02 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 16.0 2023-11-20 22:40:34,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1249360.0, ans=0.025 2023-11-20 22:40:48,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1249426.6666666667, ans=0.1 2023-11-20 22:41:25,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1249626.6666666667, ans=0.1 2023-11-20 22:41:29,407 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187450 2023-11-20 22:41:37,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1249693.3333333333, ans=0.1 2023-11-20 22:41:37,950 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7100, loss[loss=0.09015, simple_loss=0.1098, pruned_loss=0.02629, audio_tagging_loss=0.008945, over 13499.00 frames. ], tot_loss[loss=0.07765, simple_loss=0.09889, pruned_loss=0.01849, audio_tagging_loss=0.009712, over 3034496.10 frames. ], batch size: 53, lr: 4.29e-03, grad_scale: 16.0 2023-11-20 22:41:43,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1249693.3333333333, ans=0.2 2023-11-20 22:41:54,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.61 vs. limit=15.0 2023-11-20 22:42:02,250 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.191e+01 8.839e+01 9.583e+01 1.103e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-20 22:42:03,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1249826.6666666667, ans=0.1 2023-11-20 22:42:06,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1249826.6666666667, ans=0.125 2023-11-20 22:42:13,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1249826.6666666667, ans=0.0 2023-11-20 22:42:19,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1249893.3333333333, ans=0.125 2023-11-20 22:42:34,631 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187500 2023-11-20 22:42:34,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1249960.0, ans=0.125 2023-11-20 22:42:41,836 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7150, loss[loss=0.07077, simple_loss=0.08473, pruned_loss=0.01863, audio_tagging_loss=0.009769, over 15303.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09897, pruned_loss=0.01832, audio_tagging_loss=0.009788, over 3040864.98 frames. ], batch size: 60, lr: 4.29e-03, grad_scale: 16.0 2023-11-20 22:42:52,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1250026.6666666667, ans=0.125 2023-11-20 22:43:16,758 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 22:43:29,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1250226.6666666667, ans=0.1 2023-11-20 22:43:32,474 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0 2023-11-20 22:43:38,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187550 2023-11-20 22:43:45,996 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7200, loss[loss=0.08501, simple_loss=0.1068, pruned_loss=0.02294, audio_tagging_loss=0.008677, over 14539.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09812, pruned_loss=0.01807, audio_tagging_loss=0.009914, over 3035826.76 frames. ], batch size: 55, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:43:46,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1250360.0, ans=0.0 2023-11-20 22:44:09,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1250426.6666666667, ans=0.2 2023-11-20 22:44:10,057 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.608e+01 8.172e+01 8.869e+01 9.437e+01 2.740e+02, threshold=1.774e+02, percent-clipped=1.0 2023-11-20 22:44:14,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1250493.3333333333, ans=0.125 2023-11-20 22:44:22,009 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 22:44:39,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1250626.6666666667, ans=0.125 2023-11-20 22:44:42,012 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187600 2023-11-20 22:44:45,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1250626.6666666667, ans=0.07 2023-11-20 22:44:50,282 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7250, loss[loss=0.09144, simple_loss=0.1158, pruned_loss=0.02427, audio_tagging_loss=0.009285, over 14832.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09762, pruned_loss=0.01804, audio_tagging_loss=0.00999, over 3035979.93 frames. ], batch size: 53, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:45:01,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1250760.0, ans=0.1 2023-11-20 22:45:35,340 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.55 vs. limit=15.0 2023-11-20 22:45:47,068 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187650 2023-11-20 22:45:51,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1250960.0, ans=0.125 2023-11-20 22:45:55,045 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7300, loss[loss=0.08989, simple_loss=0.1188, pruned_loss=0.02383, audio_tagging_loss=0.006682, over 15151.00 frames. ], tot_loss[loss=0.07712, simple_loss=0.09801, pruned_loss=0.01817, audio_tagging_loss=0.009937, over 3041199.44 frames. ], batch size: 55, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:46:18,767 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 8.202e+01 8.886e+01 9.738e+01 1.326e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-20 22:46:25,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1251160.0, ans=0.125 2023-11-20 22:46:30,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1251160.0, ans=0.125 2023-11-20 22:46:36,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1251226.6666666667, ans=0.0 2023-11-20 22:46:36,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1251226.6666666667, ans=0.1 2023-11-20 22:46:51,315 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187700 2023-11-20 22:46:59,071 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7350, loss[loss=0.06024, simple_loss=0.08224, pruned_loss=0.01073, audio_tagging_loss=0.008396, over 14537.00 frames. ], tot_loss[loss=0.07689, simple_loss=0.09799, pruned_loss=0.01807, audio_tagging_loss=0.00982, over 3036397.64 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:47:06,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1251360.0, ans=0.0 2023-11-20 22:47:14,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.17 vs. limit=15.0 2023-11-20 22:47:33,791 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.50 vs. limit=15.0 2023-11-20 22:47:38,022 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.25 vs. limit=10.0 2023-11-20 22:47:41,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1251560.0, ans=0.125 2023-11-20 22:47:55,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187750 2023-11-20 22:48:02,464 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7400, loss[loss=0.0691, simple_loss=0.08935, pruned_loss=0.01589, audio_tagging_loss=0.008536, over 14499.00 frames. ], tot_loss[loss=0.07655, simple_loss=0.09777, pruned_loss=0.01795, audio_tagging_loss=0.009722, over 3041896.29 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 32.0 2023-11-20 22:48:03,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1251693.3333333333, ans=0.125 2023-11-20 22:48:26,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1251760.0, ans=0.0 2023-11-20 22:48:27,600 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.213e+01 8.738e+01 9.709e+01 1.431e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-20 22:48:44,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1251893.3333333333, ans=0.125 2023-11-20 22:48:46,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1251893.3333333333, ans=0.2 2023-11-20 22:48:49,504 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.47 vs. limit=22.5 2023-11-20 22:48:54,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1251960.0, ans=0.125 2023-11-20 22:48:55,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1251960.0, ans=0.125 2023-11-20 22:48:59,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187800 2023-11-20 22:49:00,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1251960.0, ans=0.125 2023-11-20 22:49:06,803 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7450, loss[loss=0.06368, simple_loss=0.07719, pruned_loss=0.01395, audio_tagging_loss=0.01114, over 16657.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09799, pruned_loss=0.0179, audio_tagging_loss=0.009703, over 3039985.97 frames. ], batch size: 64, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:49:51,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1252226.6666666667, ans=0.125 2023-11-20 22:49:54,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1252226.6666666667, ans=0.0 2023-11-20 22:50:01,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=1252293.3333333333, ans=0.05 2023-11-20 22:50:02,357 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187850 2023-11-20 22:50:05,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1252293.3333333333, ans=0.0 2023-11-20 22:50:10,843 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7500, loss[loss=0.04937, simple_loss=0.06133, pruned_loss=0.00928, audio_tagging_loss=0.00943, over 15555.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09728, pruned_loss=0.01779, audio_tagging_loss=0.009683, over 3039012.30 frames. ], batch size: 62, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:50:14,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1252360.0, ans=0.0 2023-11-20 22:50:25,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1252426.6666666667, ans=0.1 2023-11-20 22:50:34,837 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.198e+01 8.879e+01 9.496e+01 1.257e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-20 22:50:40,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1252493.3333333333, ans=0.125 2023-11-20 22:50:49,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1252560.0, ans=0.125 2023-11-20 22:50:56,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1252560.0, ans=0.125 2023-11-20 22:51:04,792 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.28 vs. limit=15.0 2023-11-20 22:51:06,691 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187900 2023-11-20 22:51:09,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1252626.6666666667, ans=0.0 2023-11-20 22:51:09,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1252626.6666666667, ans=0.04949747468305833 2023-11-20 22:51:13,960 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7550, loss[loss=0.09183, simple_loss=0.1232, pruned_loss=0.01961, audio_tagging_loss=0.01064, over 16472.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09817, pruned_loss=0.01802, audio_tagging_loss=0.009574, over 3041288.85 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:51:23,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1252693.3333333333, ans=0.125 2023-11-20 22:52:10,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 187950 2023-11-20 22:52:11,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1252960.0, ans=0.125 2023-11-20 22:52:12,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1252960.0, ans=0.0 2023-11-20 22:52:17,970 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7600, loss[loss=0.07635, simple_loss=0.1031, pruned_loss=0.01743, audio_tagging_loss=0.007351, over 15403.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09767, pruned_loss=0.01789, audio_tagging_loss=0.009546, over 3040923.72 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:52:30,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1253093.3333333333, ans=0.05 2023-11-20 22:52:39,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1253093.3333333333, ans=0.125 2023-11-20 22:52:42,481 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.162e+01 8.819e+01 9.727e+01 1.348e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-20 22:52:43,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.24 vs. limit=15.0 2023-11-20 22:52:49,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1253160.0, ans=0.125 2023-11-20 22:53:00,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1253226.6666666667, ans=0.125 2023-11-20 22:53:06,637 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.63 vs. limit=10.0 2023-11-20 22:53:14,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188000 2023-11-20 22:53:25,793 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7650, loss[loss=0.07637, simple_loss=0.09802, pruned_loss=0.01701, audio_tagging_loss=0.01035, over 16052.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09768, pruned_loss=0.01789, audio_tagging_loss=0.009521, over 3043337.51 frames. ], batch size: 59, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:53:26,310 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.71 vs. limit=15.0 2023-11-20 22:53:29,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1253360.0, ans=0.0 2023-11-20 22:53:38,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1253426.6666666667, ans=0.125 2023-11-20 22:53:39,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1253426.6666666667, ans=0.125 2023-11-20 22:53:50,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1253426.6666666667, ans=0.2 2023-11-20 22:54:10,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1253560.0, ans=0.09899494936611666 2023-11-20 22:54:10,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.00 vs. limit=22.5 2023-11-20 22:54:19,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1253626.6666666667, ans=0.0 2023-11-20 22:54:22,931 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188050 2023-11-20 22:54:26,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1253626.6666666667, ans=0.025 2023-11-20 22:54:30,083 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7700, loss[loss=0.08578, simple_loss=0.1221, pruned_loss=0.01924, audio_tagging_loss=0.005484, over 15472.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09713, pruned_loss=0.0177, audio_tagging_loss=0.009564, over 3036589.79 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:54:30,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.47 vs. limit=15.0 2023-11-20 22:54:54,496 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.666e+01 7.984e+01 8.508e+01 9.260e+01 1.144e+02, threshold=1.702e+02, percent-clipped=0.0 2023-11-20 22:55:04,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.74 vs. limit=22.5 2023-11-20 22:55:26,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188100 2023-11-20 22:55:34,509 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7750, loss[loss=0.05477, simple_loss=0.06291, pruned_loss=0.01189, audio_tagging_loss=0.01142, over 14833.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.0974, pruned_loss=0.01777, audio_tagging_loss=0.009636, over 3034374.74 frames. ], batch size: 59, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:56:07,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1254160.0, ans=0.05 2023-11-20 22:56:07,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1254160.0, ans=0.0 2023-11-20 22:56:30,404 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188150 2023-11-20 22:56:33,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1254293.3333333333, ans=0.125 2023-11-20 22:56:36,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1254360.0, ans=0.0 2023-11-20 22:56:37,337 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.18 vs. limit=10.0 2023-11-20 22:56:37,564 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7800, loss[loss=0.08625, simple_loss=0.1135, pruned_loss=0.02094, audio_tagging_loss=0.008576, over 15243.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09745, pruned_loss=0.01777, audio_tagging_loss=0.009605, over 3036441.84 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:57:00,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.14 vs. limit=15.0 2023-11-20 22:57:01,921 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.318e+01 8.003e+01 8.886e+01 9.530e+01 1.591e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-20 22:57:07,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1254493.3333333333, ans=0.07 2023-11-20 22:57:19,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1254560.0, ans=0.2 2023-11-20 22:57:25,585 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 22:57:26,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1254560.0, ans=0.1 2023-11-20 22:57:34,354 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188200 2023-11-20 22:57:34,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1254626.6666666667, ans=0.125 2023-11-20 22:57:38,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1254626.6666666667, ans=0.0 2023-11-20 22:57:42,547 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7850, loss[loss=0.0976, simple_loss=0.1171, pruned_loss=0.02583, audio_tagging_loss=0.0132, over 15557.00 frames. ], tot_loss[loss=0.07722, simple_loss=0.09883, pruned_loss=0.0181, audio_tagging_loss=0.009699, over 3042670.12 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:57:57,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1254760.0, ans=0.05 2023-11-20 22:58:10,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1254826.6666666667, ans=0.125 2023-11-20 22:58:21,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.68 vs. limit=15.0 2023-11-20 22:58:39,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188250 2023-11-20 22:58:40,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1254960.0, ans=0.125 2023-11-20 22:58:47,245 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7900, loss[loss=0.07833, simple_loss=0.1083, pruned_loss=0.01603, audio_tagging_loss=0.008149, over 15867.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09891, pruned_loss=0.01821, audio_tagging_loss=0.009823, over 3041337.65 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:58:47,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.53 vs. limit=22.5 2023-11-20 22:58:57,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1255026.6666666667, ans=0.09899494936611666 2023-11-20 22:58:59,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1255093.3333333333, ans=0.0 2023-11-20 22:59:10,911 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.310e+01 8.933e+01 9.750e+01 1.293e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-20 22:59:16,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1255160.0, ans=0.0 2023-11-20 22:59:42,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1255293.3333333333, ans=0.125 2023-11-20 22:59:43,352 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188300 2023-11-20 22:59:50,526 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 7950, loss[loss=0.0598, simple_loss=0.06875, pruned_loss=0.01179, audio_tagging_loss=0.01363, over 15132.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09858, pruned_loss=0.01803, audio_tagging_loss=0.009967, over 3038565.00 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 22:59:52,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0 2023-11-20 22:59:54,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1255360.0, ans=0.0 2023-11-20 22:59:57,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1255360.0, ans=0.1 2023-11-20 23:00:04,580 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:00:10,020 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0 2023-11-20 23:00:27,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1255493.3333333333, ans=0.125 2023-11-20 23:00:28,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1255560.0, ans=0.125 2023-11-20 23:00:40,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1255626.6666666667, ans=0.2 2023-11-20 23:00:44,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1255626.6666666667, ans=0.0 2023-11-20 23:00:46,728 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188350 2023-11-20 23:00:54,430 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8000, loss[loss=0.1138, simple_loss=0.1418, pruned_loss=0.03386, audio_tagging_loss=0.009054, over 15935.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09826, pruned_loss=0.01804, audio_tagging_loss=0.01004, over 3035033.78 frames. ], batch size: 59, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 23:00:54,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1255693.3333333333, ans=0.0 2023-11-20 23:01:13,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1255760.0, ans=0.125 2023-11-20 23:01:19,518 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.786e+01 8.042e+01 8.522e+01 9.463e+01 1.266e+02, threshold=1.704e+02, percent-clipped=0.0 2023-11-20 23:01:48,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1255960.0, ans=0.125 2023-11-20 23:01:50,577 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188400 2023-11-20 23:01:59,870 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8050, loss[loss=0.08829, simple_loss=0.1184, pruned_loss=0.02409, audio_tagging_loss=0.005021, over 15830.00 frames. ], tot_loss[loss=0.07724, simple_loss=0.09825, pruned_loss=0.01813, audio_tagging_loss=0.009985, over 3038573.00 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 23:02:17,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1256093.3333333333, ans=0.07 2023-11-20 23:02:49,515 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.53 vs. limit=12.0 2023-11-20 23:02:56,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188450 2023-11-20 23:03:03,276 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8100, loss[loss=0.07639, simple_loss=0.09706, pruned_loss=0.01992, audio_tagging_loss=0.007939, over 15009.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09823, pruned_loss=0.01816, audio_tagging_loss=0.009879, over 3029115.01 frames. ], batch size: 59, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 23:03:16,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1256426.6666666667, ans=0.0 2023-11-20 23:03:25,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1256426.6666666667, ans=10.0 2023-11-20 23:03:26,967 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.476e+01 8.237e+01 9.309e+01 1.004e+02 1.229e+02, threshold=1.862e+02, percent-clipped=0.0 2023-11-20 23:03:37,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1256493.3333333333, ans=0.0 2023-11-20 23:03:43,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1256560.0, ans=0.125 2023-11-20 23:03:52,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0 2023-11-20 23:03:58,840 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188500 2023-11-20 23:04:06,655 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8150, loss[loss=0.06569, simple_loss=0.08024, pruned_loss=0.01274, audio_tagging_loss=0.01283, over 15507.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09863, pruned_loss=0.01818, audio_tagging_loss=0.00987, over 3030695.38 frames. ], batch size: 60, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 23:04:15,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1256693.3333333333, ans=0.125 2023-11-20 23:04:34,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1256826.6666666667, ans=0.125 2023-11-20 23:04:52,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1256893.3333333333, ans=0.1 2023-11-20 23:05:00,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1256960.0, ans=22.5 2023-11-20 23:05:02,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188550 2023-11-20 23:05:08,389 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:05:09,560 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8200, loss[loss=0.07913, simple_loss=0.1032, pruned_loss=0.02151, audio_tagging_loss=0.00599, over 15386.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09955, pruned_loss=0.01831, audio_tagging_loss=0.009741, over 3032721.52 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 23:05:23,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1257093.3333333333, ans=0.125 2023-11-20 23:05:29,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.73 vs. limit=15.0 2023-11-20 23:05:34,613 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.637e+01 8.183e+01 8.865e+01 9.614e+01 1.183e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-20 23:05:43,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1257160.0, ans=0.125 2023-11-20 23:05:53,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1257226.6666666667, ans=0.0 2023-11-20 23:05:53,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1257226.6666666667, ans=0.0 2023-11-20 23:06:07,404 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188600 2023-11-20 23:06:15,101 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8250, loss[loss=0.08585, simple_loss=0.1197, pruned_loss=0.01855, audio_tagging_loss=0.007429, over 15788.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09942, pruned_loss=0.01815, audio_tagging_loss=0.009605, over 3035745.68 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 23:06:50,288 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.14 vs. limit=12.0 2023-11-20 23:06:59,552 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.54 vs. limit=22.5 2023-11-20 23:07:11,214 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188650 2023-11-20 23:07:18,995 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8300, loss[loss=0.06104, simple_loss=0.08026, pruned_loss=0.009908, audio_tagging_loss=0.011, over 15137.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09891, pruned_loss=0.01804, audio_tagging_loss=0.009545, over 3038077.18 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0 2023-11-20 23:07:35,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1257760.0, ans=0.1 2023-11-20 23:07:43,191 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.445e+01 8.113e+01 8.858e+01 9.502e+01 1.553e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-20 23:07:57,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1257893.3333333333, ans=0.125 2023-11-20 23:08:15,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188700 2023-11-20 23:08:22,653 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8350, loss[loss=0.07378, simple_loss=0.09873, pruned_loss=0.0149, audio_tagging_loss=0.009513, over 14374.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.1, pruned_loss=0.0182, audio_tagging_loss=0.009432, over 3032103.30 frames. ], batch size: 52, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:08:30,512 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.99 vs. limit=10.0 2023-11-20 23:08:46,809 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:08:48,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1258160.0, ans=0.125 2023-11-20 23:08:56,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1258160.0, ans=0.1 2023-11-20 23:09:09,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.04 vs. limit=15.0 2023-11-20 23:09:11,727 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.49 vs. limit=15.0 2023-11-20 23:09:14,704 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.96 vs. limit=15.0 2023-11-20 23:09:19,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188750 2023-11-20 23:09:27,483 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8400, loss[loss=0.08094, simple_loss=0.1117, pruned_loss=0.01791, audio_tagging_loss=0.007167, over 14599.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.1001, pruned_loss=0.01823, audio_tagging_loss=0.009522, over 3039871.02 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:09:33,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1258360.0, ans=0.0 2023-11-20 23:09:34,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1258360.0, ans=0.125 2023-11-20 23:09:49,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1258426.6666666667, ans=0.1 2023-11-20 23:09:51,126 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.077e+01 8.855e+01 9.404e+01 1.397e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-20 23:10:09,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.03 vs. limit=22.5 2023-11-20 23:10:23,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188800 2023-11-20 23:10:31,310 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8450, loss[loss=0.06891, simple_loss=0.09001, pruned_loss=0.01554, audio_tagging_loss=0.008364, over 14629.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.0997, pruned_loss=0.01824, audio_tagging_loss=0.009609, over 3046504.21 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:10:46,475 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.56 vs. limit=6.0 2023-11-20 23:10:57,388 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.24 vs. limit=15.0 2023-11-20 23:11:20,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1258893.3333333333, ans=0.125 2023-11-20 23:11:27,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188850 2023-11-20 23:11:33,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1259026.6666666667, ans=0.1 2023-11-20 23:11:33,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1259026.6666666667, ans=0.125 2023-11-20 23:11:34,538 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8500, loss[loss=0.0826, simple_loss=0.1071, pruned_loss=0.02022, audio_tagging_loss=0.008812, over 15556.00 frames. ], tot_loss[loss=0.07838, simple_loss=0.1005, pruned_loss=0.01856, audio_tagging_loss=0.00959, over 3052126.66 frames. ], batch size: 59, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:11:53,687 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=22.16 vs. limit=15.0 2023-11-20 23:11:59,747 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.153e+01 8.872e+01 9.707e+01 1.209e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-20 23:12:03,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1259160.0, ans=0.07 2023-11-20 23:12:11,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1259160.0, ans=0.07 2023-11-20 23:12:17,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1259226.6666666667, ans=0.125 2023-11-20 23:12:27,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1259293.3333333333, ans=0.0 2023-11-20 23:12:31,308 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188900 2023-11-20 23:12:39,223 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8550, loss[loss=0.07407, simple_loss=0.09749, pruned_loss=0.01543, audio_tagging_loss=0.009901, over 16101.00 frames. ], tot_loss[loss=0.07782, simple_loss=0.0996, pruned_loss=0.01838, audio_tagging_loss=0.00964, over 3049012.76 frames. ], batch size: 61, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:12:39,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1259360.0, ans=0.125 2023-11-20 23:13:07,322 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:13:07,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1259493.3333333333, ans=0.2 2023-11-20 23:13:13,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1259493.3333333333, ans=0.2 2023-11-20 23:13:24,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1259560.0, ans=0.0 2023-11-20 23:13:29,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1259626.6666666667, ans=0.125 2023-11-20 23:13:34,999 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 188950 2023-11-20 23:13:42,790 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8600, loss[loss=0.07363, simple_loss=0.08744, pruned_loss=0.02, audio_tagging_loss=0.009907, over 15295.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.1005, pruned_loss=0.01867, audio_tagging_loss=0.009699, over 3044347.29 frames. ], batch size: 58, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:13:56,089 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0 2023-11-20 23:14:06,713 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.038e+01 8.664e+01 9.287e+01 1.205e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-20 23:14:15,451 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.79 vs. limit=15.0 2023-11-20 23:14:25,622 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.90 vs. limit=12.0 2023-11-20 23:14:27,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1259893.3333333333, ans=0.125 2023-11-20 23:14:29,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1259893.3333333333, ans=0.125 2023-11-20 23:14:39,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189000 2023-11-20 23:14:43,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1259960.0, ans=0.0 2023-11-20 23:14:46,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1260026.6666666667, ans=0.0 2023-11-20 23:14:47,238 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8650, loss[loss=0.08752, simple_loss=0.1232, pruned_loss=0.01823, audio_tagging_loss=0.007676, over 15080.00 frames. ], tot_loss[loss=0.07911, simple_loss=0.1011, pruned_loss=0.01883, audio_tagging_loss=0.009751, over 3040996.43 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:14:57,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1260026.6666666667, ans=0.125 2023-11-20 23:15:27,857 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:15:39,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1260293.3333333333, ans=0.025 2023-11-20 23:15:43,548 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189050 2023-11-20 23:15:46,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1260293.3333333333, ans=0.125 2023-11-20 23:15:52,064 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8700, loss[loss=0.09067, simple_loss=0.1148, pruned_loss=0.02554, audio_tagging_loss=0.007742, over 14787.00 frames. ], tot_loss[loss=0.07881, simple_loss=0.1004, pruned_loss=0.01875, audio_tagging_loss=0.009835, over 3035109.71 frames. ], batch size: 57, lr: 4.27e-03, grad_scale: 16.0 2023-11-20 23:15:58,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1260360.0, ans=0.125 2023-11-20 23:16:00,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.27 vs. limit=22.5 2023-11-20 23:16:01,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1260360.0, ans=0.125 2023-11-20 23:16:17,581 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.100e+01 8.533e+01 9.341e+01 1.224e+02, threshold=1.707e+02, percent-clipped=0.0 2023-11-20 23:16:20,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1260493.3333333333, ans=0.0 2023-11-20 23:16:33,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1260560.0, ans=15.0 2023-11-20 23:16:46,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0 2023-11-20 23:16:48,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189100 2023-11-20 23:16:52,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.80 vs. limit=10.0 2023-11-20 23:16:55,219 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8750, loss[loss=0.05953, simple_loss=0.07945, pruned_loss=0.009986, audio_tagging_loss=0.009813, over 15270.00 frames. ], tot_loss[loss=0.0787, simple_loss=0.1004, pruned_loss=0.01862, audio_tagging_loss=0.009907, over 3040888.68 frames. ], batch size: 59, lr: 4.27e-03, grad_scale: 16.0 2023-11-20 23:17:08,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1260760.0, ans=0.125 2023-11-20 23:17:35,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1260893.3333333333, ans=0.125 2023-11-20 23:17:36,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1260893.3333333333, ans=0.125 2023-11-20 23:17:51,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189150 2023-11-20 23:17:57,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1261026.6666666667, ans=0.125 2023-11-20 23:17:58,622 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8800, loss[loss=0.08141, simple_loss=0.1081, pruned_loss=0.01641, audio_tagging_loss=0.01093, over 14049.00 frames. ], tot_loss[loss=0.07958, simple_loss=0.1017, pruned_loss=0.01886, audio_tagging_loss=0.009886, over 3041760.77 frames. ], batch size: 54, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:18:23,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1261160.0, ans=0.0 2023-11-20 23:18:24,225 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.201e+01 8.889e+01 9.657e+01 1.880e+02, threshold=1.778e+02, percent-clipped=1.0 2023-11-20 23:18:47,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1261226.6666666667, ans=10.0 2023-11-20 23:18:48,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1261293.3333333333, ans=0.0 2023-11-20 23:18:49,155 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.68 vs. limit=22.5 2023-11-20 23:18:53,895 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.12 vs. limit=15.0 2023-11-20 23:18:54,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189200 2023-11-20 23:19:03,173 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8850, loss[loss=0.05609, simple_loss=0.06348, pruned_loss=0.0129, audio_tagging_loss=0.01144, over 15732.00 frames. ], tot_loss[loss=0.07934, simple_loss=0.1011, pruned_loss=0.01878, audio_tagging_loss=0.009988, over 3046227.46 frames. ], batch size: 60, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:19:12,997 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:19:16,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1261426.6666666667, ans=0.125 2023-11-20 23:19:16,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1261426.6666666667, ans=0.125 2023-11-20 23:19:33,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1261493.3333333333, ans=22.5 2023-11-20 23:19:58,742 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189250 2023-11-20 23:20:03,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1261626.6666666667, ans=0.0 2023-11-20 23:20:04,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.30 vs. limit=15.0 2023-11-20 23:20:05,956 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8900, loss[loss=0.08433, simple_loss=0.1019, pruned_loss=0.02254, audio_tagging_loss=0.01085, over 13561.00 frames. ], tot_loss[loss=0.07871, simple_loss=0.1005, pruned_loss=0.01855, audio_tagging_loss=0.00991, over 3050806.37 frames. ], batch size: 54, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:20:16,849 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.39 vs. limit=15.0 2023-11-20 23:20:17,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1261760.0, ans=0.2 2023-11-20 23:20:21,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1261760.0, ans=0.125 2023-11-20 23:20:31,205 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.227e+01 8.677e+01 9.378e+01 1.174e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-20 23:20:47,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1261893.3333333333, ans=0.1 2023-11-20 23:21:01,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189300 2023-11-20 23:21:09,626 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 8950, loss[loss=0.08544, simple_loss=0.1141, pruned_loss=0.019, audio_tagging_loss=0.009367, over 15127.00 frames. ], tot_loss[loss=0.07801, simple_loss=0.09993, pruned_loss=0.01829, audio_tagging_loss=0.00976, over 3051955.54 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:21:13,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1262026.6666666667, ans=0.1 2023-11-20 23:21:13,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1262026.6666666667, ans=0.125 2023-11-20 23:21:18,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1262026.6666666667, ans=0.0 2023-11-20 23:21:19,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1262026.6666666667, ans=0.125 2023-11-20 23:21:32,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1262093.3333333333, ans=0.125 2023-11-20 23:21:35,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1262160.0, ans=0.125 2023-11-20 23:21:42,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1262160.0, ans=0.125 2023-11-20 23:21:55,102 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.97 vs. limit=6.0 2023-11-20 23:21:56,533 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.35 vs. limit=15.0 2023-11-20 23:22:02,251 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.94 vs. limit=15.0 2023-11-20 23:22:05,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189350 2023-11-20 23:22:07,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1262293.3333333333, ans=0.125 2023-11-20 23:22:13,048 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9000, loss[loss=0.09187, simple_loss=0.1209, pruned_loss=0.02211, audio_tagging_loss=0.009307, over 14696.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09996, pruned_loss=0.01826, audio_tagging_loss=0.009664, over 3052956.79 frames. ], batch size: 53, lr: 4.27e-03, grad_scale: 16.0 2023-11-20 23:22:13,048 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-20 23:22:55,288 INFO [train_asr.py:1253] (3/4) Epoch 16, validation: loss=0.06115, simple_loss=0.05296, pruned_loss=0.005511, audio_tagging_loss=0.02916, over 4681554.00 frames. 2023-11-20 23:22:55,289 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-20 23:23:22,726 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.262e+01 9.175e+01 9.933e+01 1.311e+02, threshold=1.835e+02, percent-clipped=0.0 2023-11-20 23:23:26,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1262493.3333333333, ans=0.125 2023-11-20 23:23:33,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1262560.0, ans=0.2 2023-11-20 23:23:39,456 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:23:51,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189400 2023-11-20 23:23:56,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1262626.6666666667, ans=0.125 2023-11-20 23:23:59,686 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9050, loss[loss=0.08292, simple_loss=0.1056, pruned_loss=0.02111, audio_tagging_loss=0.009018, over 15363.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.09981, pruned_loss=0.01821, audio_tagging_loss=0.009685, over 3060396.21 frames. ], batch size: 57, lr: 4.27e-03, grad_scale: 16.0 2023-11-20 23:24:09,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1262693.3333333333, ans=0.0 2023-11-20 23:24:22,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1262760.0, ans=0.125 2023-11-20 23:24:46,304 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.34 vs. limit=15.0 2023-11-20 23:24:56,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189450 2023-11-20 23:25:04,469 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9100, loss[loss=0.0891, simple_loss=0.1202, pruned_loss=0.02172, audio_tagging_loss=0.007285, over 15633.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09909, pruned_loss=0.0182, audio_tagging_loss=0.009644, over 3060642.87 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 16.0 2023-11-20 23:25:17,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1263093.3333333333, ans=0.125 2023-11-20 23:25:27,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1263093.3333333333, ans=0.125 2023-11-20 23:25:30,560 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.518e+01 8.183e+01 8.786e+01 9.651e+01 1.253e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-20 23:25:33,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.97 vs. limit=10.0 2023-11-20 23:26:00,246 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189500 2023-11-20 23:26:07,501 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9150, loss[loss=0.0778, simple_loss=0.1021, pruned_loss=0.01645, audio_tagging_loss=0.0103, over 14502.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.0985, pruned_loss=0.01822, audio_tagging_loss=0.009625, over 3060017.67 frames. ], batch size: 56, lr: 4.27e-03, grad_scale: 16.0 2023-11-20 23:26:26,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1263426.6666666667, ans=0.2 2023-11-20 23:26:27,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1263426.6666666667, ans=0.0 2023-11-20 23:26:40,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=12.0 2023-11-20 23:27:03,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189550 2023-11-20 23:27:05,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1263626.6666666667, ans=0.0 2023-11-20 23:27:10,411 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9200, loss[loss=0.08019, simple_loss=0.09946, pruned_loss=0.01975, audio_tagging_loss=0.01071, over 15248.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09941, pruned_loss=0.01845, audio_tagging_loss=0.00959, over 3052739.80 frames. ], batch size: 58, lr: 4.27e-03, grad_scale: 32.0 2023-11-20 23:27:29,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1263760.0, ans=0.1 2023-11-20 23:27:31,834 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.33 vs. limit=15.0 2023-11-20 23:27:33,141 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.92 vs. limit=15.0 2023-11-20 23:27:37,273 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.197e+01 8.151e+01 8.702e+01 9.490e+01 1.171e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-20 23:27:42,826 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.33 vs. limit=22.5 2023-11-20 23:28:06,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189600 2023-11-20 23:28:08,238 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=22.5 2023-11-20 23:28:14,509 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9250, loss[loss=0.08633, simple_loss=0.1128, pruned_loss=0.02191, audio_tagging_loss=0.008013, over 15342.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09943, pruned_loss=0.01859, audio_tagging_loss=0.009524, over 3063660.50 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 32.0 2023-11-20 23:28:34,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1264093.3333333333, ans=0.125 2023-11-20 23:28:44,511 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:29:10,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189650 2023-11-20 23:29:18,276 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9300, loss[loss=0.08407, simple_loss=0.1103, pruned_loss=0.02085, audio_tagging_loss=0.008084, over 15219.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09938, pruned_loss=0.01854, audio_tagging_loss=0.009629, over 3060040.81 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 32.0 2023-11-20 23:29:37,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1264426.6666666667, ans=0.0 2023-11-20 23:29:44,507 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.124e+01 8.662e+01 9.573e+01 1.521e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-20 23:29:49,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1264493.3333333333, ans=0.0 2023-11-20 23:29:52,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1264493.3333333333, ans=0.1 2023-11-20 23:29:54,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1264493.3333333333, ans=0.2 2023-11-20 23:29:56,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1264560.0, ans=0.0 2023-11-20 23:30:14,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189700 2023-11-20 23:30:22,387 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9350, loss[loss=0.05493, simple_loss=0.06676, pruned_loss=0.01381, audio_tagging_loss=0.007749, over 15695.00 frames. ], tot_loss[loss=0.07766, simple_loss=0.09899, pruned_loss=0.01851, audio_tagging_loss=0.009652, over 3058225.10 frames. ], batch size: 59, lr: 4.26e-03, grad_scale: 32.0 2023-11-20 23:30:47,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1264826.6666666667, ans=0.125 2023-11-20 23:30:52,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1264826.6666666667, ans=0.1 2023-11-20 23:30:56,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1264826.6666666667, ans=0.1 2023-11-20 23:30:59,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1264893.3333333333, ans=0.125 2023-11-20 23:31:14,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1264960.0, ans=0.1 2023-11-20 23:31:17,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1264960.0, ans=0.125 2023-11-20 23:31:18,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189750 2023-11-20 23:31:26,047 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9400, loss[loss=0.06669, simple_loss=0.08869, pruned_loss=0.01226, audio_tagging_loss=0.01009, over 14993.00 frames. ], tot_loss[loss=0.07797, simple_loss=0.09935, pruned_loss=0.01855, audio_tagging_loss=0.009741, over 3058600.19 frames. ], batch size: 56, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:31:26,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1265026.6666666667, ans=0.0 2023-11-20 23:31:27,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1265026.6666666667, ans=0.125 2023-11-20 23:31:32,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1265026.6666666667, ans=0.125 2023-11-20 23:31:42,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1265093.3333333333, ans=0.0 2023-11-20 23:31:48,359 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:31:51,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1265160.0, ans=0.125 2023-11-20 23:31:54,704 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.398e+01 8.103e+01 8.759e+01 9.361e+01 1.221e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-20 23:31:59,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1265160.0, ans=0.1 2023-11-20 23:32:23,007 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189800 2023-11-20 23:32:27,755 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:32:31,398 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9450, loss[loss=0.07935, simple_loss=0.1038, pruned_loss=0.01652, audio_tagging_loss=0.01091, over 15701.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09881, pruned_loss=0.01841, audio_tagging_loss=0.009806, over 3056879.21 frames. ], batch size: 59, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:32:41,528 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:33:01,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1265493.3333333333, ans=0.0 2023-11-20 23:33:19,714 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.04 vs. limit=15.0 2023-11-20 23:33:27,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189850 2023-11-20 23:33:27,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1265626.6666666667, ans=0.125 2023-11-20 23:33:34,927 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9500, loss[loss=0.06423, simple_loss=0.08124, pruned_loss=0.01303, audio_tagging_loss=0.01058, over 15172.00 frames. ], tot_loss[loss=0.0775, simple_loss=0.0987, pruned_loss=0.01828, audio_tagging_loss=0.009861, over 3050251.85 frames. ], batch size: 59, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:33:42,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1265693.3333333333, ans=0.125 2023-11-20 23:34:02,678 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.779e+01 8.467e+01 9.097e+01 1.010e+02 1.655e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-20 23:34:23,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1265893.3333333333, ans=0.125 2023-11-20 23:34:30,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189900 2023-11-20 23:34:31,402 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.04 vs. limit=15.0 2023-11-20 23:34:32,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1265960.0, ans=0.0 2023-11-20 23:34:32,549 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.28 vs. limit=15.0 2023-11-20 23:34:38,071 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9550, loss[loss=0.08544, simple_loss=0.1079, pruned_loss=0.01983, audio_tagging_loss=0.01167, over 16070.00 frames. ], tot_loss[loss=0.07703, simple_loss=0.09803, pruned_loss=0.01806, audio_tagging_loss=0.009955, over 3040069.61 frames. ], batch size: 58, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:34:49,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1266026.6666666667, ans=0.2 2023-11-20 23:35:23,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1266226.6666666667, ans=0.125 2023-11-20 23:35:23,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1266226.6666666667, ans=0.1 2023-11-20 23:35:24,865 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.26 vs. limit=22.5 2023-11-20 23:35:25,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1266226.6666666667, ans=0.1 2023-11-20 23:35:33,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1266293.3333333333, ans=0.0 2023-11-20 23:35:35,266 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 189950 2023-11-20 23:35:40,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.61 vs. limit=15.0 2023-11-20 23:35:42,335 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9600, loss[loss=0.05874, simple_loss=0.06058, pruned_loss=0.01284, audio_tagging_loss=0.0156, over 14829.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.09868, pruned_loss=0.0182, audio_tagging_loss=0.009922, over 3040665.06 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 32.0 2023-11-20 23:35:58,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1266426.6666666667, ans=0.1 2023-11-20 23:36:10,700 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.084e+01 8.487e+01 9.415e+01 1.113e+02, threshold=1.697e+02, percent-clipped=0.0 2023-11-20 23:36:26,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1266560.0, ans=0.0 2023-11-20 23:36:26,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1266560.0, ans=0.0 2023-11-20 23:36:26,985 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.43 vs. limit=15.0 2023-11-20 23:36:34,369 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.60 vs. limit=15.0 2023-11-20 23:36:35,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1266626.6666666667, ans=0.2 2023-11-20 23:36:39,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190000 2023-11-20 23:36:47,524 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9650, loss[loss=0.07986, simple_loss=0.09807, pruned_loss=0.01956, audio_tagging_loss=0.01127, over 16253.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09848, pruned_loss=0.01823, audio_tagging_loss=0.009985, over 3038247.64 frames. ], batch size: 58, lr: 4.26e-03, grad_scale: 32.0 2023-11-20 23:36:48,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1266693.3333333333, ans=0.125 2023-11-20 23:36:51,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1266693.3333333333, ans=0.125 2023-11-20 23:36:55,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0 2023-11-20 23:37:22,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1266826.6666666667, ans=0.0 2023-11-20 23:37:24,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1266893.3333333333, ans=0.04949747468305833 2023-11-20 23:37:30,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1266893.3333333333, ans=0.0 2023-11-20 23:37:44,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190050 2023-11-20 23:37:51,305 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9700, loss[loss=0.08141, simple_loss=0.1028, pruned_loss=0.02313, audio_tagging_loss=0.006896, over 16016.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09815, pruned_loss=0.01816, audio_tagging_loss=0.009802, over 3045235.41 frames. ], batch size: 61, lr: 4.26e-03, grad_scale: 32.0 2023-11-20 23:37:52,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1267026.6666666667, ans=0.0 2023-11-20 23:38:19,500 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.47 vs. limit=22.5 2023-11-20 23:38:20,893 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.48 vs. limit=15.0 2023-11-20 23:38:21,185 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.452e+01 8.015e+01 8.951e+01 9.775e+01 1.301e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-20 23:38:48,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190100 2023-11-20 23:38:56,265 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9750, loss[loss=0.07315, simple_loss=0.1024, pruned_loss=0.01281, audio_tagging_loss=0.009144, over 14433.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09759, pruned_loss=0.01793, audio_tagging_loss=0.009695, over 3045893.19 frames. ], batch size: 55, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:39:16,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.40 vs. limit=15.0 2023-11-20 23:39:16,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.44 vs. limit=15.0 2023-11-20 23:39:17,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1267426.6666666667, ans=0.125 2023-11-20 23:39:27,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1267493.3333333333, ans=0.025 2023-11-20 23:39:34,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0 2023-11-20 23:39:44,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1267560.0, ans=0.125 2023-11-20 23:39:52,894 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190150 2023-11-20 23:40:00,678 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9800, loss[loss=0.0745, simple_loss=0.08779, pruned_loss=0.01746, audio_tagging_loss=0.01315, over 14862.00 frames. ], tot_loss[loss=0.07631, simple_loss=0.09755, pruned_loss=0.01793, audio_tagging_loss=0.009606, over 3042343.16 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:40:07,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1267693.3333333333, ans=0.1 2023-11-20 23:40:14,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.49 vs. limit=15.0 2023-11-20 23:40:30,322 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.597e+01 7.945e+01 8.740e+01 9.752e+01 1.304e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-20 23:40:56,340 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:40:57,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190200 2023-11-20 23:41:05,211 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9850, loss[loss=0.08176, simple_loss=0.111, pruned_loss=0.0176, audio_tagging_loss=0.008656, over 14896.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09782, pruned_loss=0.01782, audio_tagging_loss=0.009545, over 3044163.56 frames. ], batch size: 55, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:41:50,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1268226.6666666667, ans=0.125 2023-11-20 23:42:00,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190250 2023-11-20 23:42:08,617 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9900, loss[loss=0.06709, simple_loss=0.07819, pruned_loss=0.01472, audio_tagging_loss=0.01328, over 15609.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09809, pruned_loss=0.01795, audio_tagging_loss=0.009515, over 3044733.82 frames. ], batch size: 59, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:42:12,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1268360.0, ans=0.125 2023-11-20 23:42:35,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1268493.3333333333, ans=0.125 2023-11-20 23:42:37,743 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.053e+01 8.764e+01 9.412e+01 1.307e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-20 23:42:51,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1268560.0, ans=0.0 2023-11-20 23:43:04,900 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190300 2023-11-20 23:43:06,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1268626.6666666667, ans=0.125 2023-11-20 23:43:10,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1268626.6666666667, ans=0.2 2023-11-20 23:43:12,086 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 9950, loss[loss=0.06212, simple_loss=0.0692, pruned_loss=0.01637, audio_tagging_loss=0.01115, over 14285.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09741, pruned_loss=0.01772, audio_tagging_loss=0.009605, over 3047055.54 frames. ], batch size: 55, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:43:31,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1268760.0, ans=0.0 2023-11-20 23:43:33,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0 2023-11-20 23:43:37,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1268826.6666666667, ans=0.125 2023-11-20 23:43:47,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1268826.6666666667, ans=0.0 2023-11-20 23:43:56,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1268893.3333333333, ans=0.07 2023-11-20 23:43:58,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1268893.3333333333, ans=0.025 2023-11-20 23:44:08,785 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190350 2023-11-20 23:44:16,751 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10000, loss[loss=0.09797, simple_loss=0.1353, pruned_loss=0.02342, audio_tagging_loss=0.006888, over 16626.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09888, pruned_loss=0.01818, audio_tagging_loss=0.009466, over 3047125.95 frames. ], batch size: 60, lr: 4.26e-03, grad_scale: 32.0 2023-11-20 23:44:17,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1269026.6666666667, ans=0.125 2023-11-20 23:44:21,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1269026.6666666667, ans=0.125 2023-11-20 23:44:34,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1269093.3333333333, ans=0.125 2023-11-20 23:44:35,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1269093.3333333333, ans=0.015 2023-11-20 23:44:45,925 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.173e+01 8.806e+01 9.776e+01 1.486e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-20 23:44:47,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1269160.0, ans=0.125 2023-11-20 23:44:48,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.53 vs. limit=15.0 2023-11-20 23:44:51,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1269160.0, ans=0.0 2023-11-20 23:44:55,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1269226.6666666667, ans=0.1 2023-11-20 23:45:03,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1269226.6666666667, ans=0.1 2023-11-20 23:45:13,489 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190400 2023-11-20 23:45:20,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1269360.0, ans=0.1 2023-11-20 23:45:21,701 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10050, loss[loss=0.06657, simple_loss=0.08831, pruned_loss=0.01184, audio_tagging_loss=0.01058, over 14775.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.09758, pruned_loss=0.0179, audio_tagging_loss=0.009589, over 3040434.66 frames. ], batch size: 56, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:45:29,814 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.66 vs. limit=22.5 2023-11-20 23:45:33,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1269426.6666666667, ans=0.125 2023-11-20 23:45:42,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1269426.6666666667, ans=0.05 2023-11-20 23:45:47,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1269493.3333333333, ans=0.2 2023-11-20 23:46:01,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1269560.0, ans=0.125 2023-11-20 23:46:01,939 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.69 vs. limit=15.0 2023-11-20 23:46:08,735 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=4.19 vs. limit=15.0 2023-11-20 23:46:17,965 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190450 2023-11-20 23:46:25,270 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10100, loss[loss=0.08435, simple_loss=0.1051, pruned_loss=0.02293, audio_tagging_loss=0.008869, over 14103.00 frames. ], tot_loss[loss=0.07703, simple_loss=0.09838, pruned_loss=0.01822, audio_tagging_loss=0.009624, over 3039511.62 frames. ], batch size: 53, lr: 4.26e-03, grad_scale: 16.0 2023-11-20 23:46:34,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=1269693.3333333333, ans=0.1 2023-11-20 23:46:50,908 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.33 vs. limit=10.0 2023-11-20 23:46:56,619 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.119e+01 8.749e+01 9.415e+01 1.269e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-20 23:47:02,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.94 vs. limit=22.5 2023-11-20 23:47:08,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.23 vs. limit=12.0 2023-11-20 23:47:10,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1269893.3333333333, ans=0.0 2023-11-20 23:47:15,147 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:47:22,279 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190500 2023-11-20 23:47:26,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1269960.0, ans=0.0 2023-11-20 23:47:28,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1270026.6666666667, ans=0.2 2023-11-20 23:47:29,417 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10150, loss[loss=0.06433, simple_loss=0.07813, pruned_loss=0.01492, audio_tagging_loss=0.01034, over 14712.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09852, pruned_loss=0.01833, audio_tagging_loss=0.009729, over 3042463.56 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:47:35,475 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.91 vs. limit=22.5 2023-11-20 23:47:37,859 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.36 vs. limit=22.5 2023-11-20 23:47:40,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1270026.6666666667, ans=0.125 2023-11-20 23:47:44,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1270093.3333333333, ans=0.2 2023-11-20 23:47:57,648 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:48:09,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1270226.6666666667, ans=0.07 2023-11-20 23:48:25,845 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190550 2023-11-20 23:48:29,682 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.799e-01 2023-11-20 23:48:33,568 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10200, loss[loss=0.08587, simple_loss=0.1114, pruned_loss=0.02318, audio_tagging_loss=0.006971, over 14951.00 frames. ], tot_loss[loss=0.07765, simple_loss=0.09888, pruned_loss=0.0184, audio_tagging_loss=0.009805, over 3049234.08 frames. ], batch size: 55, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:48:49,856 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.70 vs. limit=10.0 2023-11-20 23:48:50,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1270426.6666666667, ans=0.0 2023-11-20 23:48:54,073 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-20 23:48:54,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1270426.6666666667, ans=0.125 2023-11-20 23:49:00,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1270493.3333333333, ans=10.0 2023-11-20 23:49:02,934 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.808e+01 8.167e+01 8.853e+01 9.625e+01 1.935e+02, threshold=1.771e+02, percent-clipped=1.0 2023-11-20 23:49:05,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1270493.3333333333, ans=0.025 2023-11-20 23:49:10,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1270560.0, ans=0.04949747468305833 2023-11-20 23:49:16,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1270560.0, ans=0.2 2023-11-20 23:49:17,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1270560.0, ans=0.125 2023-11-20 23:49:22,135 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.19 vs. limit=15.0 2023-11-20 23:49:27,854 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.43 vs. limit=15.0 2023-11-20 23:49:28,628 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190600 2023-11-20 23:49:35,986 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10250, loss[loss=0.07976, simple_loss=0.098, pruned_loss=0.02101, audio_tagging_loss=0.009749, over 14737.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.0987, pruned_loss=0.01844, audio_tagging_loss=0.009926, over 3043743.36 frames. ], batch size: 54, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:49:37,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1270693.3333333333, ans=0.125 2023-11-20 23:49:48,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1270760.0, ans=0.1 2023-11-20 23:50:08,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1270826.6666666667, ans=0.2 2023-11-20 23:50:32,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190650 2023-11-20 23:50:34,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1270960.0, ans=0.125 2023-11-20 23:50:36,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1270960.0, ans=0.125 2023-11-20 23:50:36,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1270960.0, ans=0.2 2023-11-20 23:50:39,969 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10300, loss[loss=0.0848, simple_loss=0.1089, pruned_loss=0.02053, audio_tagging_loss=0.009848, over 14543.00 frames. ], tot_loss[loss=0.07801, simple_loss=0.09879, pruned_loss=0.01854, audio_tagging_loss=0.01007, over 3054452.82 frames. ], batch size: 55, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:50:54,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1271093.3333333333, ans=0.125 2023-11-20 23:51:05,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1271160.0, ans=0.04949747468305833 2023-11-20 23:51:11,231 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.201e+01 8.870e+01 9.848e+01 1.361e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-20 23:51:15,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1271160.0, ans=0.125 2023-11-20 23:51:36,413 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190700 2023-11-20 23:51:40,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.68 vs. limit=15.0 2023-11-20 23:51:44,194 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10350, loss[loss=0.07043, simple_loss=0.08757, pruned_loss=0.01675, audio_tagging_loss=0.009899, over 14135.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.09815, pruned_loss=0.01831, audio_tagging_loss=0.01016, over 3052660.54 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 8.0 2023-11-20 23:51:48,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1271360.0, ans=0.125 2023-11-20 23:51:48,316 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.29 vs. limit=12.0 2023-11-20 23:51:56,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1271426.6666666667, ans=0.07 2023-11-20 23:52:20,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1271493.3333333333, ans=0.125 2023-11-20 23:52:23,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1271560.0, ans=0.2 2023-11-20 23:52:28,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1271560.0, ans=0.5 2023-11-20 23:52:34,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1271626.6666666667, ans=0.125 2023-11-20 23:52:35,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1271626.6666666667, ans=0.1 2023-11-20 23:52:40,417 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190750 2023-11-20 23:52:41,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1271626.6666666667, ans=0.125 2023-11-20 23:52:41,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1271626.6666666667, ans=0.125 2023-11-20 23:52:45,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1271626.6666666667, ans=0.1 2023-11-20 23:52:46,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1271693.3333333333, ans=0.5 2023-11-20 23:52:47,644 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10400, loss[loss=0.07144, simple_loss=0.09224, pruned_loss=0.01571, audio_tagging_loss=0.009611, over 15708.00 frames. ], tot_loss[loss=0.07734, simple_loss=0.09791, pruned_loss=0.01815, audio_tagging_loss=0.01024, over 3054405.84 frames. ], batch size: 60, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:53:08,006 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:53:18,605 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.195e+01 8.838e+01 9.520e+01 1.388e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-20 23:53:20,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1271826.6666666667, ans=0.125 2023-11-20 23:53:43,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190800 2023-11-20 23:53:50,996 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10450, loss[loss=0.0677, simple_loss=0.09442, pruned_loss=0.01276, audio_tagging_loss=0.00773, over 14782.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.09883, pruned_loss=0.01842, audio_tagging_loss=0.01008, over 3055596.35 frames. ], batch size: 55, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:53:56,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.80 vs. limit=22.5 2023-11-20 23:54:10,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1272093.3333333333, ans=0.2 2023-11-20 23:54:17,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1272160.0, ans=0.125 2023-11-20 23:54:29,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1272226.6666666667, ans=0.0 2023-11-20 23:54:34,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1272226.6666666667, ans=0.1 2023-11-20 23:54:46,556 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190850 2023-11-20 23:54:53,833 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10500, loss[loss=0.07463, simple_loss=0.09865, pruned_loss=0.01555, audio_tagging_loss=0.009751, over 14991.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09835, pruned_loss=0.01819, audio_tagging_loss=0.00995, over 3052062.49 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:54:59,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1272360.0, ans=0.0 2023-11-20 23:55:06,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1272426.6666666667, ans=0.125 2023-11-20 23:55:08,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1272426.6666666667, ans=0.125 2023-11-20 23:55:12,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1272426.6666666667, ans=0.0 2023-11-20 23:55:26,073 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.127e+01 8.997e+01 9.660e+01 1.233e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-20 23:55:46,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1272626.6666666667, ans=0.125 2023-11-20 23:55:50,104 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190900 2023-11-20 23:55:57,816 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10550, loss[loss=0.087, simple_loss=0.1168, pruned_loss=0.01997, audio_tagging_loss=0.008621, over 15091.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09844, pruned_loss=0.01817, audio_tagging_loss=0.009761, over 3053945.73 frames. ], batch size: 56, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:56:18,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1272760.0, ans=0.125 2023-11-20 23:56:38,450 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:56:53,467 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 190950 2023-11-20 23:57:00,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1273026.6666666667, ans=0.125 2023-11-20 23:57:01,267 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10600, loss[loss=0.08074, simple_loss=0.1086, pruned_loss=0.01654, audio_tagging_loss=0.009902, over 15289.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.09845, pruned_loss=0.01811, audio_tagging_loss=0.00966, over 3056239.07 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:57:12,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1273093.3333333333, ans=0.025 2023-11-20 23:57:25,194 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-20 23:57:32,705 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 7.871e+01 8.482e+01 9.357e+01 1.125e+02, threshold=1.696e+02, percent-clipped=0.0 2023-11-20 23:57:34,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1273160.0, ans=0.07 2023-11-20 23:57:50,804 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.24 vs. limit=12.0 2023-11-20 23:57:52,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1273293.3333333333, ans=0.0 2023-11-20 23:57:56,264 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191000 2023-11-20 23:58:03,693 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10650, loss[loss=0.07826, simple_loss=0.09326, pruned_loss=0.01953, audio_tagging_loss=0.0121, over 15341.00 frames. ], tot_loss[loss=0.07727, simple_loss=0.09881, pruned_loss=0.01824, audio_tagging_loss=0.009624, over 3051971.93 frames. ], batch size: 56, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:58:22,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1273426.6666666667, ans=0.2 2023-11-20 23:58:28,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1273493.3333333333, ans=0.125 2023-11-20 23:59:00,050 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191050 2023-11-20 23:59:07,274 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10700, loss[loss=0.08234, simple_loss=0.09113, pruned_loss=0.02194, audio_tagging_loss=0.01483, over 14088.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.0984, pruned_loss=0.01812, audio_tagging_loss=0.009694, over 3048179.71 frames. ], batch size: 56, lr: 4.25e-03, grad_scale: 16.0 2023-11-20 23:59:14,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1273693.3333333333, ans=0.1 2023-11-20 23:59:23,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1273760.0, ans=0.0 2023-11-20 23:59:34,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1273826.6666666667, ans=0.0 2023-11-20 23:59:38,723 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 7.953e+01 8.635e+01 9.470e+01 1.499e+02, threshold=1.727e+02, percent-clipped=0.0 2023-11-20 23:59:47,582 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.55 vs. limit=22.5 2023-11-21 00:00:03,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191100 2023-11-21 00:00:10,888 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10750, loss[loss=0.05644, simple_loss=0.06341, pruned_loss=0.01248, audio_tagging_loss=0.01225, over 14087.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.09757, pruned_loss=0.01778, audio_tagging_loss=0.009667, over 3044727.73 frames. ], batch size: 55, lr: 4.25e-03, grad_scale: 16.0 2023-11-21 00:00:23,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1274093.3333333333, ans=0.05 2023-11-21 00:00:35,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1274160.0, ans=0.2 2023-11-21 00:00:38,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1274160.0, ans=0.1 2023-11-21 00:01:03,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1274293.3333333333, ans=0.125 2023-11-21 00:01:06,219 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191150 2023-11-21 00:01:13,357 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10800, loss[loss=0.07729, simple_loss=0.0967, pruned_loss=0.01914, audio_tagging_loss=0.009805, over 15857.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.0973, pruned_loss=0.01772, audio_tagging_loss=0.009732, over 3045022.39 frames. ], batch size: 59, lr: 4.25e-03, grad_scale: 32.0 2023-11-21 00:01:22,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1274360.0, ans=0.0 2023-11-21 00:01:27,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1274426.6666666667, ans=0.0 2023-11-21 00:01:29,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1274426.6666666667, ans=0.0 2023-11-21 00:01:37,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1274493.3333333333, ans=0.1 2023-11-21 00:01:37,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1274493.3333333333, ans=0.0 2023-11-21 00:01:44,981 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.243e+01 8.007e+01 8.718e+01 9.213e+01 1.138e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-21 00:01:49,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1274493.3333333333, ans=0.0 2023-11-21 00:02:08,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191200 2023-11-21 00:02:09,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1274626.6666666667, ans=0.125 2023-11-21 00:02:17,049 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10850, loss[loss=0.08051, simple_loss=0.1084, pruned_loss=0.0147, audio_tagging_loss=0.01162, over 15087.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09773, pruned_loss=0.01797, audio_tagging_loss=0.009757, over 3039540.50 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 32.0 2023-11-21 00:02:23,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1274693.3333333333, ans=0.125 2023-11-21 00:02:26,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1274693.3333333333, ans=0.0 2023-11-21 00:02:27,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1274693.3333333333, ans=0.0 2023-11-21 00:02:30,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1274760.0, ans=0.125 2023-11-21 00:02:32,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1274760.0, ans=0.125 2023-11-21 00:02:48,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1274826.6666666667, ans=0.0 2023-11-21 00:02:49,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1274826.6666666667, ans=0.1 2023-11-21 00:03:13,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191250 2023-11-21 00:03:15,989 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 00:03:20,512 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.50 vs. limit=15.0 2023-11-21 00:03:21,473 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10900, loss[loss=0.08537, simple_loss=0.1159, pruned_loss=0.02037, audio_tagging_loss=0.007039, over 14296.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.0988, pruned_loss=0.01815, audio_tagging_loss=0.009683, over 3041169.12 frames. ], batch size: 54, lr: 4.25e-03, grad_scale: 32.0 2023-11-21 00:03:35,808 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.64 vs. limit=15.0 2023-11-21 00:03:40,326 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:03:52,724 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.158e+01 8.756e+01 9.674e+01 1.348e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 00:03:54,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1275160.0, ans=0.0 2023-11-21 00:03:59,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.80 vs. limit=15.0 2023-11-21 00:04:02,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1275226.6666666667, ans=0.2 2023-11-21 00:04:16,659 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191300 2023-11-21 00:04:21,875 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.30 vs. limit=15.0 2023-11-21 00:04:23,769 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 10950, loss[loss=0.08706, simple_loss=0.1108, pruned_loss=0.02108, audio_tagging_loss=0.01056, over 15330.00 frames. ], tot_loss[loss=0.0769, simple_loss=0.09832, pruned_loss=0.01801, audio_tagging_loss=0.00973, over 3037414.60 frames. ], batch size: 56, lr: 4.25e-03, grad_scale: 32.0 2023-11-21 00:04:25,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1275360.0, ans=0.1 2023-11-21 00:04:58,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.55 vs. limit=15.0 2023-11-21 00:05:19,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191350 2023-11-21 00:05:21,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1275626.6666666667, ans=0.0 2023-11-21 00:05:27,996 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11000, loss[loss=0.06699, simple_loss=0.09563, pruned_loss=0.01117, audio_tagging_loss=0.007997, over 15652.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.0986, pruned_loss=0.01806, audio_tagging_loss=0.009813, over 3038573.69 frames. ], batch size: 59, lr: 4.25e-03, grad_scale: 32.0 2023-11-21 00:05:36,711 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 00:05:48,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1275760.0, ans=0.0 2023-11-21 00:05:59,925 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.037e+01 8.729e+01 9.395e+01 1.274e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 00:06:11,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1275893.3333333333, ans=0.125 2023-11-21 00:06:23,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=1275960.0, ans=0.1 2023-11-21 00:06:24,754 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191400 2023-11-21 00:06:32,581 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11050, loss[loss=0.06341, simple_loss=0.07748, pruned_loss=0.01413, audio_tagging_loss=0.01054, over 13886.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09882, pruned_loss=0.01822, audio_tagging_loss=0.009884, over 3038815.22 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 16.0 2023-11-21 00:06:38,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1276026.6666666667, ans=0.04949747468305833 2023-11-21 00:06:41,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1276026.6666666667, ans=0.0 2023-11-21 00:06:46,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1276093.3333333333, ans=0.0 2023-11-21 00:07:25,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1276293.3333333333, ans=0.1 2023-11-21 00:07:29,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191450 2023-11-21 00:07:36,870 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11100, loss[loss=0.07532, simple_loss=0.08377, pruned_loss=0.01895, audio_tagging_loss=0.01448, over 15784.00 frames. ], tot_loss[loss=0.07741, simple_loss=0.09844, pruned_loss=0.01817, audio_tagging_loss=0.01002, over 3046139.71 frames. ], batch size: 59, lr: 4.24e-03, grad_scale: 16.0 2023-11-21 00:07:39,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1276360.0, ans=0.125 2023-11-21 00:07:40,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.75 vs. limit=15.0 2023-11-21 00:07:42,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1276360.0, ans=0.1 2023-11-21 00:07:45,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1276360.0, ans=0.125 2023-11-21 00:07:52,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1276426.6666666667, ans=0.125 2023-11-21 00:08:09,713 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.351e+01 9.133e+01 9.830e+01 1.252e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-21 00:08:32,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191500 2023-11-21 00:08:40,575 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11150, loss[loss=0.07188, simple_loss=0.08589, pruned_loss=0.01552, audio_tagging_loss=0.01341, over 15329.00 frames. ], tot_loss[loss=0.07678, simple_loss=0.09762, pruned_loss=0.01795, audio_tagging_loss=0.01002, over 3044294.69 frames. ], batch size: 56, lr: 4.24e-03, grad_scale: 16.0 2023-11-21 00:08:50,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1276693.3333333333, ans=0.125 2023-11-21 00:09:09,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1276826.6666666667, ans=0.07 2023-11-21 00:09:13,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1276826.6666666667, ans=0.0 2023-11-21 00:09:17,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.91 vs. limit=15.0 2023-11-21 00:09:23,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1276893.3333333333, ans=0.125 2023-11-21 00:09:25,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1276893.3333333333, ans=0.0 2023-11-21 00:09:37,015 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191550 2023-11-21 00:09:44,266 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11200, loss[loss=0.07773, simple_loss=0.1071, pruned_loss=0.01626, audio_tagging_loss=0.007939, over 15925.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.09777, pruned_loss=0.0178, audio_tagging_loss=0.0101, over 3046565.04 frames. ], batch size: 60, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:09:53,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1277026.6666666667, ans=0.125 2023-11-21 00:10:18,321 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.318e+01 8.964e+01 9.909e+01 1.504e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-21 00:10:30,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1277226.6666666667, ans=0.125 2023-11-21 00:10:41,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191600 2023-11-21 00:10:47,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1277360.0, ans=0.1 2023-11-21 00:10:48,717 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11250, loss[loss=0.06023, simple_loss=0.07902, pruned_loss=0.008656, audio_tagging_loss=0.01206, over 15084.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09733, pruned_loss=0.0179, audio_tagging_loss=0.01013, over 3050182.94 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:10:56,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1277360.0, ans=0.2 2023-11-21 00:11:04,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1277426.6666666667, ans=0.05 2023-11-21 00:11:17,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1277493.3333333333, ans=0.2 2023-11-21 00:11:46,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191650 2023-11-21 00:11:47,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.65 vs. limit=15.0 2023-11-21 00:11:47,804 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.65 vs. limit=15.0 2023-11-21 00:11:53,463 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11300, loss[loss=0.07399, simple_loss=0.09169, pruned_loss=0.01529, audio_tagging_loss=0.01286, over 14950.00 frames. ], tot_loss[loss=0.07735, simple_loss=0.09832, pruned_loss=0.01821, audio_tagging_loss=0.009982, over 3052884.04 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:11:56,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1277693.3333333333, ans=0.0 2023-11-21 00:12:02,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1277693.3333333333, ans=0.125 2023-11-21 00:12:06,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1277760.0, ans=0.125 2023-11-21 00:12:08,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1277760.0, ans=0.125 2023-11-21 00:12:12,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.12 vs. limit=22.5 2023-11-21 00:12:26,957 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.170e+01 7.843e+01 8.601e+01 9.135e+01 1.178e+02, threshold=1.720e+02, percent-clipped=0.0 2023-11-21 00:12:46,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1277960.0, ans=0.125 2023-11-21 00:12:50,751 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191700 2023-11-21 00:12:53,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1277960.0, ans=0.0 2023-11-21 00:12:53,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1277960.0, ans=0.0 2023-11-21 00:12:57,882 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11350, loss[loss=0.07554, simple_loss=0.1018, pruned_loss=0.01778, audio_tagging_loss=0.006841, over 15541.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09878, pruned_loss=0.01819, audio_tagging_loss=0.009801, over 3053430.42 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:13:15,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1278093.3333333333, ans=0.1 2023-11-21 00:13:36,545 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.58 vs. limit=22.5 2023-11-21 00:13:51,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.45 vs. limit=5.0 2023-11-21 00:13:53,020 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191750 2023-11-21 00:13:59,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1278360.0, ans=0.0 2023-11-21 00:14:00,807 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11400, loss[loss=0.07467, simple_loss=0.09488, pruned_loss=0.01672, audio_tagging_loss=0.01051, over 13388.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09969, pruned_loss=0.01833, audio_tagging_loss=0.009756, over 3048128.27 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 16.0 2023-11-21 00:14:25,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.18 vs. limit=15.0 2023-11-21 00:14:31,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1278493.3333333333, ans=0.0 2023-11-21 00:14:35,210 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.048e+01 8.638e+01 9.436e+01 1.129e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-21 00:14:46,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1278560.0, ans=0.125 2023-11-21 00:14:50,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1278560.0, ans=0.07 2023-11-21 00:14:57,062 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191800 2023-11-21 00:14:58,945 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.79 vs. limit=15.0 2023-11-21 00:15:05,471 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11450, loss[loss=0.07685, simple_loss=0.0946, pruned_loss=0.02004, audio_tagging_loss=0.009508, over 14530.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.09897, pruned_loss=0.01828, audio_tagging_loss=0.009812, over 3047462.14 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 16.0 2023-11-21 00:15:08,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1278693.3333333333, ans=0.04949747468305833 2023-11-21 00:15:10,450 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.24 vs. limit=15.0 2023-11-21 00:15:21,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1278760.0, ans=0.125 2023-11-21 00:15:41,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1278826.6666666667, ans=0.04949747468305833 2023-11-21 00:15:46,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1278893.3333333333, ans=0.1 2023-11-21 00:15:55,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1278960.0, ans=0.2 2023-11-21 00:15:56,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1278960.0, ans=0.1 2023-11-21 00:16:02,271 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191850 2023-11-21 00:16:09,424 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11500, loss[loss=0.07182, simple_loss=0.0933, pruned_loss=0.01766, audio_tagging_loss=0.007513, over 15791.00 frames. ], tot_loss[loss=0.07693, simple_loss=0.09809, pruned_loss=0.01806, audio_tagging_loss=0.009822, over 3046920.79 frames. ], batch size: 59, lr: 4.24e-03, grad_scale: 16.0 2023-11-21 00:16:27,737 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.05 vs. limit=22.5 2023-11-21 00:16:42,857 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 7.974e+01 8.630e+01 9.256e+01 1.145e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-21 00:16:54,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1279226.6666666667, ans=0.1 2023-11-21 00:16:57,097 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.01 vs. limit=10.0 2023-11-21 00:17:03,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1279293.3333333333, ans=0.125 2023-11-21 00:17:05,364 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191900 2023-11-21 00:17:08,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.12 vs. limit=15.0 2023-11-21 00:17:13,190 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11550, loss[loss=0.05401, simple_loss=0.06534, pruned_loss=0.00809, audio_tagging_loss=0.01325, over 14539.00 frames. ], tot_loss[loss=0.077, simple_loss=0.09828, pruned_loss=0.01807, audio_tagging_loss=0.009783, over 3042822.77 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 16.0 2023-11-21 00:17:20,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1279360.0, ans=0.0 2023-11-21 00:17:22,041 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:17:31,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1279426.6666666667, ans=0.125 2023-11-21 00:17:41,524 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:17:50,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1279560.0, ans=0.0 2023-11-21 00:17:51,068 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 00:17:57,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1279560.0, ans=0.1 2023-11-21 00:18:05,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.81 vs. limit=8.0 2023-11-21 00:18:08,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 191950 2023-11-21 00:18:14,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1279693.3333333333, ans=0.125 2023-11-21 00:18:16,013 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11600, loss[loss=0.08772, simple_loss=0.1111, pruned_loss=0.0198, audio_tagging_loss=0.01235, over 14726.00 frames. ], tot_loss[loss=0.07718, simple_loss=0.09865, pruned_loss=0.01811, audio_tagging_loss=0.009745, over 3043264.39 frames. ], batch size: 56, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:18:36,462 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.73 vs. limit=22.5 2023-11-21 00:18:37,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1279760.0, ans=0.125 2023-11-21 00:18:51,113 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.722e+01 8.133e+01 8.974e+01 9.770e+01 1.687e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-21 00:19:04,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1279893.3333333333, ans=0.0 2023-11-21 00:19:07,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0 2023-11-21 00:19:13,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192000 2023-11-21 00:19:21,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1279960.0, ans=0.0 2023-11-21 00:19:24,882 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11650, loss[loss=0.07321, simple_loss=0.09465, pruned_loss=0.01535, audio_tagging_loss=0.01053, over 14720.00 frames. ], tot_loss[loss=0.07722, simple_loss=0.0986, pruned_loss=0.0181, audio_tagging_loss=0.009815, over 3039431.52 frames. ], batch size: 56, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:19:25,402 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.68 vs. limit=15.0 2023-11-21 00:19:40,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1280093.3333333333, ans=10.0 2023-11-21 00:19:44,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1280093.3333333333, ans=0.125 2023-11-21 00:19:57,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1280160.0, ans=0.125 2023-11-21 00:19:59,212 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.73 vs. limit=15.0 2023-11-21 00:20:21,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192050 2023-11-21 00:20:28,814 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11700, loss[loss=0.06527, simple_loss=0.07434, pruned_loss=0.01655, audio_tagging_loss=0.01155, over 15754.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09754, pruned_loss=0.01781, audio_tagging_loss=0.009927, over 3039987.44 frames. ], batch size: 62, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:20:38,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.70 vs. limit=15.0 2023-11-21 00:21:03,122 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.317e+01 8.852e+01 9.641e+01 1.228e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-21 00:21:04,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.96 vs. limit=10.0 2023-11-21 00:21:10,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1280560.0, ans=0.0 2023-11-21 00:21:19,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1280626.6666666667, ans=0.125 2023-11-21 00:21:20,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1280626.6666666667, ans=0.0 2023-11-21 00:21:24,882 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192100 2023-11-21 00:21:32,257 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11750, loss[loss=0.1146, simple_loss=0.1498, pruned_loss=0.03187, audio_tagging_loss=0.007844, over 16474.00 frames. ], tot_loss[loss=0.07661, simple_loss=0.09772, pruned_loss=0.01785, audio_tagging_loss=0.0099, over 3037321.16 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:21:33,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1280693.3333333333, ans=0.0 2023-11-21 00:21:36,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1280693.3333333333, ans=0.0 2023-11-21 00:21:50,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1280760.0, ans=0.0 2023-11-21 00:21:57,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.10 vs. limit=15.0 2023-11-21 00:22:08,969 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.64 vs. limit=8.0 2023-11-21 00:22:15,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1280893.3333333333, ans=0.125 2023-11-21 00:22:27,421 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192150 2023-11-21 00:22:33,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1280960.0, ans=0.0 2023-11-21 00:22:35,284 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11800, loss[loss=0.07491, simple_loss=0.1017, pruned_loss=0.01577, audio_tagging_loss=0.008307, over 14722.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09755, pruned_loss=0.0179, audio_tagging_loss=0.009883, over 3035198.58 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:22:42,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1281026.6666666667, ans=0.125 2023-11-21 00:22:43,545 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:23:09,494 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.064e+01 8.621e+01 9.481e+01 1.242e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-21 00:23:24,316 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.47 vs. limit=15.0 2023-11-21 00:23:26,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0 2023-11-21 00:23:31,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192200 2023-11-21 00:23:39,817 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11850, loss[loss=0.07727, simple_loss=0.09658, pruned_loss=0.01812, audio_tagging_loss=0.01086, over 15006.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.09749, pruned_loss=0.01781, audio_tagging_loss=0.009967, over 3039831.75 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:23:49,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1281360.0, ans=0.2 2023-11-21 00:23:52,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1281426.6666666667, ans=0.125 2023-11-21 00:23:56,020 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:24:01,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.68 vs. limit=15.0 2023-11-21 00:24:06,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1281493.3333333333, ans=0.125 2023-11-21 00:24:10,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1281493.3333333333, ans=0.125 2023-11-21 00:24:35,080 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192250 2023-11-21 00:24:38,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1281626.6666666667, ans=0.125 2023-11-21 00:24:42,291 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11900, loss[loss=0.06483, simple_loss=0.09183, pruned_loss=0.009601, audio_tagging_loss=0.009313, over 16057.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09835, pruned_loss=0.01783, audio_tagging_loss=0.01004, over 3039414.33 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 32.0 2023-11-21 00:25:06,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1281826.6666666667, ans=0.2 2023-11-21 00:25:17,101 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.295e+01 9.404e+01 1.030e+02 1.674e+02, threshold=1.881e+02, percent-clipped=0.0 2023-11-21 00:25:27,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1281893.3333333333, ans=0.125 2023-11-21 00:25:37,977 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192300 2023-11-21 00:25:45,944 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 11950, loss[loss=0.06555, simple_loss=0.08634, pruned_loss=0.01222, audio_tagging_loss=0.01016, over 16562.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09839, pruned_loss=0.01789, audio_tagging_loss=0.01003, over 3037206.49 frames. ], batch size: 61, lr: 4.23e-03, grad_scale: 32.0 2023-11-21 00:25:51,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1282026.6666666667, ans=0.1 2023-11-21 00:26:16,723 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.11 vs. limit=15.0 2023-11-21 00:26:22,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1282226.6666666667, ans=0.125 2023-11-21 00:26:31,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0 2023-11-21 00:26:35,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1282293.3333333333, ans=0.125 2023-11-21 00:26:39,421 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192350 2023-11-21 00:26:40,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1282293.3333333333, ans=0.1 2023-11-21 00:26:47,158 INFO [train_asr.py:1221] (3/4) Epoch 16, batch 12000, loss[loss=0.08876, simple_loss=0.1174, pruned_loss=0.02141, audio_tagging_loss=0.008658, over 15025.00 frames. ], tot_loss[loss=0.07688, simple_loss=0.09766, pruned_loss=0.01792, audio_tagging_loss=0.01013, over 3033965.30 frames. ], batch size: 54, lr: 4.23e-03, grad_scale: 32.0 2023-11-21 00:26:47,159 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 00:27:09,200 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([5.1764, 4.4458, 4.4536, 4.4428], device='cuda:3') 2023-11-21 00:27:28,019 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([3.2366, 2.1127, 2.5485, 2.1702], device='cuda:3') 2023-11-21 00:27:30,234 INFO [train_asr.py:1253] (3/4) Epoch 16, validation: loss=0.06114, simple_loss=0.05299, pruned_loss=0.005583, audio_tagging_loss=0.02906, over 4681554.00 frames. 2023-11-21 00:27:30,235 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 00:27:39,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1282360.0, ans=0.05 2023-11-21 00:27:44,408 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.53 vs. limit=15.0 2023-11-21 00:27:47,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0 2023-11-21 00:28:33,070 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 0, loss[loss=0.0756, simple_loss=0.0785, pruned_loss=0.01358, audio_tagging_loss=0.02277, over 15428.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.0785, pruned_loss=0.01358, audio_tagging_loss=0.02277, over 15428.00 frames. ], batch size: 58, lr: 4.11e-03, grad_scale: 32.0 2023-11-21 00:28:33,071 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 00:28:55,091 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.6927, 3.7113, 4.7905, 3.9511], device='cuda:3') 2023-11-21 00:29:03,653 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8333, 4.9254, 4.9309, 4.8936], device='cuda:3') 2023-11-21 00:29:12,185 INFO [train_asr.py:1253] (3/4) Epoch 17, validation: loss=0.06074, simple_loss=0.05295, pruned_loss=0.005487, audio_tagging_loss=0.02878, over 4681554.00 frames. 2023-11-21 00:29:12,186 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 00:29:13,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1282513.3333333333, ans=0.1 2023-11-21 00:29:18,854 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.043e+01 8.765e+01 9.548e+01 1.252e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-21 00:29:22,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1282513.3333333333, ans=0.09899494936611666 2023-11-21 00:29:39,185 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192400 2023-11-21 00:29:46,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1282646.6666666667, ans=0.125 2023-11-21 00:30:14,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1282780.0, ans=0.0 2023-11-21 00:30:16,446 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 50, loss[loss=0.1019, simple_loss=0.1235, pruned_loss=0.02432, audio_tagging_loss=0.01587, over 16145.00 frames. ], tot_loss[loss=0.08583, simple_loss=0.0991, pruned_loss=0.01765, audio_tagging_loss=0.01863, over 688725.33 frames. ], batch size: 57, lr: 4.11e-03, grad_scale: 32.0 2023-11-21 00:30:16,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1282846.6666666667, ans=0.125 2023-11-21 00:30:29,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1282913.3333333333, ans=0.125 2023-11-21 00:30:34,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1282913.3333333333, ans=0.0 2023-11-21 00:30:42,573 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192450 2023-11-21 00:30:45,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1282980.0, ans=0.125 2023-11-21 00:30:54,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1283046.6666666667, ans=0.5 2023-11-21 00:30:56,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1283046.6666666667, ans=0.1 2023-11-21 00:31:17,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1283113.3333333333, ans=0.0 2023-11-21 00:31:19,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1283180.0, ans=0.0 2023-11-21 00:31:20,504 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 100, loss[loss=0.07618, simple_loss=0.08809, pruned_loss=0.01613, audio_tagging_loss=0.01601, over 15878.00 frames. ], tot_loss[loss=0.08752, simple_loss=0.1021, pruned_loss=0.01863, audio_tagging_loss=0.01784, over 1204526.57 frames. ], batch size: 62, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:31:27,989 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.701e+01 8.625e+01 9.421e+01 9.913e+01 1.432e+02, threshold=1.884e+02, percent-clipped=0.0 2023-11-21 00:31:47,730 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192500 2023-11-21 00:31:55,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1283313.3333333333, ans=0.125 2023-11-21 00:32:24,250 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 150, loss[loss=0.04659, simple_loss=0.04984, pruned_loss=0.006134, audio_tagging_loss=0.01554, over 15067.00 frames. ], tot_loss[loss=0.08417, simple_loss=0.1002, pruned_loss=0.01813, audio_tagging_loss=0.01594, over 1617835.24 frames. ], batch size: 59, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:32:36,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1283580.0, ans=0.125 2023-11-21 00:32:42,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1283580.0, ans=0.125 2023-11-21 00:32:43,207 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.24 vs. limit=15.0 2023-11-21 00:32:47,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.18 vs. limit=22.5 2023-11-21 00:32:51,324 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192550 2023-11-21 00:32:56,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1283646.6666666667, ans=0.125 2023-11-21 00:33:13,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff3.min_abs, batch_count=1283713.3333333333, ans=0.2 2023-11-21 00:33:26,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1283780.0, ans=0.1 2023-11-21 00:33:27,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1283780.0, ans=0.5 2023-11-21 00:33:29,844 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 200, loss[loss=0.07588, simple_loss=0.09557, pruned_loss=0.0193, audio_tagging_loss=0.008802, over 15861.00 frames. ], tot_loss[loss=0.08255, simple_loss=0.1006, pruned_loss=0.01814, audio_tagging_loss=0.01412, over 1929165.01 frames. ], batch size: 60, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:33:30,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1283846.6666666667, ans=0.025 2023-11-21 00:33:30,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1283846.6666666667, ans=0.1 2023-11-21 00:33:37,143 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 8.165e+01 8.890e+01 9.863e+01 2.020e+02, threshold=1.778e+02, percent-clipped=1.0 2023-11-21 00:33:41,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1283913.3333333333, ans=0.1 2023-11-21 00:33:55,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1283980.0, ans=0.0 2023-11-21 00:33:56,891 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192600 2023-11-21 00:33:58,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1283980.0, ans=0.125 2023-11-21 00:34:11,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0 2023-11-21 00:34:16,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1284046.6666666667, ans=0.125 2023-11-21 00:34:21,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1284113.3333333333, ans=0.5 2023-11-21 00:34:33,592 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 250, loss[loss=0.1049, simple_loss=0.1334, pruned_loss=0.02823, audio_tagging_loss=0.009955, over 13983.00 frames. ], tot_loss[loss=0.08069, simple_loss=0.09932, pruned_loss=0.01809, audio_tagging_loss=0.01293, over 2180479.03 frames. ], batch size: 53, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:34:47,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1284246.6666666667, ans=0.125 2023-11-21 00:35:00,410 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192650 2023-11-21 00:35:05,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1284313.3333333333, ans=0.125 2023-11-21 00:35:25,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1284446.6666666667, ans=0.125 2023-11-21 00:35:37,470 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 300, loss[loss=0.0767, simple_loss=0.1005, pruned_loss=0.01871, audio_tagging_loss=0.00775, over 15296.00 frames. ], tot_loss[loss=0.07906, simple_loss=0.09802, pruned_loss=0.01796, audio_tagging_loss=0.01209, over 2371685.38 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:35:38,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1284513.3333333333, ans=0.1 2023-11-21 00:35:41,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.39 vs. limit=15.0 2023-11-21 00:35:45,409 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.137e+01 8.919e+01 9.657e+01 1.268e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-21 00:36:03,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192700 2023-11-21 00:36:08,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=1284646.6666666667, ans=0.1 2023-11-21 00:36:33,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1284780.0, ans=0.1 2023-11-21 00:36:36,007 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:36:40,665 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 350, loss[loss=0.06704, simple_loss=0.08439, pruned_loss=0.01593, audio_tagging_loss=0.008913, over 14969.00 frames. ], tot_loss[loss=0.07922, simple_loss=0.09879, pruned_loss=0.01834, audio_tagging_loss=0.01149, over 2518389.65 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:36:42,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1284846.6666666667, ans=0.125 2023-11-21 00:37:07,011 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192750 2023-11-21 00:37:09,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.whiten.whitening_limit, batch_count=1284980.0, ans=12.0 2023-11-21 00:37:18,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1285046.6666666667, ans=0.04949747468305833 2023-11-21 00:37:27,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1285046.6666666667, ans=0.1 2023-11-21 00:37:28,472 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:37:29,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1285046.6666666667, ans=0.0 2023-11-21 00:37:38,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1285113.3333333333, ans=0.0 2023-11-21 00:37:44,027 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 400, loss[loss=0.07595, simple_loss=0.08872, pruned_loss=0.01874, audio_tagging_loss=0.01284, over 15053.00 frames. ], tot_loss[loss=0.07846, simple_loss=0.09851, pruned_loss=0.01811, audio_tagging_loss=0.0111, over 2639927.17 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 32.0 2023-11-21 00:37:51,990 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.176e+01 8.863e+01 9.795e+01 2.108e+02, threshold=1.773e+02, percent-clipped=1.0 2023-11-21 00:37:59,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1285246.6666666667, ans=0.0 2023-11-21 00:38:11,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.75 vs. limit=15.0 2023-11-21 00:38:11,642 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192800 2023-11-21 00:38:16,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1285313.3333333333, ans=0.1 2023-11-21 00:38:22,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1285380.0, ans=0.0 2023-11-21 00:38:35,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1285446.6666666667, ans=0.0 2023-11-21 00:38:40,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1285446.6666666667, ans=0.125 2023-11-21 00:38:47,679 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 450, loss[loss=0.06552, simple_loss=0.09136, pruned_loss=0.01078, audio_tagging_loss=0.009063, over 15309.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09854, pruned_loss=0.01809, audio_tagging_loss=0.01073, over 2728639.95 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:38:55,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1285513.3333333333, ans=0.2 2023-11-21 00:39:14,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192850 2023-11-21 00:39:30,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1285713.3333333333, ans=0.0 2023-11-21 00:39:33,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1285713.3333333333, ans=0.1 2023-11-21 00:39:37,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1285780.0, ans=0.0 2023-11-21 00:39:49,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1285780.0, ans=0.125 2023-11-21 00:39:49,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1285780.0, ans=0.04949747468305833 2023-11-21 00:39:51,821 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 500, loss[loss=0.07803, simple_loss=0.1024, pruned_loss=0.01758, audio_tagging_loss=0.009239, over 15573.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09934, pruned_loss=0.01836, audio_tagging_loss=0.01044, over 2809584.69 frames. ], batch size: 58, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:39:53,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1285846.6666666667, ans=0.125 2023-11-21 00:40:00,403 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 7.999e+01 8.766e+01 9.500e+01 1.232e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-21 00:40:11,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1285913.3333333333, ans=0.125 2023-11-21 00:40:18,514 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192900 2023-11-21 00:40:26,322 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.93 vs. limit=12.0 2023-11-21 00:40:28,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1286046.6666666667, ans=0.1 2023-11-21 00:40:35,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1286046.6666666667, ans=0.125 2023-11-21 00:40:47,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1286113.3333333333, ans=0.1 2023-11-21 00:40:55,088 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 550, loss[loss=0.07467, simple_loss=0.1003, pruned_loss=0.01696, audio_tagging_loss=0.007581, over 15547.00 frames. ], tot_loss[loss=0.07806, simple_loss=0.09894, pruned_loss=0.01828, audio_tagging_loss=0.01031, over 2860666.33 frames. ], batch size: 58, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:40:56,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1286180.0, ans=0.125 2023-11-21 00:41:20,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1286313.3333333333, ans=0.0 2023-11-21 00:41:22,224 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 192950 2023-11-21 00:41:24,922 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.24 vs. limit=8.0 2023-11-21 00:41:49,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1286446.6666666667, ans=0.125 2023-11-21 00:41:54,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1286446.6666666667, ans=0.125 2023-11-21 00:41:58,734 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 600, loss[loss=0.07443, simple_loss=0.09242, pruned_loss=0.02067, audio_tagging_loss=0.007548, over 15843.00 frames. ], tot_loss[loss=0.07769, simple_loss=0.09837, pruned_loss=0.01819, audio_tagging_loss=0.01032, over 2896916.26 frames. ], batch size: 61, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:42:06,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1286513.3333333333, ans=0.0 2023-11-21 00:42:07,182 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.459e+01 7.723e+01 8.573e+01 9.272e+01 1.267e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-21 00:42:24,975 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193000 2023-11-21 00:42:34,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1286646.6666666667, ans=0.125 2023-11-21 00:42:37,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1286713.3333333333, ans=0.125 2023-11-21 00:42:42,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1286713.3333333333, ans=0.125 2023-11-21 00:43:02,803 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 650, loss[loss=0.04967, simple_loss=0.05932, pruned_loss=0.007579, audio_tagging_loss=0.01243, over 15821.00 frames. ], tot_loss[loss=0.07716, simple_loss=0.09821, pruned_loss=0.01788, audio_tagging_loss=0.01018, over 2929734.46 frames. ], batch size: 60, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:43:06,818 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 00:43:14,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1286913.3333333333, ans=0.1 2023-11-21 00:43:29,231 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193050 2023-11-21 00:43:33,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1286980.0, ans=0.125 2023-11-21 00:43:40,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.27 vs. limit=15.0 2023-11-21 00:44:01,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1287113.3333333333, ans=0.0 2023-11-21 00:44:05,990 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 700, loss[loss=0.07681, simple_loss=0.0929, pruned_loss=0.02, audio_tagging_loss=0.01036, over 14795.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09767, pruned_loss=0.01776, audio_tagging_loss=0.01006, over 2955355.97 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:44:15,336 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 7.999e+01 8.466e+01 9.113e+01 1.191e+02, threshold=1.693e+02, percent-clipped=0.0 2023-11-21 00:44:26,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=1287246.6666666667, ans=0.1 2023-11-21 00:44:32,781 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193100 2023-11-21 00:44:34,619 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=15.0 2023-11-21 00:45:00,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1287446.6666666667, ans=0.0 2023-11-21 00:45:04,862 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0 2023-11-21 00:45:10,274 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 750, loss[loss=0.0708, simple_loss=0.09581, pruned_loss=0.01389, audio_tagging_loss=0.009003, over 15935.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.09827, pruned_loss=0.01779, audio_tagging_loss=0.01004, over 2972857.16 frames. ], batch size: 58, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:45:11,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1287513.3333333333, ans=0.07 2023-11-21 00:45:13,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1287513.3333333333, ans=0.125 2023-11-21 00:45:37,823 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193150 2023-11-21 00:45:58,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1287713.3333333333, ans=0.0 2023-11-21 00:46:01,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.69 vs. limit=15.0 2023-11-21 00:46:14,614 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 800, loss[loss=0.06553, simple_loss=0.0859, pruned_loss=0.01281, audio_tagging_loss=0.009767, over 14974.00 frames. ], tot_loss[loss=0.07678, simple_loss=0.09821, pruned_loss=0.01759, audio_tagging_loss=0.01008, over 2989633.29 frames. ], batch size: 58, lr: 4.10e-03, grad_scale: 32.0 2023-11-21 00:46:24,324 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.140e+01 8.941e+01 9.385e+01 1.183e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-21 00:46:27,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1287913.3333333333, ans=0.2 2023-11-21 00:46:29,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.22 vs. limit=22.5 2023-11-21 00:46:31,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1287913.3333333333, ans=0.125 2023-11-21 00:46:36,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1287913.3333333333, ans=0.0 2023-11-21 00:46:42,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193200 2023-11-21 00:46:46,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1287980.0, ans=0.125 2023-11-21 00:46:49,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1287980.0, ans=0.0 2023-11-21 00:47:13,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288113.3333333333, ans=0.1 2023-11-21 00:47:18,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1288180.0, ans=0.125 2023-11-21 00:47:19,548 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 850, loss[loss=0.06757, simple_loss=0.08542, pruned_loss=0.01205, audio_tagging_loss=0.01281, over 16382.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09869, pruned_loss=0.01776, audio_tagging_loss=0.01004, over 2999088.40 frames. ], batch size: 62, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:47:21,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288180.0, ans=0.1 2023-11-21 00:47:24,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1288180.0, ans=0.125 2023-11-21 00:47:25,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1288180.0, ans=0.0 2023-11-21 00:47:35,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1288246.6666666667, ans=0.125 2023-11-21 00:47:36,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1288246.6666666667, ans=0.125 2023-11-21 00:47:45,977 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193250 2023-11-21 00:47:57,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1288380.0, ans=0.0 2023-11-21 00:48:06,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1288380.0, ans=0.035 2023-11-21 00:48:16,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1288446.6666666667, ans=0.05 2023-11-21 00:48:21,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.24 vs. limit=22.5 2023-11-21 00:48:23,676 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 900, loss[loss=0.08036, simple_loss=0.1003, pruned_loss=0.01784, audio_tagging_loss=0.01235, over 15238.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.09915, pruned_loss=0.01785, audio_tagging_loss=0.01012, over 3007877.72 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:48:23,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288513.3333333333, ans=0.1 2023-11-21 00:48:27,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.36 vs. limit=15.0 2023-11-21 00:48:33,414 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.920e+01 8.040e+01 8.568e+01 9.329e+01 1.303e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 00:48:49,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.14 vs. limit=15.0 2023-11-21 00:48:50,882 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193300 2023-11-21 00:48:55,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1288646.6666666667, ans=0.2 2023-11-21 00:48:56,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1288646.6666666667, ans=0.07 2023-11-21 00:48:57,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1288646.6666666667, ans=0.0 2023-11-21 00:49:00,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1288646.6666666667, ans=0.125 2023-11-21 00:49:15,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1288780.0, ans=0.2 2023-11-21 00:49:27,459 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 950, loss[loss=0.06876, simple_loss=0.08317, pruned_loss=0.01707, audio_tagging_loss=0.0101, over 16752.00 frames. ], tot_loss[loss=0.07728, simple_loss=0.09891, pruned_loss=0.01793, audio_tagging_loss=0.009902, over 3014599.39 frames. ], batch size: 63, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:49:38,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1288846.6666666667, ans=0.1 2023-11-21 00:49:50,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.33 vs. limit=15.0 2023-11-21 00:49:55,840 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193350 2023-11-21 00:50:03,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1288980.0, ans=0.125 2023-11-21 00:50:16,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1289046.6666666667, ans=0.125 2023-11-21 00:50:24,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1289113.3333333333, ans=0.1 2023-11-21 00:50:27,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1289113.3333333333, ans=0.125 2023-11-21 00:50:28,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1289113.3333333333, ans=0.125 2023-11-21 00:50:32,458 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1000, loss[loss=0.1009, simple_loss=0.1194, pruned_loss=0.03176, audio_tagging_loss=0.009451, over 14358.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09919, pruned_loss=0.01801, audio_tagging_loss=0.009771, over 3024515.93 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0 2023-11-21 00:50:42,902 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.002e+01 8.372e+01 8.934e+01 9.683e+01 2.784e+02, threshold=1.787e+02, percent-clipped=1.0 2023-11-21 00:50:48,179 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.83 vs. limit=12.0 2023-11-21 00:50:50,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1289246.6666666667, ans=0.125 2023-11-21 00:50:59,514 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 00:50:59,586 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193400 2023-11-21 00:51:02,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1289313.3333333333, ans=0.125 2023-11-21 00:51:13,033 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.63 vs. limit=10.0 2023-11-21 00:51:24,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1289446.6666666667, ans=0.1 2023-11-21 00:51:38,127 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1050, loss[loss=0.07728, simple_loss=0.09445, pruned_loss=0.01972, audio_tagging_loss=0.01034, over 14742.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09813, pruned_loss=0.01773, audio_tagging_loss=0.009608, over 3029076.98 frames. ], batch size: 57, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 00:51:40,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.75 vs. limit=22.5 2023-11-21 00:51:53,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1289580.0, ans=0.0 2023-11-21 00:52:05,430 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193450 2023-11-21 00:52:07,126 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.49 vs. limit=12.0 2023-11-21 00:52:09,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1289646.6666666667, ans=0.125 2023-11-21 00:52:36,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1289780.0, ans=0.125 2023-11-21 00:52:38,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1289780.0, ans=0.0 2023-11-21 00:52:41,789 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1100, loss[loss=0.0898, simple_loss=0.1166, pruned_loss=0.0224, audio_tagging_loss=0.009077, over 15519.00 frames. ], tot_loss[loss=0.07577, simple_loss=0.09729, pruned_loss=0.01748, audio_tagging_loss=0.009639, over 3025317.38 frames. ], batch size: 55, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 00:52:44,377 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 00:52:46,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.78 vs. limit=15.0 2023-11-21 00:52:47,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1289846.6666666667, ans=0.125 2023-11-21 00:52:52,955 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.642e+01 8.168e+01 8.741e+01 9.235e+01 1.137e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 00:52:59,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1289913.3333333333, ans=0.2 2023-11-21 00:53:08,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1289980.0, ans=0.125 2023-11-21 00:53:09,457 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193500 2023-11-21 00:53:10,257 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.35 vs. limit=22.5 2023-11-21 00:53:18,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1289980.0, ans=0.0 2023-11-21 00:53:31,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1290046.6666666667, ans=0.125 2023-11-21 00:53:34,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1290113.3333333333, ans=0.125 2023-11-21 00:53:46,393 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1150, loss[loss=0.08206, simple_loss=0.09705, pruned_loss=0.0224, audio_tagging_loss=0.01113, over 15240.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.0979, pruned_loss=0.01778, audio_tagging_loss=0.009591, over 3034552.23 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 00:54:03,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1290246.6666666667, ans=0.125 2023-11-21 00:54:08,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1290246.6666666667, ans=0.2 2023-11-21 00:54:09,089 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.24 vs. limit=15.0 2023-11-21 00:54:12,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1290313.3333333333, ans=0.125 2023-11-21 00:54:13,272 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193550 2023-11-21 00:54:25,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1290380.0, ans=0.0 2023-11-21 00:54:51,219 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1200, loss[loss=0.08137, simple_loss=0.1096, pruned_loss=0.01807, audio_tagging_loss=0.00849, over 14824.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09833, pruned_loss=0.01785, audio_tagging_loss=0.00949, over 3037832.40 frames. ], batch size: 54, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 00:54:56,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1290513.3333333333, ans=0.125 2023-11-21 00:55:00,965 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 7.968e+01 8.692e+01 9.308e+01 1.785e+02, threshold=1.738e+02, percent-clipped=1.0 2023-11-21 00:55:18,143 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193600 2023-11-21 00:55:43,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1290780.0, ans=0.0 2023-11-21 00:55:50,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1290780.0, ans=0.125 2023-11-21 00:55:55,007 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1250, loss[loss=0.06271, simple_loss=0.08526, pruned_loss=0.01302, audio_tagging_loss=0.007058, over 14814.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09928, pruned_loss=0.01807, audio_tagging_loss=0.009506, over 3039388.63 frames. ], batch size: 57, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 00:56:22,019 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193650 2023-11-21 00:56:23,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1290980.0, ans=0.0 2023-11-21 00:56:23,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1290980.0, ans=0.125 2023-11-21 00:56:59,707 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1300, loss[loss=0.08303, simple_loss=0.1133, pruned_loss=0.01979, audio_tagging_loss=0.006579, over 16260.00 frames. ], tot_loss[loss=0.07683, simple_loss=0.0989, pruned_loss=0.0179, audio_tagging_loss=0.009483, over 3036439.87 frames. ], batch size: 60, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 00:57:03,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1291180.0, ans=0.125 2023-11-21 00:57:09,537 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.443e+01 7.829e+01 8.638e+01 9.326e+01 1.163e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-21 00:57:17,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1291246.6666666667, ans=0.125 2023-11-21 00:57:20,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1291246.6666666667, ans=0.125 2023-11-21 00:57:26,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193700 2023-11-21 00:57:31,092 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.69 vs. limit=15.0 2023-11-21 00:57:43,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1291380.0, ans=0.125 2023-11-21 00:57:59,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1291446.6666666667, ans=0.1 2023-11-21 00:58:02,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1291513.3333333333, ans=0.1 2023-11-21 00:58:03,407 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1350, loss[loss=0.07709, simple_loss=0.09359, pruned_loss=0.01897, audio_tagging_loss=0.01133, over 14811.00 frames. ], tot_loss[loss=0.07683, simple_loss=0.09869, pruned_loss=0.01783, audio_tagging_loss=0.009649, over 3043798.29 frames. ], batch size: 58, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 00:58:22,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1291580.0, ans=0.1 2023-11-21 00:58:30,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten.whitening_limit, batch_count=1291646.6666666667, ans=15.0 2023-11-21 00:58:30,941 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193750 2023-11-21 00:58:49,304 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 00:59:03,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1291780.0, ans=0.95 2023-11-21 00:59:07,701 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1400, loss[loss=0.06346, simple_loss=0.07915, pruned_loss=0.01308, audio_tagging_loss=0.01081, over 14716.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.09935, pruned_loss=0.01803, audio_tagging_loss=0.009663, over 3050249.24 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 00:59:10,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1291846.6666666667, ans=0.0 2023-11-21 00:59:16,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1291846.6666666667, ans=0.125 2023-11-21 00:59:18,074 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.121e+01 7.749e+01 8.445e+01 9.345e+01 1.750e+02, threshold=1.689e+02, percent-clipped=1.0 2023-11-21 00:59:34,631 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193800 2023-11-21 00:59:34,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1291980.0, ans=0.125 2023-11-21 00:59:46,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1292046.6666666667, ans=0.1 2023-11-21 00:59:51,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1292046.6666666667, ans=0.0 2023-11-21 00:59:59,849 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.03 vs. limit=15.0 2023-11-21 01:00:02,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1292113.3333333333, ans=15.0 2023-11-21 01:00:12,433 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1450, loss[loss=0.07827, simple_loss=0.1042, pruned_loss=0.01716, audio_tagging_loss=0.008997, over 15161.00 frames. ], tot_loss[loss=0.07733, simple_loss=0.09902, pruned_loss=0.01801, audio_tagging_loss=0.009807, over 3048812.54 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 01:00:14,290 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.49 vs. limit=22.5 2023-11-21 01:00:17,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.25 vs. limit=15.0 2023-11-21 01:00:20,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1292180.0, ans=0.2 2023-11-21 01:00:23,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1292246.6666666667, ans=0.0 2023-11-21 01:00:34,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.95 vs. limit=6.0 2023-11-21 01:00:38,753 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193850 2023-11-21 01:01:00,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1292380.0, ans=0.0 2023-11-21 01:01:07,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1292446.6666666667, ans=0.125 2023-11-21 01:01:16,080 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1500, loss[loss=0.08607, simple_loss=0.1153, pruned_loss=0.02042, audio_tagging_loss=0.007982, over 14404.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.09994, pruned_loss=0.01821, audio_tagging_loss=0.009837, over 3050273.60 frames. ], batch size: 54, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 01:01:17,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1292513.3333333333, ans=0.2 2023-11-21 01:01:17,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1292513.3333333333, ans=0.125 2023-11-21 01:01:27,687 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.416e+01 9.067e+01 9.838e+01 1.487e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-21 01:01:29,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1292580.0, ans=0.1 2023-11-21 01:01:42,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193900 2023-11-21 01:01:43,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1292646.6666666667, ans=0.0 2023-11-21 01:01:46,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1292646.6666666667, ans=0.125 2023-11-21 01:01:50,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1292646.6666666667, ans=0.125 2023-11-21 01:02:14,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1292780.0, ans=0.1 2023-11-21 01:02:15,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1292780.0, ans=0.125 2023-11-21 01:02:20,446 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1550, loss[loss=0.07024, simple_loss=0.09594, pruned_loss=0.01206, audio_tagging_loss=0.01021, over 15340.00 frames. ], tot_loss[loss=0.07728, simple_loss=0.09874, pruned_loss=0.01797, audio_tagging_loss=0.009943, over 3054054.15 frames. ], batch size: 57, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 01:02:22,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.73 vs. limit=22.5 2023-11-21 01:02:31,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1292846.6666666667, ans=0.0 2023-11-21 01:02:44,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1292913.3333333333, ans=0.125 2023-11-21 01:02:44,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1292913.3333333333, ans=0.125 2023-11-21 01:02:47,735 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 193950 2023-11-21 01:03:03,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.01 vs. limit=15.0 2023-11-21 01:03:25,726 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1600, loss[loss=0.06574, simple_loss=0.08102, pruned_loss=0.01463, audio_tagging_loss=0.0106, over 16374.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09814, pruned_loss=0.01806, audio_tagging_loss=0.01012, over 3047162.71 frames. ], batch size: 62, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 01:03:37,000 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.454e+01 8.126e+01 8.877e+01 9.641e+01 1.501e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-21 01:03:38,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1293246.6666666667, ans=0.125 2023-11-21 01:03:49,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1293246.6666666667, ans=0.2 2023-11-21 01:03:52,658 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194000 2023-11-21 01:04:30,482 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1650, loss[loss=0.06656, simple_loss=0.07819, pruned_loss=0.01695, audio_tagging_loss=0.01052, over 14080.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09827, pruned_loss=0.018, audio_tagging_loss=0.01016, over 3048489.39 frames. ], batch size: 53, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 01:04:37,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1293513.3333333333, ans=0.125 2023-11-21 01:04:56,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194050 2023-11-21 01:05:13,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1293713.3333333333, ans=0.125 2023-11-21 01:05:32,298 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.00 vs. limit=22.5 2023-11-21 01:05:35,456 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1700, loss[loss=0.07297, simple_loss=0.09218, pruned_loss=0.01665, audio_tagging_loss=0.01022, over 15274.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09812, pruned_loss=0.01793, audio_tagging_loss=0.01016, over 3049699.78 frames. ], batch size: 59, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 01:05:46,330 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.167e+01 8.614e+01 9.209e+01 1.216e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 01:06:02,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194100 2023-11-21 01:06:13,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.24 vs. limit=22.5 2023-11-21 01:06:14,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1294046.6666666667, ans=0.1 2023-11-21 01:06:29,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1294113.3333333333, ans=0.0 2023-11-21 01:06:32,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1294113.3333333333, ans=0.125 2023-11-21 01:06:39,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1294180.0, ans=0.0 2023-11-21 01:06:40,573 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1750, loss[loss=0.08792, simple_loss=0.1172, pruned_loss=0.02012, audio_tagging_loss=0.009182, over 15510.00 frames. ], tot_loss[loss=0.07647, simple_loss=0.09756, pruned_loss=0.01764, audio_tagging_loss=0.01005, over 3059348.17 frames. ], batch size: 55, lr: 4.09e-03, grad_scale: 32.0 2023-11-21 01:06:42,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1294180.0, ans=0.125 2023-11-21 01:06:51,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1294246.6666666667, ans=0.0 2023-11-21 01:07:07,045 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194150 2023-11-21 01:07:42,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1294446.6666666667, ans=0.125 2023-11-21 01:07:44,353 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1800, loss[loss=0.08142, simple_loss=0.111, pruned_loss=0.01911, audio_tagging_loss=0.006816, over 13809.00 frames. ], tot_loss[loss=0.07665, simple_loss=0.0983, pruned_loss=0.01768, audio_tagging_loss=0.009823, over 3049556.25 frames. ], batch size: 53, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 01:07:45,943 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:07:49,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1294513.3333333333, ans=0.0 2023-11-21 01:07:55,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1294513.3333333333, ans=0.0 2023-11-21 01:07:55,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1294513.3333333333, ans=0.125 2023-11-21 01:07:57,247 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.340e+01 8.982e+01 9.918e+01 1.410e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-21 01:08:06,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1294580.0, ans=0.125 2023-11-21 01:08:10,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194200 2023-11-21 01:08:31,394 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:08:48,998 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1850, loss[loss=0.08009, simple_loss=0.1017, pruned_loss=0.01854, audio_tagging_loss=0.0107, over 16422.00 frames. ], tot_loss[loss=0.07645, simple_loss=0.0982, pruned_loss=0.01764, audio_tagging_loss=0.009716, over 3051873.68 frames. ], batch size: 64, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 01:08:54,044 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:09:09,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1294913.3333333333, ans=0.0 2023-11-21 01:09:15,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1294980.0, ans=0.2 2023-11-21 01:09:16,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194250 2023-11-21 01:09:17,510 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.41 vs. limit=15.0 2023-11-21 01:09:19,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1294980.0, ans=0.09899494936611666 2023-11-21 01:09:40,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.85 vs. limit=15.0 2023-11-21 01:09:52,803 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1900, loss[loss=0.07719, simple_loss=0.09933, pruned_loss=0.0193, audio_tagging_loss=0.008226, over 15915.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09714, pruned_loss=0.01761, audio_tagging_loss=0.009663, over 3053818.76 frames. ], batch size: 59, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 01:09:55,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1295180.0, ans=0.0 2023-11-21 01:10:01,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1295180.0, ans=0.125 2023-11-21 01:10:03,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1295180.0, ans=0.125 2023-11-21 01:10:06,891 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.112e+01 7.944e+01 8.677e+01 9.299e+01 1.429e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 01:10:21,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194300 2023-11-21 01:10:26,545 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.32 vs. limit=15.0 2023-11-21 01:10:31,371 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.90 vs. limit=22.5 2023-11-21 01:10:53,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1295446.6666666667, ans=0.125 2023-11-21 01:10:58,318 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 1950, loss[loss=0.0815, simple_loss=0.1005, pruned_loss=0.02219, audio_tagging_loss=0.009037, over 15642.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.0975, pruned_loss=0.01792, audio_tagging_loss=0.009643, over 3046871.15 frames. ], batch size: 62, lr: 4.09e-03, grad_scale: 16.0 2023-11-21 01:11:24,801 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194350 2023-11-21 01:11:35,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.46 vs. limit=12.0 2023-11-21 01:12:02,469 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2000, loss[loss=0.07966, simple_loss=0.1047, pruned_loss=0.01932, audio_tagging_loss=0.008003, over 15069.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09608, pruned_loss=0.01775, audio_tagging_loss=0.009827, over 3048191.54 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:12:14,649 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.274e+01 9.033e+01 9.827e+01 1.298e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-21 01:12:19,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.09 vs. limit=10.0 2023-11-21 01:12:20,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1295913.3333333333, ans=0.125 2023-11-21 01:12:24,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1295913.3333333333, ans=0.2 2023-11-21 01:12:29,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194400 2023-11-21 01:12:40,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1296046.6666666667, ans=0.07 2023-11-21 01:12:42,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1296046.6666666667, ans=0.2 2023-11-21 01:12:49,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1296046.6666666667, ans=0.125 2023-11-21 01:12:55,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1296113.3333333333, ans=0.1 2023-11-21 01:13:01,593 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:13:06,216 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2050, loss[loss=0.08682, simple_loss=0.117, pruned_loss=0.02183, audio_tagging_loss=0.006479, over 15624.00 frames. ], tot_loss[loss=0.07703, simple_loss=0.09811, pruned_loss=0.01832, audio_tagging_loss=0.009655, over 3048044.37 frames. ], batch size: 58, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:13:06,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1296180.0, ans=0.0 2023-11-21 01:13:25,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1296246.6666666667, ans=0.1 2023-11-21 01:13:33,623 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194450 2023-11-21 01:13:43,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1296313.3333333333, ans=0.0 2023-11-21 01:13:44,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1296380.0, ans=0.125 2023-11-21 01:13:48,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1296380.0, ans=0.125 2023-11-21 01:14:07,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1296446.6666666667, ans=0.125 2023-11-21 01:14:08,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.03 vs. limit=15.0 2023-11-21 01:14:10,866 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2100, loss[loss=0.08098, simple_loss=0.1096, pruned_loss=0.01556, audio_tagging_loss=0.01063, over 14821.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.09924, pruned_loss=0.01854, audio_tagging_loss=0.009555, over 3048030.81 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:14:11,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1296513.3333333333, ans=0.0 2023-11-21 01:14:22,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1296580.0, ans=0.125 2023-11-21 01:14:24,176 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.053e+01 8.707e+01 9.415e+01 1.178e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-21 01:14:37,790 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194500 2023-11-21 01:14:47,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1296646.6666666667, ans=0.125 2023-11-21 01:14:54,257 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.31 vs. limit=15.0 2023-11-21 01:15:01,552 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.89 vs. limit=10.0 2023-11-21 01:15:10,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1296780.0, ans=0.1 2023-11-21 01:15:15,765 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2150, loss[loss=0.05604, simple_loss=0.06717, pruned_loss=0.00905, audio_tagging_loss=0.01341, over 14266.00 frames. ], tot_loss[loss=0.07773, simple_loss=0.09929, pruned_loss=0.01838, audio_tagging_loss=0.009699, over 3046220.36 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:15:16,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1296846.6666666667, ans=0.2 2023-11-21 01:15:19,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1296846.6666666667, ans=0.0 2023-11-21 01:15:26,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1296913.3333333333, ans=0.0 2023-11-21 01:15:32,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.34 vs. limit=22.5 2023-11-21 01:15:42,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194550 2023-11-21 01:15:52,153 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:16:18,805 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2200, loss[loss=0.05255, simple_loss=0.06732, pruned_loss=0.007428, audio_tagging_loss=0.01146, over 16271.00 frames. ], tot_loss[loss=0.07693, simple_loss=0.0984, pruned_loss=0.01805, audio_tagging_loss=0.009685, over 3054648.31 frames. ], batch size: 64, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:16:30,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1297180.0, ans=0.1 2023-11-21 01:16:32,227 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.162e+01 8.764e+01 9.413e+01 1.161e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-21 01:16:35,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.16 vs. limit=15.0 2023-11-21 01:16:45,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194600 2023-11-21 01:16:45,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1297313.3333333333, ans=0.125 2023-11-21 01:17:14,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1297446.6666666667, ans=0.125 2023-11-21 01:17:17,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.34 vs. limit=22.5 2023-11-21 01:17:23,468 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2250, loss[loss=0.09574, simple_loss=0.1188, pruned_loss=0.02692, audio_tagging_loss=0.009414, over 15459.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.09996, pruned_loss=0.01846, audio_tagging_loss=0.00958, over 3052029.75 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:17:50,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194650 2023-11-21 01:18:12,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1297713.3333333333, ans=0.2 2023-11-21 01:18:19,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1297780.0, ans=0.2 2023-11-21 01:18:27,466 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2300, loss[loss=0.0851, simple_loss=0.1143, pruned_loss=0.02071, audio_tagging_loss=0.007238, over 14944.00 frames. ], tot_loss[loss=0.0771, simple_loss=0.09862, pruned_loss=0.01808, audio_tagging_loss=0.009714, over 3051116.67 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:18:37,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1297846.6666666667, ans=0.125 2023-11-21 01:18:41,610 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.557e+01 8.363e+01 8.884e+01 1.011e+02 1.300e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-21 01:18:55,308 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194700 2023-11-21 01:18:57,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1297980.0, ans=0.125 2023-11-21 01:19:22,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1298113.3333333333, ans=0.1 2023-11-21 01:19:23,577 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:19:25,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1298113.3333333333, ans=10.0 2023-11-21 01:19:26,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1298113.3333333333, ans=0.1 2023-11-21 01:19:32,008 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2350, loss[loss=0.07694, simple_loss=0.105, pruned_loss=0.01621, audio_tagging_loss=0.008235, over 14637.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09907, pruned_loss=0.01813, audio_tagging_loss=0.009803, over 3046371.41 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:19:46,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1298246.6666666667, ans=0.2 2023-11-21 01:19:53,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1298246.6666666667, ans=0.125 2023-11-21 01:19:59,066 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194750 2023-11-21 01:20:13,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1298380.0, ans=0.125 2023-11-21 01:20:16,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1298380.0, ans=0.125 2023-11-21 01:20:17,976 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0 2023-11-21 01:20:36,885 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2400, loss[loss=0.08644, simple_loss=0.1033, pruned_loss=0.02559, audio_tagging_loss=0.009177, over 14667.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.09885, pruned_loss=0.01809, audio_tagging_loss=0.009853, over 3046877.90 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:20:50,276 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.420e+01 8.164e+01 8.845e+01 9.386e+01 1.132e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 01:21:03,840 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194800 2023-11-21 01:21:35,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1298780.0, ans=0.2 2023-11-21 01:21:35,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1298780.0, ans=0.125 2023-11-21 01:21:40,671 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2450, loss[loss=0.07419, simple_loss=0.1002, pruned_loss=0.0162, audio_tagging_loss=0.007876, over 16148.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09914, pruned_loss=0.01807, audio_tagging_loss=0.009975, over 3041799.31 frames. ], batch size: 59, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:22:08,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194850 2023-11-21 01:22:10,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1298980.0, ans=0.125 2023-11-21 01:22:15,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1298980.0, ans=0.035 2023-11-21 01:22:16,713 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.82 vs. limit=10.0 2023-11-21 01:22:21,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1299046.6666666667, ans=0.125 2023-11-21 01:22:30,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1299113.3333333333, ans=0.125 2023-11-21 01:22:35,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1299113.3333333333, ans=0.2 2023-11-21 01:22:45,483 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2500, loss[loss=0.08294, simple_loss=0.109, pruned_loss=0.02024, audio_tagging_loss=0.008204, over 15144.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09829, pruned_loss=0.01801, audio_tagging_loss=0.01001, over 3049932.17 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:22:53,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1299180.0, ans=0.125 2023-11-21 01:23:01,471 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.181e+01 8.676e+01 9.218e+01 1.349e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 01:23:09,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1299246.6666666667, ans=0.025 2023-11-21 01:23:12,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194900 2023-11-21 01:23:50,128 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2550, loss[loss=0.06808, simple_loss=0.08497, pruned_loss=0.01248, audio_tagging_loss=0.01312, over 15277.00 frames. ], tot_loss[loss=0.07689, simple_loss=0.09803, pruned_loss=0.01791, audio_tagging_loss=0.009961, over 3054612.91 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:23:54,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1299513.3333333333, ans=0.2 2023-11-21 01:23:59,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1299513.3333333333, ans=0.125 2023-11-21 01:24:05,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.03 vs. limit=15.0 2023-11-21 01:24:11,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1299580.0, ans=0.125 2023-11-21 01:24:16,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 194950 2023-11-21 01:24:26,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1299646.6666666667, ans=0.0 2023-11-21 01:24:47,744 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:24:49,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1299780.0, ans=0.125 2023-11-21 01:24:53,743 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2600, loss[loss=0.06079, simple_loss=0.07218, pruned_loss=0.0148, audio_tagging_loss=0.009899, over 15831.00 frames. ], tot_loss[loss=0.07638, simple_loss=0.09727, pruned_loss=0.01791, audio_tagging_loss=0.009838, over 3045986.99 frames. ], batch size: 59, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:24:56,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1299846.6666666667, ans=0.125 2023-11-21 01:24:56,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1299846.6666666667, ans=0.125 2023-11-21 01:25:09,166 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.407e+01 7.907e+01 8.660e+01 9.294e+01 1.206e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 01:25:20,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195000 2023-11-21 01:25:29,891 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.00 vs. limit=22.5 2023-11-21 01:25:58,420 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2650, loss[loss=0.06509, simple_loss=0.07745, pruned_loss=0.01641, audio_tagging_loss=0.009956, over 15106.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09803, pruned_loss=0.01812, audio_tagging_loss=0.009712, over 3045324.78 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:25:59,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1300180.0, ans=0.0 2023-11-21 01:26:12,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1300246.6666666667, ans=0.125 2023-11-21 01:26:12,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.32 vs. limit=10.0 2023-11-21 01:26:13,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1300246.6666666667, ans=0.2 2023-11-21 01:26:18,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1300246.6666666667, ans=0.125 2023-11-21 01:26:26,009 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195050 2023-11-21 01:26:29,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1300313.3333333333, ans=0.2 2023-11-21 01:26:33,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1300313.3333333333, ans=0.0 2023-11-21 01:26:44,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.61 vs. limit=15.0 2023-11-21 01:27:03,578 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2700, loss[loss=0.06706, simple_loss=0.09221, pruned_loss=0.012, audio_tagging_loss=0.008947, over 15123.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09762, pruned_loss=0.01783, audio_tagging_loss=0.009726, over 3047117.24 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:27:15,965 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0 2023-11-21 01:27:18,976 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 7.841e+01 8.793e+01 9.412e+01 1.255e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 01:27:30,627 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195100 2023-11-21 01:27:34,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1300646.6666666667, ans=0.125 2023-11-21 01:27:58,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1300780.0, ans=0.125 2023-11-21 01:28:08,080 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2750, loss[loss=0.0702, simple_loss=0.08524, pruned_loss=0.01464, audio_tagging_loss=0.01294, over 15230.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09744, pruned_loss=0.01769, audio_tagging_loss=0.009729, over 3046430.81 frames. ], batch size: 58, lr: 4.08e-03, grad_scale: 16.0 2023-11-21 01:28:09,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1300846.6666666667, ans=0.2 2023-11-21 01:28:12,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1300846.6666666667, ans=0.0 2023-11-21 01:28:34,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195150 2023-11-21 01:28:44,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1300980.0, ans=0.125 2023-11-21 01:28:59,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1301113.3333333333, ans=0.125 2023-11-21 01:29:01,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1301113.3333333333, ans=0.2 2023-11-21 01:29:02,050 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:29:12,443 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2800, loss[loss=0.06773, simple_loss=0.08675, pruned_loss=0.01448, audio_tagging_loss=0.009876, over 14679.00 frames. ], tot_loss[loss=0.07646, simple_loss=0.09794, pruned_loss=0.01777, audio_tagging_loss=0.009724, over 3043572.50 frames. ], batch size: 54, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:29:27,664 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 8.111e+01 8.659e+01 9.368e+01 1.203e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 01:29:33,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1301246.6666666667, ans=0.0 2023-11-21 01:29:39,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195200 2023-11-21 01:29:48,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1301313.3333333333, ans=0.125 2023-11-21 01:30:03,525 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:30:16,101 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2850, loss[loss=0.05844, simple_loss=0.0741, pruned_loss=0.009701, audio_tagging_loss=0.01169, over 15123.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.0986, pruned_loss=0.01776, audio_tagging_loss=0.009646, over 3047254.55 frames. ], batch size: 58, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:30:43,717 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195250 2023-11-21 01:30:51,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1301646.6666666667, ans=0.1 2023-11-21 01:30:53,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1301713.3333333333, ans=0.125 2023-11-21 01:31:21,300 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2900, loss[loss=0.07456, simple_loss=0.08735, pruned_loss=0.02032, audio_tagging_loss=0.01056, over 15379.00 frames. ], tot_loss[loss=0.07718, simple_loss=0.09919, pruned_loss=0.018, audio_tagging_loss=0.009588, over 3044739.09 frames. ], batch size: 59, lr: 4.08e-03, grad_scale: 32.0 2023-11-21 01:31:22,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1301846.6666666667, ans=0.125 2023-11-21 01:31:26,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1301846.6666666667, ans=0.1 2023-11-21 01:31:30,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1301846.6666666667, ans=0.125 2023-11-21 01:31:34,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1301913.3333333333, ans=0.0 2023-11-21 01:31:36,825 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.167e+01 8.153e+01 9.015e+01 9.723e+01 1.263e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-21 01:31:48,104 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195300 2023-11-21 01:31:51,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1301980.0, ans=0.125 2023-11-21 01:31:54,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1301980.0, ans=0.125 2023-11-21 01:31:57,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1301980.0, ans=0.0 2023-11-21 01:32:05,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1302046.6666666667, ans=0.1 2023-11-21 01:32:16,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1302113.3333333333, ans=0.2 2023-11-21 01:32:26,140 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 2950, loss[loss=0.07981, simple_loss=0.1103, pruned_loss=0.01592, audio_tagging_loss=0.008739, over 15293.00 frames. ], tot_loss[loss=0.07756, simple_loss=0.09984, pruned_loss=0.01817, audio_tagging_loss=0.009479, over 3042355.72 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:32:27,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.20 vs. limit=15.0 2023-11-21 01:32:42,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.37 vs. limit=15.0 2023-11-21 01:32:53,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195350 2023-11-21 01:33:01,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1302313.3333333333, ans=0.2 2023-11-21 01:33:25,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1302446.6666666667, ans=0.2 2023-11-21 01:33:29,881 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3000, loss[loss=0.08554, simple_loss=0.1064, pruned_loss=0.02111, audio_tagging_loss=0.01121, over 16386.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09965, pruned_loss=0.01824, audio_tagging_loss=0.009521, over 3032333.21 frames. ], batch size: 60, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:33:29,881 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 01:33:52,464 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.0622, 3.5346, 4.0045, 3.7016], device='cuda:3') 2023-11-21 01:34:10,806 INFO [train_asr.py:1253] (3/4) Epoch 17, validation: loss=0.06009, simple_loss=0.05276, pruned_loss=0.005332, audio_tagging_loss=0.02838, over 4681554.00 frames. 2023-11-21 01:34:10,806 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 01:34:15,207 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.43 vs. limit=15.0 2023-11-21 01:34:26,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1302580.0, ans=0.2 2023-11-21 01:34:27,234 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.276e+01 8.086e+01 8.879e+01 9.716e+01 1.329e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-21 01:34:29,217 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=12.0 2023-11-21 01:34:37,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195400 2023-11-21 01:34:55,873 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.50 vs. limit=15.0 2023-11-21 01:35:07,713 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:35:15,250 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3050, loss[loss=0.06154, simple_loss=0.08131, pruned_loss=0.01296, audio_tagging_loss=0.00792, over 15246.00 frames. ], tot_loss[loss=0.0767, simple_loss=0.09853, pruned_loss=0.01781, audio_tagging_loss=0.009622, over 3033924.02 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:35:34,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1302913.3333333333, ans=0.2 2023-11-21 01:35:41,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195450 2023-11-21 01:35:43,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.43 vs. limit=15.0 2023-11-21 01:35:52,275 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:35:55,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1303046.6666666667, ans=0.0 2023-11-21 01:35:57,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1303046.6666666667, ans=0.125 2023-11-21 01:36:18,592 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3100, loss[loss=0.06417, simple_loss=0.07593, pruned_loss=0.0137, audio_tagging_loss=0.0125, over 15413.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09907, pruned_loss=0.01804, audio_tagging_loss=0.009682, over 3044418.22 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:36:21,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1303180.0, ans=0.1 2023-11-21 01:36:36,455 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.588e+01 8.198e+01 8.813e+01 9.373e+01 1.166e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-21 01:36:41,914 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0 2023-11-21 01:36:46,855 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195500 2023-11-21 01:36:47,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1303313.3333333333, ans=0.125 2023-11-21 01:36:49,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1303313.3333333333, ans=0.0 2023-11-21 01:36:56,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1303380.0, ans=0.0 2023-11-21 01:37:05,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1303380.0, ans=0.0 2023-11-21 01:37:16,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1303446.6666666667, ans=10.0 2023-11-21 01:37:17,443 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.58 vs. limit=15.0 2023-11-21 01:37:18,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1303446.6666666667, ans=0.2 2023-11-21 01:37:23,934 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3150, loss[loss=0.0783, simple_loss=0.1017, pruned_loss=0.01544, audio_tagging_loss=0.01203, over 14663.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09891, pruned_loss=0.01787, audio_tagging_loss=0.009808, over 3042753.74 frames. ], batch size: 54, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:37:50,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195550 2023-11-21 01:37:50,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1303646.6666666667, ans=0.125 2023-11-21 01:38:16,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1303780.0, ans=0.1 2023-11-21 01:38:19,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1303780.0, ans=0.125 2023-11-21 01:38:22,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1303780.0, ans=0.125 2023-11-21 01:38:28,829 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3200, loss[loss=0.1036, simple_loss=0.1303, pruned_loss=0.03036, audio_tagging_loss=0.008132, over 15522.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.0977, pruned_loss=0.01763, audio_tagging_loss=0.00996, over 3040412.03 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 32.0 2023-11-21 01:38:30,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1303846.6666666667, ans=0.0 2023-11-21 01:38:30,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.37 vs. limit=15.0 2023-11-21 01:38:44,767 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.303e+01 8.338e+01 8.946e+01 9.583e+01 3.907e+02, threshold=1.789e+02, percent-clipped=1.0 2023-11-21 01:38:56,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195600 2023-11-21 01:39:28,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.82 vs. limit=15.0 2023-11-21 01:39:29,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1304113.3333333333, ans=0.125 2023-11-21 01:39:31,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1304113.3333333333, ans=0.125 2023-11-21 01:39:33,047 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3250, loss[loss=0.0871, simple_loss=0.1189, pruned_loss=0.02106, audio_tagging_loss=0.006571, over 14646.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09742, pruned_loss=0.01756, audio_tagging_loss=0.009908, over 3033500.26 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 32.0 2023-11-21 01:39:35,168 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=13.07 vs. limit=15.0 2023-11-21 01:39:39,387 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0 2023-11-21 01:39:39,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1304180.0, ans=0.2 2023-11-21 01:39:46,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1304246.6666666667, ans=0.0 2023-11-21 01:39:49,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1304246.6666666667, ans=0.1 2023-11-21 01:40:00,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195650 2023-11-21 01:40:28,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1304446.6666666667, ans=0.2 2023-11-21 01:40:31,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1304446.6666666667, ans=0.1 2023-11-21 01:40:37,901 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3300, loss[loss=0.08942, simple_loss=0.1253, pruned_loss=0.01932, audio_tagging_loss=0.007476, over 16894.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09824, pruned_loss=0.01779, audio_tagging_loss=0.00994, over 3038909.48 frames. ], batch size: 60, lr: 4.07e-03, grad_scale: 32.0 2023-11-21 01:40:46,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1304513.3333333333, ans=0.125 2023-11-21 01:40:56,300 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.228e+01 8.124e+01 8.901e+01 9.520e+01 1.377e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-21 01:41:05,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195700 2023-11-21 01:41:22,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.98 vs. limit=15.0 2023-11-21 01:41:28,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1304780.0, ans=0.125 2023-11-21 01:41:29,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1304780.0, ans=0.5 2023-11-21 01:41:34,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1304780.0, ans=0.0 2023-11-21 01:41:41,989 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3350, loss[loss=0.04902, simple_loss=0.05948, pruned_loss=0.008723, audio_tagging_loss=0.01055, over 15509.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.0988, pruned_loss=0.01792, audio_tagging_loss=0.009771, over 3038083.78 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:42:00,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1304913.3333333333, ans=0.0 2023-11-21 01:42:08,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195750 2023-11-21 01:42:36,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1305113.3333333333, ans=0.1 2023-11-21 01:42:45,555 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3400, loss[loss=0.04651, simple_loss=0.05151, pruned_loss=0.01169, audio_tagging_loss=0.009067, over 14586.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.0983, pruned_loss=0.01774, audio_tagging_loss=0.009742, over 3035590.65 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:42:56,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1305180.0, ans=0.125 2023-11-21 01:43:00,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1305246.6666666667, ans=0.125 2023-11-21 01:43:03,980 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.029e+01 8.814e+01 9.645e+01 1.986e+02, threshold=1.763e+02, percent-clipped=1.0 2023-11-21 01:43:12,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195800 2023-11-21 01:43:22,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1305313.3333333333, ans=0.1 2023-11-21 01:43:39,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1305446.6666666667, ans=0.125 2023-11-21 01:43:45,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1305446.6666666667, ans=0.1 2023-11-21 01:43:51,475 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3450, loss[loss=0.0631, simple_loss=0.08087, pruned_loss=0.01223, audio_tagging_loss=0.01043, over 15528.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.09779, pruned_loss=0.01772, audio_tagging_loss=0.009665, over 3039855.43 frames. ], batch size: 59, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:44:03,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1305580.0, ans=0.1 2023-11-21 01:44:18,383 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195850 2023-11-21 01:44:25,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1305646.6666666667, ans=0.125 2023-11-21 01:44:26,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1305646.6666666667, ans=0.0 2023-11-21 01:44:26,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1305646.6666666667, ans=10.0 2023-11-21 01:44:30,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=12.0 2023-11-21 01:44:31,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1305713.3333333333, ans=0.125 2023-11-21 01:44:31,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1305713.3333333333, ans=0.125 2023-11-21 01:44:48,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=1305780.0, ans=0.05 2023-11-21 01:44:56,170 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3500, loss[loss=0.07038, simple_loss=0.08593, pruned_loss=0.01743, audio_tagging_loss=0.009978, over 15287.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09668, pruned_loss=0.01754, audio_tagging_loss=0.009671, over 3043492.72 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:44:58,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.99 vs. limit=22.5 2023-11-21 01:45:13,925 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.440e+01 8.307e+01 8.800e+01 9.894e+01 1.360e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-21 01:45:20,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1305980.0, ans=0.0 2023-11-21 01:45:23,403 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195900 2023-11-21 01:45:29,945 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:45:32,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1305980.0, ans=0.125 2023-11-21 01:45:53,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1306113.3333333333, ans=0.0 2023-11-21 01:46:00,386 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3550, loss[loss=0.05977, simple_loss=0.0727, pruned_loss=0.01168, audio_tagging_loss=0.01174, over 14002.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09631, pruned_loss=0.01745, audio_tagging_loss=0.009639, over 3040381.68 frames. ], batch size: 53, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:46:09,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1306180.0, ans=0.0 2023-11-21 01:46:18,171 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.05 vs. limit=15.0 2023-11-21 01:46:19,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1306246.6666666667, ans=0.2 2023-11-21 01:46:23,963 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:46:27,497 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 195950 2023-11-21 01:46:28,775 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 01:46:31,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.63 vs. limit=22.5 2023-11-21 01:46:39,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1306380.0, ans=0.2 2023-11-21 01:46:45,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.20 vs. limit=15.0 2023-11-21 01:46:52,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1306446.6666666667, ans=0.1 2023-11-21 01:46:55,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1306446.6666666667, ans=0.125 2023-11-21 01:47:04,451 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3600, loss[loss=0.07262, simple_loss=0.09994, pruned_loss=0.01613, audio_tagging_loss=0.006516, over 14926.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09637, pruned_loss=0.01744, audio_tagging_loss=0.009593, over 3041143.73 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 32.0 2023-11-21 01:47:11,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1306513.3333333333, ans=0.1 2023-11-21 01:47:19,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1306580.0, ans=0.07 2023-11-21 01:47:22,589 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.078e+01 8.527e+01 9.382e+01 1.423e+02, threshold=1.705e+02, percent-clipped=0.0 2023-11-21 01:47:24,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1306580.0, ans=0.2 2023-11-21 01:47:30,580 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.17 vs. limit=22.5 2023-11-21 01:47:31,104 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196000 2023-11-21 01:47:37,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1306646.6666666667, ans=0.125 2023-11-21 01:48:02,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1306780.0, ans=0.125 2023-11-21 01:48:12,021 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3650, loss[loss=0.0821, simple_loss=0.1128, pruned_loss=0.01901, audio_tagging_loss=0.006699, over 14944.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09675, pruned_loss=0.01754, audio_tagging_loss=0.009513, over 3039738.08 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 32.0 2023-11-21 01:48:38,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196050 2023-11-21 01:49:06,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1307113.3333333333, ans=0.2 2023-11-21 01:49:14,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1307180.0, ans=0.1 2023-11-21 01:49:15,773 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3700, loss[loss=0.08662, simple_loss=0.1129, pruned_loss=0.01974, audio_tagging_loss=0.01042, over 16427.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09599, pruned_loss=0.01735, audio_tagging_loss=0.009672, over 3037610.62 frames. ], batch size: 59, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:49:15,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1307180.0, ans=0.0 2023-11-21 01:49:16,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.58 vs. limit=15.0 2023-11-21 01:49:24,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1307180.0, ans=0.125 2023-11-21 01:49:24,608 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.71 vs. limit=15.0 2023-11-21 01:49:25,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1307180.0, ans=0.0 2023-11-21 01:49:36,156 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.153e+01 8.365e+01 9.163e+01 1.051e+02 1.464e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-21 01:49:36,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1307246.6666666667, ans=0.0 2023-11-21 01:49:39,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1307246.6666666667, ans=0.125 2023-11-21 01:49:43,715 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196100 2023-11-21 01:50:00,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1307380.0, ans=0.0 2023-11-21 01:50:15,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1307446.6666666667, ans=0.125 2023-11-21 01:50:17,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1307446.6666666667, ans=0.125 2023-11-21 01:50:21,109 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3750, loss[loss=0.0746, simple_loss=0.0963, pruned_loss=0.01656, audio_tagging_loss=0.009883, over 15304.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09665, pruned_loss=0.01758, audio_tagging_loss=0.009639, over 3038686.77 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:50:22,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1307513.3333333333, ans=0.125 2023-11-21 01:50:47,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1307646.6666666667, ans=0.125 2023-11-21 01:50:48,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196150 2023-11-21 01:51:00,515 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.58 vs. limit=22.5 2023-11-21 01:51:05,795 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:51:25,106 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3800, loss[loss=0.07444, simple_loss=0.09183, pruned_loss=0.01745, audio_tagging_loss=0.01108, over 14121.00 frames. ], tot_loss[loss=0.07672, simple_loss=0.09835, pruned_loss=0.01796, audio_tagging_loss=0.009587, over 3044385.79 frames. ], batch size: 53, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:51:44,157 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.062e+01 7.915e+01 8.603e+01 9.483e+01 1.210e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-21 01:51:45,040 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.86 vs. limit=15.0 2023-11-21 01:51:52,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196200 2023-11-21 01:52:29,391 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3850, loss[loss=0.07169, simple_loss=0.08803, pruned_loss=0.01762, audio_tagging_loss=0.01005, over 15100.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.09852, pruned_loss=0.01798, audio_tagging_loss=0.009738, over 3042457.71 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 16.0 2023-11-21 01:52:30,001 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.78 vs. limit=22.5 2023-11-21 01:52:30,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1308180.0, ans=0.125 2023-11-21 01:52:34,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1308180.0, ans=0.125 2023-11-21 01:52:36,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1308180.0, ans=0.125 2023-11-21 01:52:40,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1308180.0, ans=10.0 2023-11-21 01:52:41,661 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.72 vs. limit=15.0 2023-11-21 01:52:43,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1308246.6666666667, ans=0.0 2023-11-21 01:52:51,288 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.84 vs. limit=22.5 2023-11-21 01:52:56,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196250 2023-11-21 01:52:58,565 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.28 vs. limit=10.0 2023-11-21 01:53:12,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1308380.0, ans=0.0 2023-11-21 01:53:28,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.58 vs. limit=15.0 2023-11-21 01:53:33,582 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3900, loss[loss=0.07655, simple_loss=0.1048, pruned_loss=0.01437, audio_tagging_loss=0.00979, over 16297.00 frames. ], tot_loss[loss=0.07675, simple_loss=0.09844, pruned_loss=0.0178, audio_tagging_loss=0.009735, over 3038706.84 frames. ], batch size: 59, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 01:53:52,213 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.160e+01 8.803e+01 9.797e+01 1.334e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-21 01:54:00,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196300 2023-11-21 01:54:07,089 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.48 vs. limit=15.0 2023-11-21 01:54:33,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.31 vs. limit=15.0 2023-11-21 01:54:37,449 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 3950, loss[loss=0.1059, simple_loss=0.1275, pruned_loss=0.03243, audio_tagging_loss=0.009733, over 15550.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.0994, pruned_loss=0.01809, audio_tagging_loss=0.009795, over 3044905.49 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 01:54:41,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1308846.6666666667, ans=0.125 2023-11-21 01:54:50,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1308913.3333333333, ans=0.07 2023-11-21 01:54:56,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1308913.3333333333, ans=0.2 2023-11-21 01:55:03,825 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196350 2023-11-21 01:55:06,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1308980.0, ans=0.125 2023-11-21 01:55:25,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1309046.6666666667, ans=0.1 2023-11-21 01:55:29,889 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.61 vs. limit=10.0 2023-11-21 01:55:36,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1309113.3333333333, ans=0.1 2023-11-21 01:55:41,774 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4000, loss[loss=0.06989, simple_loss=0.08816, pruned_loss=0.01405, audio_tagging_loss=0.01176, over 15737.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09801, pruned_loss=0.01769, audio_tagging_loss=0.009972, over 3040260.96 frames. ], batch size: 59, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 01:55:47,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1309180.0, ans=0.125 2023-11-21 01:56:00,786 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.301e+01 8.874e+01 9.780e+01 1.152e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-21 01:56:09,667 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196400 2023-11-21 01:56:45,650 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4050, loss[loss=0.07594, simple_loss=0.09415, pruned_loss=0.01674, audio_tagging_loss=0.01212, over 14385.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.09785, pruned_loss=0.01757, audio_tagging_loss=0.01003, over 3032131.82 frames. ], batch size: 55, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 01:56:49,891 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:57:06,173 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.49 vs. limit=15.0 2023-11-21 01:57:07,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1309580.0, ans=0.1 2023-11-21 01:57:08,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1309580.0, ans=0.125 2023-11-21 01:57:12,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1309646.6666666667, ans=0.125 2023-11-21 01:57:13,967 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196450 2023-11-21 01:57:24,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1309713.3333333333, ans=0.2 2023-11-21 01:57:26,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1309713.3333333333, ans=0.125 2023-11-21 01:57:42,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1309780.0, ans=0.1 2023-11-21 01:57:51,793 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4100, loss[loss=0.08334, simple_loss=0.1089, pruned_loss=0.02049, audio_tagging_loss=0.008414, over 14369.00 frames. ], tot_loss[loss=0.07639, simple_loss=0.0977, pruned_loss=0.01751, audio_tagging_loss=0.01003, over 3036909.50 frames. ], batch size: 55, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 01:58:10,683 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.112e+01 8.710e+01 9.440e+01 1.199e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-21 01:58:18,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196500 2023-11-21 01:58:22,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1309980.0, ans=0.0 2023-11-21 01:58:26,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1309980.0, ans=15.0 2023-11-21 01:58:34,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1310046.6666666667, ans=0.0 2023-11-21 01:58:34,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1310046.6666666667, ans=0.0 2023-11-21 01:58:38,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1310046.6666666667, ans=0.125 2023-11-21 01:58:50,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1310113.3333333333, ans=0.1 2023-11-21 01:58:56,154 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4150, loss[loss=0.06105, simple_loss=0.07542, pruned_loss=0.01315, audio_tagging_loss=0.01019, over 15374.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09764, pruned_loss=0.01751, audio_tagging_loss=0.009825, over 3042157.09 frames. ], batch size: 60, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 01:59:19,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1310246.6666666667, ans=0.125 2023-11-21 01:59:20,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1310313.3333333333, ans=0.1 2023-11-21 01:59:23,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196550 2023-11-21 01:59:27,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1310313.3333333333, ans=0.125 2023-11-21 01:59:43,278 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 01:59:43,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1310380.0, ans=0.125 2023-11-21 02:00:00,196 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4200, loss[loss=0.06914, simple_loss=0.08265, pruned_loss=0.01807, audio_tagging_loss=0.009741, over 14510.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.09767, pruned_loss=0.01756, audio_tagging_loss=0.009798, over 3056198.57 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:00:04,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1310513.3333333333, ans=0.95 2023-11-21 02:00:05,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1310513.3333333333, ans=0.125 2023-11-21 02:00:07,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1310513.3333333333, ans=0.0 2023-11-21 02:00:14,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1310580.0, ans=0.125 2023-11-21 02:00:15,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1310580.0, ans=0.2 2023-11-21 02:00:20,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1310580.0, ans=0.09899494936611666 2023-11-21 02:00:21,783 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.242e+01 8.811e+01 9.654e+01 1.216e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-21 02:00:23,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1310580.0, ans=0.5 2023-11-21 02:00:28,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196600 2023-11-21 02:00:52,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1310780.0, ans=0.0 2023-11-21 02:01:05,071 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4250, loss[loss=0.07765, simple_loss=0.09749, pruned_loss=0.01819, audio_tagging_loss=0.01071, over 15060.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09719, pruned_loss=0.0174, audio_tagging_loss=0.009787, over 3054502.42 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:01:05,826 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.37 vs. limit=15.0 2023-11-21 02:01:32,418 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196650 2023-11-21 02:01:56,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.95 vs. limit=15.0 2023-11-21 02:01:57,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.59 vs. limit=15.0 2023-11-21 02:01:59,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1311113.3333333333, ans=0.1 2023-11-21 02:02:09,886 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4300, loss[loss=0.07226, simple_loss=0.1025, pruned_loss=0.01274, audio_tagging_loss=0.008249, over 15373.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09717, pruned_loss=0.01727, audio_tagging_loss=0.009721, over 3054611.94 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:02:16,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1311180.0, ans=0.95 2023-11-21 02:02:22,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.32 vs. limit=15.0 2023-11-21 02:02:29,311 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.098e+01 8.656e+01 9.268e+01 1.139e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 02:02:32,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.22 vs. limit=15.0 2023-11-21 02:02:36,090 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196700 2023-11-21 02:02:40,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1311313.3333333333, ans=0.0 2023-11-21 02:02:41,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1311313.3333333333, ans=0.0 2023-11-21 02:02:46,965 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.30 vs. limit=15.0 2023-11-21 02:02:46,983 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.13 vs. limit=10.0 2023-11-21 02:02:47,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1311380.0, ans=0.0 2023-11-21 02:02:52,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1311380.0, ans=0.0 2023-11-21 02:02:52,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1311380.0, ans=0.125 2023-11-21 02:03:01,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1311446.6666666667, ans=0.07 2023-11-21 02:03:01,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1311446.6666666667, ans=0.125 2023-11-21 02:03:13,558 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4350, loss[loss=0.05784, simple_loss=0.06715, pruned_loss=0.01228, audio_tagging_loss=0.01198, over 15187.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09726, pruned_loss=0.01733, audio_tagging_loss=0.009719, over 3045273.93 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:03:15,332 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.81 vs. limit=22.5 2023-11-21 02:03:33,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1311580.0, ans=0.125 2023-11-21 02:03:37,661 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.25 vs. limit=12.0 2023-11-21 02:03:40,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196750 2023-11-21 02:03:42,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1311646.6666666667, ans=0.5 2023-11-21 02:03:42,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1311646.6666666667, ans=0.0 2023-11-21 02:03:42,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1311646.6666666667, ans=0.125 2023-11-21 02:03:53,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1311713.3333333333, ans=0.0 2023-11-21 02:04:13,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1311780.0, ans=0.04949747468305833 2023-11-21 02:04:17,746 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4400, loss[loss=0.0608, simple_loss=0.07736, pruned_loss=0.01188, audio_tagging_loss=0.01024, over 14330.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09641, pruned_loss=0.01723, audio_tagging_loss=0.009758, over 3034849.40 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 02:04:34,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1311913.3333333333, ans=0.0 2023-11-21 02:04:38,508 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 7.827e+01 8.393e+01 8.998e+01 1.090e+02, threshold=1.679e+02, percent-clipped=0.0 2023-11-21 02:04:44,780 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196800 2023-11-21 02:05:22,837 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4450, loss[loss=0.07776, simple_loss=0.1027, pruned_loss=0.01829, audio_tagging_loss=0.008146, over 15348.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09763, pruned_loss=0.01766, audio_tagging_loss=0.009636, over 3035152.11 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 02:05:49,468 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196850 2023-11-21 02:05:52,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1312313.3333333333, ans=0.1 2023-11-21 02:06:10,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1312380.0, ans=0.1 2023-11-21 02:06:11,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1312380.0, ans=0.0 2023-11-21 02:06:18,390 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 02:06:23,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1312446.6666666667, ans=0.125 2023-11-21 02:06:26,539 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4500, loss[loss=0.08977, simple_loss=0.1225, pruned_loss=0.01964, audio_tagging_loss=0.008883, over 15933.00 frames. ], tot_loss[loss=0.07672, simple_loss=0.09837, pruned_loss=0.01792, audio_tagging_loss=0.009615, over 3047930.33 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:06:26,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1312513.3333333333, ans=0.0 2023-11-21 02:06:48,062 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.230e+01 8.666e+01 9.490e+01 1.272e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 02:06:48,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1312580.0, ans=0.1 2023-11-21 02:06:53,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196900 2023-11-21 02:07:02,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1312646.6666666667, ans=0.0 2023-11-21 02:07:09,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1312713.3333333333, ans=0.125 2023-11-21 02:07:24,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1312780.0, ans=0.125 2023-11-21 02:07:25,536 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.47 vs. limit=15.0 2023-11-21 02:07:29,814 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.70 vs. limit=10.0 2023-11-21 02:07:30,281 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4550, loss[loss=0.05606, simple_loss=0.06538, pruned_loss=0.009451, audio_tagging_loss=0.01392, over 15237.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.09823, pruned_loss=0.01767, audio_tagging_loss=0.009732, over 3041701.62 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:07:43,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1312913.3333333333, ans=0.0 2023-11-21 02:07:51,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1312913.3333333333, ans=0.2 2023-11-21 02:07:52,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1312913.3333333333, ans=0.125 2023-11-21 02:07:57,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 196950 2023-11-21 02:07:58,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1312980.0, ans=0.05 2023-11-21 02:08:19,057 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 02:08:28,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1313113.3333333333, ans=0.125 2023-11-21 02:08:34,907 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4600, loss[loss=0.07855, simple_loss=0.09295, pruned_loss=0.01718, audio_tagging_loss=0.01489, over 16390.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09735, pruned_loss=0.0176, audio_tagging_loss=0.009888, over 3035427.29 frames. ], batch size: 61, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:08:52,083 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.89 vs. limit=15.0 2023-11-21 02:08:56,245 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.061e+01 8.654e+01 9.730e+01 1.161e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 02:09:01,175 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197000 2023-11-21 02:09:01,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1313313.3333333333, ans=0.125 2023-11-21 02:09:07,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1313313.3333333333, ans=0.0 2023-11-21 02:09:38,874 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4650, loss[loss=0.07465, simple_loss=0.0877, pruned_loss=0.01739, audio_tagging_loss=0.01341, over 14933.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09798, pruned_loss=0.0178, audio_tagging_loss=0.009847, over 3043188.21 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:09:51,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1313580.0, ans=0.1 2023-11-21 02:09:57,403 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.22 vs. limit=15.0 2023-11-21 02:10:01,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1313580.0, ans=0.2 2023-11-21 02:10:05,730 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197050 2023-11-21 02:10:08,660 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.88 vs. limit=15.0 2023-11-21 02:10:11,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0 2023-11-21 02:10:16,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1313713.3333333333, ans=0.125 2023-11-21 02:10:28,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1313713.3333333333, ans=0.1 2023-11-21 02:10:34,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1313780.0, ans=0.125 2023-11-21 02:10:42,958 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4700, loss[loss=0.06835, simple_loss=0.08245, pruned_loss=0.01488, audio_tagging_loss=0.01224, over 15111.00 frames. ], tot_loss[loss=0.07724, simple_loss=0.09865, pruned_loss=0.01808, audio_tagging_loss=0.009835, over 3049855.47 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:10:49,014 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.60 vs. limit=22.5 2023-11-21 02:10:50,109 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.05 vs. limit=22.5 2023-11-21 02:11:03,043 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.42 vs. limit=22.5 2023-11-21 02:11:03,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1313913.3333333333, ans=0.125 2023-11-21 02:11:04,736 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.068e+01 8.673e+01 9.283e+01 1.102e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 02:11:04,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1313913.3333333333, ans=0.125 2023-11-21 02:11:09,826 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197100 2023-11-21 02:11:10,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1313980.0, ans=0.5 2023-11-21 02:11:46,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0 2023-11-21 02:11:47,014 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4750, loss[loss=0.08636, simple_loss=0.1219, pruned_loss=0.01881, audio_tagging_loss=0.006604, over 15057.00 frames. ], tot_loss[loss=0.07673, simple_loss=0.09816, pruned_loss=0.01789, audio_tagging_loss=0.009759, over 3050599.87 frames. ], batch size: 54, lr: 4.06e-03, grad_scale: 16.0 2023-11-21 02:12:05,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1314246.6666666667, ans=0.125 2023-11-21 02:12:13,879 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197150 2023-11-21 02:12:24,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1314380.0, ans=0.1 2023-11-21 02:12:27,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1314380.0, ans=0.0 2023-11-21 02:12:32,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1314380.0, ans=0.035 2023-11-21 02:12:50,793 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4800, loss[loss=0.1006, simple_loss=0.1279, pruned_loss=0.02891, audio_tagging_loss=0.007726, over 14730.00 frames. ], tot_loss[loss=0.07675, simple_loss=0.09796, pruned_loss=0.01795, audio_tagging_loss=0.009818, over 3043784.76 frames. ], batch size: 55, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 02:12:56,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1314513.3333333333, ans=0.125 2023-11-21 02:13:06,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1314580.0, ans=0.09899494936611666 2023-11-21 02:13:11,900 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 02:13:12,700 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.143e+01 8.825e+01 9.696e+01 1.559e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-21 02:13:18,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197200 2023-11-21 02:13:33,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1314713.3333333333, ans=0.125 2023-11-21 02:13:40,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.36 vs. limit=15.0 2023-11-21 02:13:44,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1314780.0, ans=0.0 2023-11-21 02:13:55,408 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4850, loss[loss=0.08419, simple_loss=0.1097, pruned_loss=0.01627, audio_tagging_loss=0.01307, over 15116.00 frames. ], tot_loss[loss=0.07769, simple_loss=0.09936, pruned_loss=0.01809, audio_tagging_loss=0.009919, over 3042529.77 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 32.0 2023-11-21 02:13:59,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1314846.6666666667, ans=0.125 2023-11-21 02:14:05,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1314846.6666666667, ans=15.0 2023-11-21 02:14:20,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1314980.0, ans=0.1 2023-11-21 02:14:22,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197250 2023-11-21 02:14:34,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1315046.6666666667, ans=0.125 2023-11-21 02:14:58,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.40 vs. limit=15.0 2023-11-21 02:14:59,942 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4900, loss[loss=0.0744, simple_loss=0.1027, pruned_loss=0.01549, audio_tagging_loss=0.007532, over 15221.00 frames. ], tot_loss[loss=0.07718, simple_loss=0.09873, pruned_loss=0.01789, audio_tagging_loss=0.009931, over 3036708.19 frames. ], batch size: 56, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:15:21,167 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.125e+01 8.648e+01 9.394e+01 1.252e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-21 02:15:26,759 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197300 2023-11-21 02:15:34,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1315313.3333333333, ans=0.2 2023-11-21 02:15:45,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1315380.0, ans=0.1 2023-11-21 02:16:02,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1315513.3333333333, ans=0.125 2023-11-21 02:16:03,686 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 4950, loss[loss=0.05981, simple_loss=0.08089, pruned_loss=0.01091, audio_tagging_loss=0.008455, over 14652.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09793, pruned_loss=0.01766, audio_tagging_loss=0.009791, over 3038127.53 frames. ], batch size: 53, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:16:13,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1315513.3333333333, ans=0.0 2023-11-21 02:16:17,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1315580.0, ans=0.125 2023-11-21 02:16:19,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1315580.0, ans=0.2 2023-11-21 02:16:31,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197350 2023-11-21 02:16:37,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1315646.6666666667, ans=0.0 2023-11-21 02:16:47,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1315713.3333333333, ans=0.0 2023-11-21 02:16:50,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1315713.3333333333, ans=0.125 2023-11-21 02:17:02,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1315780.0, ans=0.125 2023-11-21 02:17:05,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1315780.0, ans=0.2 2023-11-21 02:17:07,626 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5000, loss[loss=0.05543, simple_loss=0.07045, pruned_loss=0.01223, audio_tagging_loss=0.007978, over 15334.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09847, pruned_loss=0.01766, audio_tagging_loss=0.009615, over 3038585.45 frames. ], batch size: 59, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:17:09,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1315846.6666666667, ans=0.1 2023-11-21 02:17:18,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1315846.6666666667, ans=0.125 2023-11-21 02:17:19,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1315913.3333333333, ans=0.0 2023-11-21 02:17:29,321 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.139e+01 8.846e+01 9.798e+01 1.314e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 02:17:34,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197400 2023-11-21 02:17:46,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1316046.6666666667, ans=0.2 2023-11-21 02:17:47,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1316046.6666666667, ans=0.125 2023-11-21 02:17:49,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1316046.6666666667, ans=0.1 2023-11-21 02:17:55,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1316046.6666666667, ans=0.5 2023-11-21 02:18:07,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1316113.3333333333, ans=0.125 2023-11-21 02:18:12,061 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5050, loss[loss=0.04963, simple_loss=0.06334, pruned_loss=0.007696, audio_tagging_loss=0.01026, over 15801.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09795, pruned_loss=0.0175, audio_tagging_loss=0.009538, over 3041487.26 frames. ], batch size: 61, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:18:26,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1316246.6666666667, ans=0.1 2023-11-21 02:18:30,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1316246.6666666667, ans=0.125 2023-11-21 02:18:32,691 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=15.0 2023-11-21 02:18:38,402 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197450 2023-11-21 02:18:47,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1316313.3333333333, ans=0.125 2023-11-21 02:18:48,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1316380.0, ans=0.125 2023-11-21 02:19:13,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1316446.6666666667, ans=0.125 2023-11-21 02:19:16,006 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5100, loss[loss=0.04799, simple_loss=0.06601, pruned_loss=0.008704, audio_tagging_loss=0.006282, over 15513.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09773, pruned_loss=0.01758, audio_tagging_loss=0.009606, over 3039488.23 frames. ], batch size: 60, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:19:37,079 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.469e+01 7.939e+01 8.776e+01 9.689e+01 1.456e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 02:19:42,567 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197500 2023-11-21 02:20:18,848 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5150, loss[loss=0.09188, simple_loss=0.1291, pruned_loss=0.02032, audio_tagging_loss=0.00703, over 15197.00 frames. ], tot_loss[loss=0.07564, simple_loss=0.09728, pruned_loss=0.01738, audio_tagging_loss=0.009617, over 3032494.12 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:20:22,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1316846.6666666667, ans=0.125 2023-11-21 02:20:24,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1316846.6666666667, ans=0.125 2023-11-21 02:20:26,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1316846.6666666667, ans=0.0 2023-11-21 02:20:31,066 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 02:20:37,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1316913.3333333333, ans=0.0 2023-11-21 02:20:46,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197550 2023-11-21 02:21:23,745 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5200, loss[loss=0.07434, simple_loss=0.09962, pruned_loss=0.0166, audio_tagging_loss=0.007929, over 16937.00 frames. ], tot_loss[loss=0.07658, simple_loss=0.09862, pruned_loss=0.01766, audio_tagging_loss=0.00961, over 3036051.85 frames. ], batch size: 64, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:21:23,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1317180.0, ans=0.125 2023-11-21 02:21:24,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1317180.0, ans=0.125 2023-11-21 02:21:39,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1317246.6666666667, ans=0.0 2023-11-21 02:21:44,681 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.249e+01 8.824e+01 9.467e+01 1.175e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-21 02:21:48,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1317313.3333333333, ans=0.0 2023-11-21 02:21:49,209 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.75 vs. limit=15.0 2023-11-21 02:21:49,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197600 2023-11-21 02:21:55,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.22 vs. limit=22.5 2023-11-21 02:22:15,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1317446.6666666667, ans=0.125 2023-11-21 02:22:27,623 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5250, loss[loss=0.09087, simple_loss=0.1137, pruned_loss=0.02497, audio_tagging_loss=0.009026, over 15997.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.09949, pruned_loss=0.0179, audio_tagging_loss=0.009544, over 3043498.01 frames. ], batch size: 58, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:22:27,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1317513.3333333333, ans=10.0 2023-11-21 02:22:54,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197650 2023-11-21 02:22:56,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1317646.6666666667, ans=0.0 2023-11-21 02:23:04,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.83 vs. limit=15.0 2023-11-21 02:23:21,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1317780.0, ans=0.0 2023-11-21 02:23:30,866 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5300, loss[loss=0.06552, simple_loss=0.07899, pruned_loss=0.01558, audio_tagging_loss=0.01044, over 13616.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09883, pruned_loss=0.01779, audio_tagging_loss=0.009565, over 3039944.92 frames. ], batch size: 53, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:23:33,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1317846.6666666667, ans=0.0 2023-11-21 02:23:39,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1317846.6666666667, ans=0.125 2023-11-21 02:23:53,260 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.065e+01 8.722e+01 9.594e+01 1.316e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-21 02:23:58,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197700 2023-11-21 02:23:58,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1317980.0, ans=0.07 2023-11-21 02:24:03,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1317980.0, ans=0.125 2023-11-21 02:24:20,173 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.14 vs. limit=22.5 2023-11-21 02:24:34,820 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5350, loss[loss=0.06159, simple_loss=0.08272, pruned_loss=0.01002, audio_tagging_loss=0.01021, over 15571.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.09916, pruned_loss=0.01795, audio_tagging_loss=0.009518, over 3045194.13 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:24:57,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.62 vs. limit=15.0 2023-11-21 02:25:02,200 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197750 2023-11-21 02:25:02,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1318313.3333333333, ans=0.125 2023-11-21 02:25:22,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1318380.0, ans=0.125 2023-11-21 02:25:38,680 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5400, loss[loss=0.1042, simple_loss=0.1236, pruned_loss=0.03073, audio_tagging_loss=0.01161, over 15267.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.0993, pruned_loss=0.01796, audio_tagging_loss=0.00952, over 3045811.00 frames. ], batch size: 59, lr: 4.05e-03, grad_scale: 32.0 2023-11-21 02:25:40,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1318513.3333333333, ans=0.0 2023-11-21 02:25:46,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1318513.3333333333, ans=0.0 2023-11-21 02:26:00,531 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.677e+01 8.214e+01 8.750e+01 9.356e+01 1.169e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 02:26:04,384 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197800 2023-11-21 02:26:04,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1318646.6666666667, ans=0.0 2023-11-21 02:26:05,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1318646.6666666667, ans=0.1 2023-11-21 02:26:29,190 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.77 vs. limit=15.0 2023-11-21 02:26:30,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1318780.0, ans=0.125 2023-11-21 02:26:31,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1318780.0, ans=0.1 2023-11-21 02:26:32,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1318780.0, ans=0.1 2023-11-21 02:26:33,053 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.72 vs. limit=15.0 2023-11-21 02:26:38,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1318780.0, ans=0.1 2023-11-21 02:26:41,774 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5450, loss[loss=0.04619, simple_loss=0.05243, pruned_loss=0.009399, audio_tagging_loss=0.01058, over 14309.00 frames. ], tot_loss[loss=0.07754, simple_loss=0.0996, pruned_loss=0.01812, audio_tagging_loss=0.009622, over 3052011.17 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:26:43,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1318846.6666666667, ans=0.125 2023-11-21 02:26:43,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1318846.6666666667, ans=0.2 2023-11-21 02:26:59,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1318913.3333333333, ans=0.125 2023-11-21 02:27:01,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1318913.3333333333, ans=0.07 2023-11-21 02:27:01,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1318913.3333333333, ans=0.1 2023-11-21 02:27:01,696 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.28 vs. limit=22.5 2023-11-21 02:27:02,633 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.425e-01 2023-11-21 02:27:09,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197850 2023-11-21 02:27:15,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1318980.0, ans=0.0 2023-11-21 02:27:27,009 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.66 vs. limit=10.0 2023-11-21 02:27:31,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1319113.3333333333, ans=0.0 2023-11-21 02:27:45,095 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5500, loss[loss=0.08479, simple_loss=0.1108, pruned_loss=0.01936, audio_tagging_loss=0.01004, over 15536.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09975, pruned_loss=0.01815, audio_tagging_loss=0.009684, over 3053669.52 frames. ], batch size: 58, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:28:07,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1319246.6666666667, ans=0.125 2023-11-21 02:28:08,757 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.510e+01 8.155e+01 8.783e+01 9.511e+01 1.245e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-21 02:28:12,539 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197900 2023-11-21 02:28:12,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1319313.3333333333, ans=0.0 2023-11-21 02:28:12,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1319313.3333333333, ans=0.125 2023-11-21 02:28:25,994 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 02:28:26,586 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.39 vs. limit=10.0 2023-11-21 02:28:41,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1319446.6666666667, ans=0.0 2023-11-21 02:28:49,905 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5550, loss[loss=0.06845, simple_loss=0.09012, pruned_loss=0.01351, audio_tagging_loss=0.009876, over 14671.00 frames. ], tot_loss[loss=0.0771, simple_loss=0.09853, pruned_loss=0.01797, audio_tagging_loss=0.00987, over 3052144.56 frames. ], batch size: 54, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:28:51,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.90 vs. limit=15.0 2023-11-21 02:28:58,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1319513.3333333333, ans=0.0 2023-11-21 02:28:58,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1319513.3333333333, ans=0.0 2023-11-21 02:29:15,328 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 197950 2023-11-21 02:29:31,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1319713.3333333333, ans=0.125 2023-11-21 02:29:36,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1319713.3333333333, ans=0.2 2023-11-21 02:29:44,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1319780.0, ans=0.1 2023-11-21 02:29:46,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.51 vs. limit=15.0 2023-11-21 02:29:52,783 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5600, loss[loss=0.1131, simple_loss=0.1451, pruned_loss=0.03159, audio_tagging_loss=0.008936, over 15268.00 frames. ], tot_loss[loss=0.07782, simple_loss=0.09952, pruned_loss=0.01823, audio_tagging_loss=0.009828, over 3056044.33 frames. ], batch size: 53, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:29:56,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1319846.6666666667, ans=0.125 2023-11-21 02:30:07,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1319913.3333333333, ans=0.0 2023-11-21 02:30:15,401 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.50 vs. limit=22.5 2023-11-21 02:30:17,151 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.123e+01 8.788e+01 9.495e+01 1.515e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 02:30:17,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1319980.0, ans=0.1 2023-11-21 02:30:19,712 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198000 2023-11-21 02:30:32,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1320046.6666666667, ans=0.0 2023-11-21 02:30:37,257 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=15.0 2023-11-21 02:30:37,685 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 02:30:47,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1320113.3333333333, ans=0.09899494936611666 2023-11-21 02:30:51,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1320113.3333333333, ans=0.125 2023-11-21 02:30:55,890 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5650, loss[loss=0.06882, simple_loss=0.08391, pruned_loss=0.01416, audio_tagging_loss=0.01271, over 14265.00 frames. ], tot_loss[loss=0.07735, simple_loss=0.09896, pruned_loss=0.01794, audio_tagging_loss=0.009935, over 3052232.31 frames. ], batch size: 55, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:31:21,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1320313.3333333333, ans=0.0 2023-11-21 02:31:23,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198050 2023-11-21 02:31:26,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1320313.3333333333, ans=0.1 2023-11-21 02:31:59,780 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5700, loss[loss=0.08091, simple_loss=0.1029, pruned_loss=0.01728, audio_tagging_loss=0.0122, over 14719.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.09844, pruned_loss=0.01782, audio_tagging_loss=0.009893, over 3046952.60 frames. ], batch size: 56, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:32:09,147 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.49 vs. limit=15.0 2023-11-21 02:32:13,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1320580.0, ans=0.0 2023-11-21 02:32:14,976 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.92 vs. limit=22.5 2023-11-21 02:32:16,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1320580.0, ans=0.0 2023-11-21 02:32:21,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1320580.0, ans=0.125 2023-11-21 02:32:23,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1320580.0, ans=0.2 2023-11-21 02:32:23,972 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.139e+01 8.085e+01 8.943e+01 9.516e+01 1.355e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-21 02:32:26,602 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198100 2023-11-21 02:32:42,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0 2023-11-21 02:32:52,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1320780.0, ans=0.125 2023-11-21 02:32:54,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1320780.0, ans=0.0 2023-11-21 02:33:03,883 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5750, loss[loss=0.07491, simple_loss=0.09658, pruned_loss=0.0146, audio_tagging_loss=0.01202, over 15295.00 frames. ], tot_loss[loss=0.07655, simple_loss=0.0981, pruned_loss=0.01775, audio_tagging_loss=0.009745, over 3045670.96 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:33:06,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1320846.6666666667, ans=0.09899494936611666 2023-11-21 02:33:30,769 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198150 2023-11-21 02:33:32,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1320980.0, ans=0.0 2023-11-21 02:33:32,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1320980.0, ans=0.07 2023-11-21 02:33:54,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1321113.3333333333, ans=0.125 2023-11-21 02:34:06,825 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5800, loss[loss=0.05624, simple_loss=0.07206, pruned_loss=0.01078, audio_tagging_loss=0.009426, over 16018.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09823, pruned_loss=0.0179, audio_tagging_loss=0.009699, over 3046246.60 frames. ], batch size: 63, lr: 4.05e-03, grad_scale: 16.0 2023-11-21 02:34:13,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1321180.0, ans=0.125 2023-11-21 02:34:18,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys.whitening_limit, batch_count=1321180.0, ans=6.0 2023-11-21 02:34:20,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1321246.6666666667, ans=0.2 2023-11-21 02:34:30,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1321246.6666666667, ans=0.125 2023-11-21 02:34:31,410 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.619e+01 8.028e+01 8.750e+01 9.322e+01 1.418e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 02:34:34,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198200 2023-11-21 02:34:50,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.95 vs. limit=15.0 2023-11-21 02:35:11,408 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5850, loss[loss=0.08128, simple_loss=0.1055, pruned_loss=0.01716, audio_tagging_loss=0.01138, over 15593.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09746, pruned_loss=0.01765, audio_tagging_loss=0.009678, over 3045534.35 frames. ], batch size: 59, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:35:11,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1321513.3333333333, ans=0.125 2023-11-21 02:35:32,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1321580.0, ans=0.1 2023-11-21 02:35:37,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1321646.6666666667, ans=0.1 2023-11-21 02:35:38,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198250 2023-11-21 02:35:39,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1321646.6666666667, ans=0.125 2023-11-21 02:35:40,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1321646.6666666667, ans=0.09899494936611666 2023-11-21 02:36:15,426 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5900, loss[loss=0.07231, simple_loss=0.09701, pruned_loss=0.01594, audio_tagging_loss=0.007869, over 14934.00 frames. ], tot_loss[loss=0.07633, simple_loss=0.09811, pruned_loss=0.01762, audio_tagging_loss=0.009653, over 3040384.96 frames. ], batch size: 55, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:36:18,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1321846.6666666667, ans=10.0 2023-11-21 02:36:19,753 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.26 vs. limit=15.0 2023-11-21 02:36:26,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1321913.3333333333, ans=0.1 2023-11-21 02:36:39,028 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.990e+01 8.061e+01 8.738e+01 9.398e+01 2.090e+02, threshold=1.748e+02, percent-clipped=1.0 2023-11-21 02:36:41,560 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198300 2023-11-21 02:37:05,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1322113.3333333333, ans=0.0 2023-11-21 02:37:06,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1322113.3333333333, ans=0.2 2023-11-21 02:37:15,337 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.95 vs. limit=22.5 2023-11-21 02:37:17,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1322180.0, ans=0.125 2023-11-21 02:37:18,286 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 5950, loss[loss=0.07126, simple_loss=0.08601, pruned_loss=0.01677, audio_tagging_loss=0.01148, over 15024.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09933, pruned_loss=0.01784, audio_tagging_loss=0.0096, over 3044063.51 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:37:33,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=1322246.6666666667, ans=0.2 2023-11-21 02:37:45,620 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198350 2023-11-21 02:37:54,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1322313.3333333333, ans=0.125 2023-11-21 02:37:57,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.72 vs. limit=6.0 2023-11-21 02:37:58,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.64 vs. limit=15.0 2023-11-21 02:37:58,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.44 vs. limit=15.0 2023-11-21 02:38:04,906 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.30 vs. limit=15.0 2023-11-21 02:38:13,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1322446.6666666667, ans=0.0 2023-11-21 02:38:17,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1322446.6666666667, ans=0.0 2023-11-21 02:38:22,706 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6000, loss[loss=0.07084, simple_loss=0.09727, pruned_loss=0.01521, audio_tagging_loss=0.007, over 15327.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09807, pruned_loss=0.0175, audio_tagging_loss=0.009628, over 3042878.88 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:38:22,707 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 02:38:42,976 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.5829, 2.5293, 3.4499, 2.6763], device='cuda:3') 2023-11-21 02:38:58,470 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7566, 5.7960, 5.8557, 5.8468], device='cuda:3') 2023-11-21 02:39:04,188 INFO [train_asr.py:1253] (3/4) Epoch 17, validation: loss=0.06056, simple_loss=0.05273, pruned_loss=0.005281, audio_tagging_loss=0.02892, over 4681554.00 frames. 2023-11-21 02:39:04,189 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 02:39:04,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.01 vs. limit=12.0 2023-11-21 02:39:28,373 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.213e+01 8.088e+01 8.691e+01 9.569e+01 1.376e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-21 02:39:31,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198400 2023-11-21 02:39:50,590 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 02:40:08,235 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6050, loss[loss=0.07959, simple_loss=0.1001, pruned_loss=0.02069, audio_tagging_loss=0.008865, over 15334.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.098, pruned_loss=0.0177, audio_tagging_loss=0.009561, over 3045947.79 frames. ], batch size: 60, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:40:10,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1322846.6666666667, ans=0.125 2023-11-21 02:40:35,356 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198450 2023-11-21 02:41:06,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1323113.3333333333, ans=0.125 2023-11-21 02:41:12,328 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6100, loss[loss=0.07214, simple_loss=0.1005, pruned_loss=0.0134, audio_tagging_loss=0.00849, over 15797.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09834, pruned_loss=0.01769, audio_tagging_loss=0.009644, over 3039930.37 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:41:15,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1323180.0, ans=0.0 2023-11-21 02:41:26,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1323246.6666666667, ans=0.125 2023-11-21 02:41:32,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1323246.6666666667, ans=0.125 2023-11-21 02:41:36,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1323246.6666666667, ans=0.0 2023-11-21 02:41:36,166 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.78 vs. limit=10.0 2023-11-21 02:41:36,816 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.186e+01 8.873e+01 9.701e+01 1.239e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-21 02:41:39,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198500 2023-11-21 02:42:01,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.30 vs. limit=15.0 2023-11-21 02:42:10,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1323446.6666666667, ans=0.125 2023-11-21 02:42:16,016 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6150, loss[loss=0.08931, simple_loss=0.1151, pruned_loss=0.02228, audio_tagging_loss=0.009467, over 15509.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09784, pruned_loss=0.01751, audio_tagging_loss=0.009698, over 3047545.45 frames. ], batch size: 60, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:42:28,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1323580.0, ans=0.0 2023-11-21 02:42:32,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1323580.0, ans=0.0 2023-11-21 02:42:35,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1323580.0, ans=0.125 2023-11-21 02:42:43,478 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198550 2023-11-21 02:42:52,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1323646.6666666667, ans=0.125 2023-11-21 02:43:01,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1323713.3333333333, ans=0.125 2023-11-21 02:43:12,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.43 vs. limit=15.0 2023-11-21 02:43:17,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1323780.0, ans=0.125 2023-11-21 02:43:19,580 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6200, loss[loss=0.06207, simple_loss=0.0738, pruned_loss=0.01324, audio_tagging_loss=0.01193, over 14633.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.09746, pruned_loss=0.01736, audio_tagging_loss=0.009706, over 3053443.81 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:43:29,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1323846.6666666667, ans=0.0 2023-11-21 02:43:45,305 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.093e+01 8.158e+01 8.699e+01 9.470e+01 1.213e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 02:43:47,217 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198600 2023-11-21 02:43:47,333 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 02:43:52,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1323980.0, ans=0.125 2023-11-21 02:44:00,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1324046.6666666667, ans=0.1 2023-11-21 02:44:03,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1324046.6666666667, ans=0.2 2023-11-21 02:44:24,712 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6250, loss[loss=0.05448, simple_loss=0.0715, pruned_loss=0.009, audio_tagging_loss=0.009729, over 15121.00 frames. ], tot_loss[loss=0.07598, simple_loss=0.09759, pruned_loss=0.01743, audio_tagging_loss=0.009754, over 3049763.78 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:44:39,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.03 vs. limit=15.0 2023-11-21 02:44:50,992 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198650 2023-11-21 02:44:53,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1324313.3333333333, ans=0.0 2023-11-21 02:45:27,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1324513.3333333333, ans=0.1 2023-11-21 02:45:28,201 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6300, loss[loss=0.09041, simple_loss=0.1162, pruned_loss=0.02476, audio_tagging_loss=0.007559, over 14761.00 frames. ], tot_loss[loss=0.07661, simple_loss=0.09845, pruned_loss=0.0176, audio_tagging_loss=0.009791, over 3047197.21 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:45:28,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1324513.3333333333, ans=0.125 2023-11-21 02:45:30,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1324513.3333333333, ans=0.125 2023-11-21 02:45:42,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1324580.0, ans=0.125 2023-11-21 02:45:52,942 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.42 vs. limit=22.5 2023-11-21 02:45:53,589 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.806e+01 8.207e+01 8.844e+01 9.523e+01 1.211e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 02:45:56,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198700 2023-11-21 02:46:06,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.58 vs. limit=15.0 2023-11-21 02:46:08,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=1324713.3333333333, ans=0.025 2023-11-21 02:46:11,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1324713.3333333333, ans=0.0 2023-11-21 02:46:12,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1324713.3333333333, ans=0.09899494936611666 2023-11-21 02:46:15,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1324713.3333333333, ans=0.0 2023-11-21 02:46:15,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1324713.3333333333, ans=0.0 2023-11-21 02:46:31,997 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6350, loss[loss=0.08647, simple_loss=0.1219, pruned_loss=0.01787, audio_tagging_loss=0.00767, over 15155.00 frames. ], tot_loss[loss=0.07691, simple_loss=0.09883, pruned_loss=0.01761, audio_tagging_loss=0.009882, over 3047979.86 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:46:49,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1324913.3333333333, ans=0.125 2023-11-21 02:46:54,392 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.19 vs. limit=8.0 2023-11-21 02:46:59,711 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198750 2023-11-21 02:47:13,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1325046.6666666667, ans=0.125 2023-11-21 02:47:17,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1325046.6666666667, ans=0.0 2023-11-21 02:47:22,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1325113.3333333333, ans=0.125 2023-11-21 02:47:26,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1325113.3333333333, ans=0.1 2023-11-21 02:47:33,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1325113.3333333333, ans=0.125 2023-11-21 02:47:37,356 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6400, loss[loss=0.1006, simple_loss=0.1324, pruned_loss=0.02694, audio_tagging_loss=0.007457, over 15008.00 frames. ], tot_loss[loss=0.077, simple_loss=0.09876, pruned_loss=0.01764, audio_tagging_loss=0.009981, over 3044377.53 frames. ], batch size: 55, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:47:53,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1325246.6666666667, ans=0.1 2023-11-21 02:48:02,549 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 8.215e+01 8.788e+01 9.513e+01 1.963e+02, threshold=1.758e+02, percent-clipped=1.0 2023-11-21 02:48:03,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198800 2023-11-21 02:48:05,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1325313.3333333333, ans=0.09899494936611666 2023-11-21 02:48:14,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1325380.0, ans=0.09899494936611666 2023-11-21 02:48:19,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1325380.0, ans=0.125 2023-11-21 02:48:41,908 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6450, loss[loss=0.08133, simple_loss=0.0965, pruned_loss=0.02174, audio_tagging_loss=0.01134, over 14693.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.0985, pruned_loss=0.0177, audio_tagging_loss=0.01004, over 3043117.98 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:48:44,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1325513.3333333333, ans=0.125 2023-11-21 02:49:07,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198850 2023-11-21 02:49:16,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.23 vs. limit=15.0 2023-11-21 02:49:25,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1325713.3333333333, ans=0.015 2023-11-21 02:49:31,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1325780.0, ans=0.1 2023-11-21 02:49:31,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1325780.0, ans=0.125 2023-11-21 02:49:34,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1325780.0, ans=0.0 2023-11-21 02:49:38,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1325780.0, ans=0.125 2023-11-21 02:49:44,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1325846.6666666667, ans=0.125 2023-11-21 02:49:45,160 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6500, loss[loss=0.09131, simple_loss=0.1193, pruned_loss=0.02297, audio_tagging_loss=0.008661, over 15177.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09813, pruned_loss=0.01743, audio_tagging_loss=0.01001, over 3036905.81 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:49:46,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1325846.6666666667, ans=0.125 2023-11-21 02:50:10,556 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.009e+01 8.684e+01 9.310e+01 1.221e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 02:50:10,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1325980.0, ans=0.05 2023-11-21 02:50:11,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198900 2023-11-21 02:50:48,731 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6550, loss[loss=0.09337, simple_loss=0.1274, pruned_loss=0.02224, audio_tagging_loss=0.007446, over 15060.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.09927, pruned_loss=0.01777, audio_tagging_loss=0.009792, over 3043716.56 frames. ], batch size: 54, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:50:53,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1326180.0, ans=0.0 2023-11-21 02:50:57,273 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0 2023-11-21 02:51:02,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1326246.6666666667, ans=0.025 2023-11-21 02:51:04,922 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.25 vs. limit=10.0 2023-11-21 02:51:14,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1326313.3333333333, ans=0.025 2023-11-21 02:51:15,791 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 198950 2023-11-21 02:51:19,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1326313.3333333333, ans=0.125 2023-11-21 02:51:27,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1326380.0, ans=22.5 2023-11-21 02:51:35,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1326380.0, ans=0.125 2023-11-21 02:51:49,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1326446.6666666667, ans=0.125 2023-11-21 02:51:52,142 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.63 vs. limit=15.0 2023-11-21 02:51:52,727 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6600, loss[loss=0.06738, simple_loss=0.08422, pruned_loss=0.01497, audio_tagging_loss=0.01031, over 14880.00 frames. ], tot_loss[loss=0.07682, simple_loss=0.09887, pruned_loss=0.01773, audio_tagging_loss=0.009658, over 3042317.44 frames. ], batch size: 59, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:52:01,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1326513.3333333333, ans=0.125 2023-11-21 02:52:03,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1326513.3333333333, ans=0.0 2023-11-21 02:52:17,919 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.252e+01 8.813e+01 9.521e+01 1.214e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-21 02:52:19,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199000 2023-11-21 02:52:30,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.28 vs. limit=22.5 2023-11-21 02:52:57,307 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6650, loss[loss=0.07287, simple_loss=0.09795, pruned_loss=0.01666, audio_tagging_loss=0.007238, over 15398.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09823, pruned_loss=0.01743, audio_tagging_loss=0.009564, over 3036623.83 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0 2023-11-21 02:53:02,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1326846.6666666667, ans=0.125 2023-11-21 02:53:02,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1326846.6666666667, ans=0.125 2023-11-21 02:53:03,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1326846.6666666667, ans=0.2 2023-11-21 02:53:05,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1326846.6666666667, ans=0.125 2023-11-21 02:53:14,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1326913.3333333333, ans=0.125 2023-11-21 02:53:23,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.48 vs. limit=12.0 2023-11-21 02:53:25,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199050 2023-11-21 02:53:31,854 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.61 vs. limit=15.0 2023-11-21 02:53:36,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1327046.6666666667, ans=0.1 2023-11-21 02:53:44,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1327046.6666666667, ans=0.07 2023-11-21 02:54:01,150 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6700, loss[loss=0.06876, simple_loss=0.08729, pruned_loss=0.01648, audio_tagging_loss=0.008635, over 14634.00 frames. ], tot_loss[loss=0.07556, simple_loss=0.0971, pruned_loss=0.01745, audio_tagging_loss=0.009563, over 3039569.65 frames. ], batch size: 53, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:54:07,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1327180.0, ans=0.2 2023-11-21 02:54:15,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1327246.6666666667, ans=0.0 2023-11-21 02:54:15,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1327246.6666666667, ans=0.125 2023-11-21 02:54:22,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1327246.6666666667, ans=0.125 2023-11-21 02:54:24,089 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.38 vs. limit=15.0 2023-11-21 02:54:27,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.45 vs. limit=10.0 2023-11-21 02:54:29,391 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 7.825e+01 8.466e+01 9.524e+01 1.131e+02, threshold=1.693e+02, percent-clipped=0.0 2023-11-21 02:54:29,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199100 2023-11-21 02:54:33,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1327313.3333333333, ans=0.125 2023-11-21 02:54:38,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1327313.3333333333, ans=0.1 2023-11-21 02:54:45,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1327380.0, ans=0.125 2023-11-21 02:54:46,064 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.39 vs. limit=15.0 2023-11-21 02:55:06,857 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6750, loss[loss=0.07389, simple_loss=0.09374, pruned_loss=0.01878, audio_tagging_loss=0.008234, over 14586.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09814, pruned_loss=0.01766, audio_tagging_loss=0.009572, over 3040810.45 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:55:11,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1327513.3333333333, ans=0.125 2023-11-21 02:55:12,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1327513.3333333333, ans=0.1 2023-11-21 02:55:18,962 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.20 vs. limit=6.0 2023-11-21 02:55:32,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199150 2023-11-21 02:55:54,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1327713.3333333333, ans=0.125 2023-11-21 02:56:05,209 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.74 vs. limit=15.0 2023-11-21 02:56:10,671 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6800, loss[loss=0.07353, simple_loss=0.09243, pruned_loss=0.01724, audio_tagging_loss=0.01007, over 15062.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09664, pruned_loss=0.01746, audio_tagging_loss=0.009748, over 3032637.21 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 16.0 2023-11-21 02:56:18,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1327846.6666666667, ans=0.125 2023-11-21 02:56:18,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1327846.6666666667, ans=0.125 2023-11-21 02:56:24,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1327913.3333333333, ans=0.0 2023-11-21 02:56:24,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1327913.3333333333, ans=0.0 2023-11-21 02:56:24,925 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.80 vs. limit=15.0 2023-11-21 02:56:31,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1327913.3333333333, ans=0.0 2023-11-21 02:56:37,887 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199200 2023-11-21 02:56:38,908 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.071e+01 8.833e+01 9.608e+01 1.176e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 02:57:14,627 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6850, loss[loss=0.06783, simple_loss=0.08534, pruned_loss=0.01615, audio_tagging_loss=0.009014, over 15324.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09619, pruned_loss=0.01726, audio_tagging_loss=0.009687, over 3024863.67 frames. ], batch size: 59, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 02:57:19,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1328180.0, ans=0.125 2023-11-21 02:57:22,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.19 vs. limit=15.0 2023-11-21 02:57:25,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1328180.0, ans=0.125 2023-11-21 02:57:28,545 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0 2023-11-21 02:57:42,694 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199250 2023-11-21 02:57:50,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.14 vs. limit=10.0 2023-11-21 02:57:52,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1328380.0, ans=0.2 2023-11-21 02:58:03,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1328380.0, ans=0.125 2023-11-21 02:58:13,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1328446.6666666667, ans=0.0 2023-11-21 02:58:19,385 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6900, loss[loss=0.1041, simple_loss=0.1436, pruned_loss=0.02571, audio_tagging_loss=0.006604, over 15741.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.09717, pruned_loss=0.01756, audio_tagging_loss=0.009648, over 3035866.72 frames. ], batch size: 54, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 02:58:33,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1328580.0, ans=0.0 2023-11-21 02:58:33,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1328580.0, ans=0.2 2023-11-21 02:58:46,562 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199300 2023-11-21 02:58:47,626 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.846e+01 8.588e+01 9.245e+01 1.215e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-21 02:58:49,602 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.22 vs. limit=15.0 2023-11-21 02:59:08,586 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 02:59:11,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.75 vs. limit=6.0 2023-11-21 02:59:19,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1328780.0, ans=0.2 2023-11-21 02:59:23,856 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 6950, loss[loss=0.08939, simple_loss=0.1214, pruned_loss=0.01956, audio_tagging_loss=0.009137, over 16036.00 frames. ], tot_loss[loss=0.07624, simple_loss=0.09777, pruned_loss=0.01776, audio_tagging_loss=0.009598, over 3043532.49 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 02:59:28,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.69 vs. limit=22.5 2023-11-21 02:59:51,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199350 2023-11-21 02:59:51,314 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=5.253e-03 2023-11-21 02:59:54,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.91 vs. limit=15.0 2023-11-21 02:59:55,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.59 vs. limit=8.0 2023-11-21 03:00:10,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1329046.6666666667, ans=0.1 2023-11-21 03:00:27,831 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7000, loss[loss=0.07352, simple_loss=0.1018, pruned_loss=0.01481, audio_tagging_loss=0.007793, over 15615.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.09843, pruned_loss=0.01773, audio_tagging_loss=0.009592, over 3045785.17 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:00:33,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1329180.0, ans=0.125 2023-11-21 03:00:55,574 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199400 2023-11-21 03:00:57,286 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.260e+01 8.901e+01 9.434e+01 1.193e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-21 03:01:00,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1329313.3333333333, ans=0.0 2023-11-21 03:01:08,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1329380.0, ans=0.0 2023-11-21 03:01:11,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1329380.0, ans=0.125 2023-11-21 03:01:32,980 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7050, loss[loss=0.07449, simple_loss=0.0882, pruned_loss=0.02045, audio_tagging_loss=0.00994, over 14998.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09786, pruned_loss=0.01754, audio_tagging_loss=0.009779, over 3047096.04 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:01:34,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1329513.3333333333, ans=0.125 2023-11-21 03:01:37,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1329513.3333333333, ans=0.125 2023-11-21 03:01:38,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1329513.3333333333, ans=0.125 2023-11-21 03:01:46,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1329580.0, ans=0.125 2023-11-21 03:01:46,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1329580.0, ans=0.125 2023-11-21 03:01:52,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1329580.0, ans=0.125 2023-11-21 03:01:55,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1329580.0, ans=0.125 2023-11-21 03:01:56,988 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.84 vs. limit=15.0 2023-11-21 03:02:00,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199450 2023-11-21 03:02:01,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1329646.6666666667, ans=0.125 2023-11-21 03:02:04,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.02 vs. limit=6.0 2023-11-21 03:02:09,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1329646.6666666667, ans=0.2 2023-11-21 03:02:11,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1329713.3333333333, ans=0.1 2023-11-21 03:02:14,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1329713.3333333333, ans=0.125 2023-11-21 03:02:38,001 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7100, loss[loss=0.07601, simple_loss=0.1011, pruned_loss=0.01555, audio_tagging_loss=0.009897, over 14830.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09729, pruned_loss=0.01734, audio_tagging_loss=0.009828, over 3049237.15 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:02:40,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.40 vs. limit=15.0 2023-11-21 03:02:42,438 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.77 vs. limit=15.0 2023-11-21 03:02:56,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1329913.3333333333, ans=0.0 2023-11-21 03:03:05,034 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199500 2023-11-21 03:03:06,066 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 7.897e+01 8.673e+01 9.642e+01 1.180e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 03:03:42,059 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7150, loss[loss=0.04887, simple_loss=0.04915, pruned_loss=0.008786, audio_tagging_loss=0.01551, over 14707.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.09761, pruned_loss=0.01752, audio_tagging_loss=0.00996, over 3049707.29 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:03:42,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1330180.0, ans=0.0 2023-11-21 03:03:56,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1330246.6666666667, ans=0.5 2023-11-21 03:04:09,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199550 2023-11-21 03:04:15,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.59 vs. limit=15.0 2023-11-21 03:04:17,492 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.43 vs. limit=15.0 2023-11-21 03:04:46,671 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7200, loss[loss=0.05692, simple_loss=0.0703, pruned_loss=0.01023, audio_tagging_loss=0.01154, over 16149.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09822, pruned_loss=0.01759, audio_tagging_loss=0.009945, over 3051241.39 frames. ], batch size: 62, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:04:49,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1330513.3333333333, ans=0.125 2023-11-21 03:04:51,198 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.76 vs. limit=10.0 2023-11-21 03:04:56,990 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.41 vs. limit=15.0 2023-11-21 03:05:13,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199600 2023-11-21 03:05:16,287 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 7.951e+01 8.767e+01 9.502e+01 1.229e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-21 03:05:34,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1330713.3333333333, ans=0.95 2023-11-21 03:05:37,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1330780.0, ans=0.05 2023-11-21 03:05:39,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1330780.0, ans=0.0 2023-11-21 03:05:50,868 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7250, loss[loss=0.08303, simple_loss=0.113, pruned_loss=0.01901, audio_tagging_loss=0.00751, over 16987.00 frames. ], tot_loss[loss=0.07631, simple_loss=0.0976, pruned_loss=0.01756, audio_tagging_loss=0.009945, over 3053907.65 frames. ], batch size: 66, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:05:57,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1330846.6666666667, ans=0.2 2023-11-21 03:05:59,969 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.85 vs. limit=22.5 2023-11-21 03:06:03,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1330913.3333333333, ans=0.1 2023-11-21 03:06:11,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1330913.3333333333, ans=0.0 2023-11-21 03:06:12,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1330913.3333333333, ans=0.2 2023-11-21 03:06:17,357 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199650 2023-11-21 03:06:24,062 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 03:06:26,855 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0 2023-11-21 03:06:40,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1331113.3333333333, ans=0.2 2023-11-21 03:06:43,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1331113.3333333333, ans=0.0 2023-11-21 03:06:54,326 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7300, loss[loss=0.086, simple_loss=0.1037, pruned_loss=0.02495, audio_tagging_loss=0.009221, over 16195.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09735, pruned_loss=0.01758, audio_tagging_loss=0.009873, over 3053753.01 frames. ], batch size: 62, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:07:12,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1331246.6666666667, ans=0.0 2023-11-21 03:07:18,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1331313.3333333333, ans=0.0 2023-11-21 03:07:20,999 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199700 2023-11-21 03:07:23,858 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.086e+01 8.011e+01 8.733e+01 9.607e+01 1.282e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 03:07:24,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1331313.3333333333, ans=0.2 2023-11-21 03:07:26,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1331313.3333333333, ans=0.125 2023-11-21 03:07:41,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1331380.0, ans=0.125 2023-11-21 03:07:50,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1331446.6666666667, ans=0.2 2023-11-21 03:07:52,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1331446.6666666667, ans=0.0 2023-11-21 03:07:52,773 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.048e-01 2023-11-21 03:07:58,470 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7350, loss[loss=0.0585, simple_loss=0.07092, pruned_loss=0.01158, audio_tagging_loss=0.01145, over 14516.00 frames. ], tot_loss[loss=0.07589, simple_loss=0.09707, pruned_loss=0.01764, audio_tagging_loss=0.009713, over 3053274.57 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:08:05,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1331513.3333333333, ans=0.0 2023-11-21 03:08:17,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1331580.0, ans=0.125 2023-11-21 03:08:18,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1331580.0, ans=0.0 2023-11-21 03:08:24,781 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199750 2023-11-21 03:08:38,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.66 vs. limit=6.0 2023-11-21 03:08:48,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1331780.0, ans=0.1 2023-11-21 03:09:02,762 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7400, loss[loss=0.08066, simple_loss=0.1088, pruned_loss=0.01575, audio_tagging_loss=0.0105, over 15589.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09689, pruned_loss=0.01758, audio_tagging_loss=0.009703, over 3047779.43 frames. ], batch size: 57, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:09:02,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1331846.6666666667, ans=10.0 2023-11-21 03:09:06,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1331846.6666666667, ans=0.125 2023-11-21 03:09:11,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1331846.6666666667, ans=0.95 2023-11-21 03:09:14,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1331913.3333333333, ans=0.0 2023-11-21 03:09:14,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.52 vs. limit=10.0 2023-11-21 03:09:29,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199800 2023-11-21 03:09:32,477 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.394e+01 8.010e+01 8.731e+01 9.621e+01 1.537e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 03:09:49,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1332046.6666666667, ans=0.1 2023-11-21 03:09:55,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1332113.3333333333, ans=0.5 2023-11-21 03:10:00,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1332113.3333333333, ans=0.125 2023-11-21 03:10:06,589 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7450, loss[loss=0.06959, simple_loss=0.08627, pruned_loss=0.01577, audio_tagging_loss=0.01068, over 16289.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09708, pruned_loss=0.01752, audio_tagging_loss=0.009593, over 3045863.08 frames. ], batch size: 62, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:10:30,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.19 vs. limit=22.5 2023-11-21 03:10:33,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199850 2023-11-21 03:10:44,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1332380.0, ans=0.125 2023-11-21 03:11:10,950 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7500, loss[loss=0.06726, simple_loss=0.09358, pruned_loss=0.01159, audio_tagging_loss=0.008883, over 15591.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09741, pruned_loss=0.01762, audio_tagging_loss=0.009603, over 3043717.36 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:11:37,143 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199900 2023-11-21 03:11:39,369 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.979e+01 8.350e+01 8.922e+01 9.540e+01 1.272e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-21 03:11:51,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.05 vs. limit=12.0 2023-11-21 03:12:14,027 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7550, loss[loss=0.05327, simple_loss=0.06938, pruned_loss=0.01134, audio_tagging_loss=0.007239, over 16028.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09746, pruned_loss=0.01771, audio_tagging_loss=0.009591, over 3040539.82 frames. ], batch size: 61, lr: 4.03e-03, grad_scale: 16.0 2023-11-21 03:12:14,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.62 vs. limit=15.0 2023-11-21 03:12:39,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1332980.0, ans=0.2 2023-11-21 03:12:40,268 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 199950 2023-11-21 03:12:40,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1332980.0, ans=0.2 2023-11-21 03:12:54,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.01 vs. limit=12.0 2023-11-21 03:13:05,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1333113.3333333333, ans=0.2 2023-11-21 03:13:17,621 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7600, loss[loss=0.0696, simple_loss=0.08915, pruned_loss=0.0175, audio_tagging_loss=0.007522, over 16416.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09764, pruned_loss=0.01797, audio_tagging_loss=0.009465, over 3040523.92 frames. ], batch size: 62, lr: 4.03e-03, grad_scale: 32.0 2023-11-21 03:13:17,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1333180.0, ans=0.0 2023-11-21 03:13:33,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1333246.6666666667, ans=0.125 2023-11-21 03:13:35,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1333246.6666666667, ans=0.125 2023-11-21 03:13:36,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1333246.6666666667, ans=0.125 2023-11-21 03:13:45,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200000 2023-11-21 03:13:50,913 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.857e+01 8.060e+01 8.807e+01 9.387e+01 1.277e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-21 03:13:57,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1333313.3333333333, ans=0.125 2023-11-21 03:14:05,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1333380.0, ans=0.125 2023-11-21 03:14:06,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1333380.0, ans=0.0 2023-11-21 03:14:13,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1333446.6666666667, ans=0.0 2023-11-21 03:14:25,308 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7650, loss[loss=0.05695, simple_loss=0.08031, pruned_loss=0.007342, audio_tagging_loss=0.009456, over 14554.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.0983, pruned_loss=0.01787, audio_tagging_loss=0.009491, over 3036952.97 frames. ], batch size: 55, lr: 4.03e-03, grad_scale: 32.0 2023-11-21 03:14:34,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1333513.3333333333, ans=0.0 2023-11-21 03:14:49,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1333580.0, ans=0.0 2023-11-21 03:14:50,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1333646.6666666667, ans=0.125 2023-11-21 03:14:52,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200050 2023-11-21 03:14:55,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1333646.6666666667, ans=0.1 2023-11-21 03:15:29,950 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7700, loss[loss=0.09803, simple_loss=0.1243, pruned_loss=0.02585, audio_tagging_loss=0.01003, over 14842.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.09787, pruned_loss=0.01772, audio_tagging_loss=0.009573, over 3040909.67 frames. ], batch size: 55, lr: 4.03e-03, grad_scale: 32.0 2023-11-21 03:15:31,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1333846.6666666667, ans=0.125 2023-11-21 03:15:35,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1333846.6666666667, ans=0.125 2023-11-21 03:15:54,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1333980.0, ans=0.0 2023-11-21 03:15:55,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200100 2023-11-21 03:15:58,171 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.520e+01 8.098e+01 8.698e+01 9.708e+01 1.361e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 03:16:05,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1333980.0, ans=0.0 2023-11-21 03:16:13,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1334046.6666666667, ans=0.125 2023-11-21 03:16:13,671 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.02 vs. limit=6.0 2023-11-21 03:16:28,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1334113.3333333333, ans=0.0 2023-11-21 03:16:33,109 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7750, loss[loss=0.08041, simple_loss=0.1033, pruned_loss=0.01703, audio_tagging_loss=0.01175, over 15515.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.0979, pruned_loss=0.01765, audio_tagging_loss=0.00961, over 3043207.22 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 32.0 2023-11-21 03:16:33,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1334180.0, ans=0.125 2023-11-21 03:16:35,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1334180.0, ans=0.04949747468305833 2023-11-21 03:16:37,535 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=12.0 2023-11-21 03:16:46,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1334246.6666666667, ans=0.125 2023-11-21 03:17:00,604 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200150 2023-11-21 03:17:04,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1334313.3333333333, ans=0.0 2023-11-21 03:17:07,427 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0 2023-11-21 03:17:14,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1334380.0, ans=10.0 2023-11-21 03:17:17,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1334380.0, ans=0.125 2023-11-21 03:17:20,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1334380.0, ans=0.125 2023-11-21 03:17:29,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1334446.6666666667, ans=0.0 2023-11-21 03:17:36,755 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7800, loss[loss=0.1136, simple_loss=0.1484, pruned_loss=0.03301, audio_tagging_loss=0.006396, over 15585.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09815, pruned_loss=0.01777, audio_tagging_loss=0.009572, over 3045190.91 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 32.0 2023-11-21 03:17:38,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1334513.3333333333, ans=0.09899494936611666 2023-11-21 03:18:04,895 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200200 2023-11-21 03:18:07,465 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.821e+01 7.982e+01 8.620e+01 9.218e+01 1.876e+02, threshold=1.724e+02, percent-clipped=1.0 2023-11-21 03:18:36,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.31 vs. limit=15.0 2023-11-21 03:18:40,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1334780.0, ans=0.2 2023-11-21 03:18:42,227 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7850, loss[loss=0.08607, simple_loss=0.1115, pruned_loss=0.0196, audio_tagging_loss=0.01071, over 14058.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09902, pruned_loss=0.01804, audio_tagging_loss=0.009655, over 3042292.98 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:19:08,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200250 2023-11-21 03:19:37,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1335113.3333333333, ans=0.125 2023-11-21 03:19:42,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1335113.3333333333, ans=0.125 2023-11-21 03:19:43,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1335113.3333333333, ans=0.125 2023-11-21 03:19:46,652 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7900, loss[loss=0.07939, simple_loss=0.1075, pruned_loss=0.01297, audio_tagging_loss=0.01266, over 15429.00 frames. ], tot_loss[loss=0.07691, simple_loss=0.0988, pruned_loss=0.01774, audio_tagging_loss=0.009768, over 3044872.00 frames. ], batch size: 56, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:19:49,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1335180.0, ans=0.2 2023-11-21 03:19:56,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1335180.0, ans=0.04949747468305833 2023-11-21 03:20:01,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1335246.6666666667, ans=0.2 2023-11-21 03:20:04,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.26 vs. limit=15.0 2023-11-21 03:20:13,781 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200300 2023-11-21 03:20:16,600 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.124e+01 8.233e+01 9.036e+01 9.766e+01 1.372e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-21 03:20:31,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1335380.0, ans=0.95 2023-11-21 03:20:39,002 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 03:20:49,706 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 7950, loss[loss=0.0698, simple_loss=0.08591, pruned_loss=0.01459, audio_tagging_loss=0.01226, over 14099.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09788, pruned_loss=0.01771, audio_tagging_loss=0.01004, over 3040369.67 frames. ], batch size: 56, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:21:02,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1335580.0, ans=0.1 2023-11-21 03:21:05,448 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 03:21:17,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200350 2023-11-21 03:21:19,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1335646.6666666667, ans=0.07 2023-11-21 03:21:42,950 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0 2023-11-21 03:21:50,453 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.02 vs. limit=22.5 2023-11-21 03:21:54,638 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8000, loss[loss=0.08788, simple_loss=0.1093, pruned_loss=0.02232, audio_tagging_loss=0.01093, over 15825.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.0957, pruned_loss=0.01725, audio_tagging_loss=0.01015, over 3029941.44 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:22:21,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200400 2023-11-21 03:22:21,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1335980.0, ans=0.125 2023-11-21 03:22:24,185 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.612e+01 7.996e+01 8.659e+01 9.313e+01 1.449e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 03:22:31,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1336046.6666666667, ans=0.125 2023-11-21 03:22:34,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1336046.6666666667, ans=0.0 2023-11-21 03:22:35,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1336046.6666666667, ans=0.2 2023-11-21 03:22:36,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1336046.6666666667, ans=0.125 2023-11-21 03:22:43,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1336046.6666666667, ans=0.1 2023-11-21 03:22:59,641 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8050, loss[loss=0.05841, simple_loss=0.06545, pruned_loss=0.009784, audio_tagging_loss=0.0159, over 15434.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09496, pruned_loss=0.01718, audio_tagging_loss=0.01022, over 3030402.03 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:23:26,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200450 2023-11-21 03:23:29,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=22.5 2023-11-21 03:23:41,097 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.47 vs. limit=22.5 2023-11-21 03:24:02,437 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8100, loss[loss=0.0763, simple_loss=0.1018, pruned_loss=0.02029, audio_tagging_loss=0.005115, over 14980.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09588, pruned_loss=0.01733, audio_tagging_loss=0.0101, over 3035290.99 frames. ], batch size: 54, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:24:06,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1336513.3333333333, ans=0.2 2023-11-21 03:24:29,731 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200500 2023-11-21 03:24:29,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1336646.6666666667, ans=0.05 2023-11-21 03:24:30,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1336646.6666666667, ans=0.0 2023-11-21 03:24:32,124 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.135e+01 8.794e+01 9.348e+01 1.385e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 03:24:33,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1336646.6666666667, ans=0.125 2023-11-21 03:24:50,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.70 vs. limit=15.0 2023-11-21 03:24:52,692 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.47 vs. limit=15.0 2023-11-21 03:24:58,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1336780.0, ans=0.125 2023-11-21 03:24:58,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.20 vs. limit=15.0 2023-11-21 03:25:02,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.32 vs. limit=15.0 2023-11-21 03:25:06,606 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8150, loss[loss=0.07814, simple_loss=0.1009, pruned_loss=0.02053, audio_tagging_loss=0.007142, over 16778.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09725, pruned_loss=0.01756, audio_tagging_loss=0.009843, over 3041099.48 frames. ], batch size: 63, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:25:28,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.59 vs. limit=15.0 2023-11-21 03:25:31,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1336980.0, ans=0.125 2023-11-21 03:25:33,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200550 2023-11-21 03:25:36,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1336980.0, ans=0.125 2023-11-21 03:25:37,409 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=2.649e-03 2023-11-21 03:25:47,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1337046.6666666667, ans=0.07 2023-11-21 03:26:01,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1337113.3333333333, ans=0.125 2023-11-21 03:26:11,142 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8200, loss[loss=0.07918, simple_loss=0.1036, pruned_loss=0.01754, audio_tagging_loss=0.009826, over 15421.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09727, pruned_loss=0.01734, audio_tagging_loss=0.009741, over 3045899.70 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:26:11,172 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 03:26:21,360 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.03 vs. limit=15.0 2023-11-21 03:26:29,184 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0 2023-11-21 03:26:38,111 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200600 2023-11-21 03:26:40,668 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.735e+01 8.383e+01 9.269e+01 1.042e+02 1.722e+02, threshold=1.854e+02, percent-clipped=0.0 2023-11-21 03:26:45,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1337313.3333333333, ans=0.125 2023-11-21 03:26:47,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.34 vs. limit=10.0 2023-11-21 03:27:00,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1337380.0, ans=0.2 2023-11-21 03:27:15,011 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8250, loss[loss=0.08275, simple_loss=0.1018, pruned_loss=0.02196, audio_tagging_loss=0.009874, over 16198.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09701, pruned_loss=0.01736, audio_tagging_loss=0.009856, over 3048857.91 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 16.0 2023-11-21 03:27:15,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1337513.3333333333, ans=0.95 2023-11-21 03:27:22,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1337513.3333333333, ans=0.1 2023-11-21 03:27:37,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1337580.0, ans=15.0 2023-11-21 03:27:41,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200650 2023-11-21 03:27:57,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0 2023-11-21 03:27:59,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=13.68 vs. limit=15.0 2023-11-21 03:28:09,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1337780.0, ans=0.95 2023-11-21 03:28:12,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1337780.0, ans=0.125 2023-11-21 03:28:13,448 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.56 vs. limit=15.0 2023-11-21 03:28:19,779 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8300, loss[loss=0.07284, simple_loss=0.09498, pruned_loss=0.01335, audio_tagging_loss=0.01201, over 15685.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.0972, pruned_loss=0.0173, audio_tagging_loss=0.009782, over 3050264.23 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 16.0 2023-11-21 03:28:25,213 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.48 vs. limit=15.0 2023-11-21 03:28:28,603 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=9.027e-02 2023-11-21 03:28:29,985 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.29 vs. limit=15.0 2023-11-21 03:28:37,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1337913.3333333333, ans=0.125 2023-11-21 03:28:43,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.75 vs. limit=22.5 2023-11-21 03:28:46,690 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200700 2023-11-21 03:28:50,270 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.519e+01 8.192e+01 8.783e+01 9.537e+01 1.149e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-21 03:29:23,792 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8350, loss[loss=0.06819, simple_loss=0.08606, pruned_loss=0.0134, audio_tagging_loss=0.01176, over 15156.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.0974, pruned_loss=0.01744, audio_tagging_loss=0.00966, over 3050465.09 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 16.0 2023-11-21 03:29:29,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.98 vs. limit=15.0 2023-11-21 03:29:41,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1338246.6666666667, ans=0.0 2023-11-21 03:29:50,146 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200750 2023-11-21 03:29:58,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1338313.3333333333, ans=0.2 2023-11-21 03:30:04,163 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.14 vs. limit=22.5 2023-11-21 03:30:07,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1338380.0, ans=0.125 2023-11-21 03:30:20,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0 2023-11-21 03:30:25,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1338446.6666666667, ans=0.0 2023-11-21 03:30:27,703 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8400, loss[loss=0.07426, simple_loss=0.0977, pruned_loss=0.01687, audio_tagging_loss=0.008542, over 15534.00 frames. ], tot_loss[loss=0.07528, simple_loss=0.09684, pruned_loss=0.01724, audio_tagging_loss=0.009622, over 3050999.75 frames. ], batch size: 57, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:30:47,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1338580.0, ans=0.0 2023-11-21 03:30:54,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200800 2023-11-21 03:30:55,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1338646.6666666667, ans=0.0 2023-11-21 03:30:57,991 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.656e+01 8.229e+01 8.784e+01 9.362e+01 1.197e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-21 03:31:05,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.73 vs. limit=10.0 2023-11-21 03:31:07,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.33 vs. limit=15.0 2023-11-21 03:31:11,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1338713.3333333333, ans=0.0 2023-11-21 03:31:31,822 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8450, loss[loss=0.08238, simple_loss=0.1017, pruned_loss=0.02178, audio_tagging_loss=0.009724, over 15651.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09724, pruned_loss=0.01727, audio_tagging_loss=0.009695, over 3053335.79 frames. ], batch size: 61, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:31:44,492 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.62 vs. limit=15.0 2023-11-21 03:31:54,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1338913.3333333333, ans=0.5 2023-11-21 03:31:55,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1338980.0, ans=0.125 2023-11-21 03:31:58,175 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200850 2023-11-21 03:32:10,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1339046.6666666667, ans=0.125 2023-11-21 03:32:10,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1339046.6666666667, ans=0.1 2023-11-21 03:32:26,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1339113.3333333333, ans=0.0 2023-11-21 03:32:35,647 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8500, loss[loss=0.09702, simple_loss=0.1302, pruned_loss=0.02308, audio_tagging_loss=0.008856, over 16040.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.0976, pruned_loss=0.01735, audio_tagging_loss=0.009633, over 3056493.29 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:32:43,099 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=15.0 2023-11-21 03:32:45,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1339180.0, ans=0.1 2023-11-21 03:32:51,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1339246.6666666667, ans=0.125 2023-11-21 03:33:02,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200900 2023-11-21 03:33:06,648 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 7.935e+01 8.661e+01 9.459e+01 1.270e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 03:33:18,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1339380.0, ans=0.0 2023-11-21 03:33:39,581 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8550, loss[loss=0.08176, simple_loss=0.1042, pruned_loss=0.01737, audio_tagging_loss=0.01228, over 15469.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.0979, pruned_loss=0.01746, audio_tagging_loss=0.009649, over 3055221.24 frames. ], batch size: 57, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:33:46,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1339513.3333333333, ans=0.0 2023-11-21 03:33:51,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1339580.0, ans=0.125 2023-11-21 03:34:05,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1339646.6666666667, ans=0.125 2023-11-21 03:34:06,575 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 200950 2023-11-21 03:34:08,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.08 vs. limit=15.0 2023-11-21 03:34:15,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1339646.6666666667, ans=0.0 2023-11-21 03:34:31,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1339780.0, ans=0.125 2023-11-21 03:34:43,518 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8600, loss[loss=0.05535, simple_loss=0.07219, pruned_loss=0.01194, audio_tagging_loss=0.007319, over 13941.00 frames. ], tot_loss[loss=0.07634, simple_loss=0.09808, pruned_loss=0.01758, audio_tagging_loss=0.009719, over 3055001.27 frames. ], batch size: 54, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:34:50,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1339846.6666666667, ans=0.05 2023-11-21 03:35:00,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1339913.3333333333, ans=0.125 2023-11-21 03:35:04,848 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.70 vs. limit=22.5 2023-11-21 03:35:10,350 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201000 2023-11-21 03:35:14,245 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.482e+01 7.981e+01 8.707e+01 9.381e+01 1.149e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-21 03:35:18,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1339980.0, ans=0.125 2023-11-21 03:35:24,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1340046.6666666667, ans=0.0 2023-11-21 03:35:35,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1340113.3333333333, ans=0.0 2023-11-21 03:35:35,753 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.21 vs. limit=15.0 2023-11-21 03:35:47,422 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8650, loss[loss=0.06209, simple_loss=0.07819, pruned_loss=0.009294, audio_tagging_loss=0.01369, over 15141.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09798, pruned_loss=0.0176, audio_tagging_loss=0.009833, over 3054864.19 frames. ], batch size: 56, lr: 4.02e-03, grad_scale: 16.0 2023-11-21 03:36:06,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1340246.6666666667, ans=0.0 2023-11-21 03:36:14,569 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201050 2023-11-21 03:36:17,492 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.74 vs. limit=15.0 2023-11-21 03:36:27,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1340380.0, ans=0.05 2023-11-21 03:36:32,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1340380.0, ans=0.125 2023-11-21 03:36:50,941 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8700, loss[loss=0.0985, simple_loss=0.1278, pruned_loss=0.02453, audio_tagging_loss=0.01007, over 16710.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09737, pruned_loss=0.01753, audio_tagging_loss=0.009906, over 3051096.05 frames. ], batch size: 63, lr: 4.02e-03, grad_scale: 16.0 2023-11-21 03:36:56,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1340513.3333333333, ans=0.125 2023-11-21 03:36:58,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1340513.3333333333, ans=0.0 2023-11-21 03:37:07,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1340580.0, ans=0.0 2023-11-21 03:37:17,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201100 2023-11-21 03:37:22,700 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.240e+01 8.992e+01 1.008e+02 1.229e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-21 03:37:30,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1340713.3333333333, ans=0.0 2023-11-21 03:37:54,263 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8750, loss[loss=0.06969, simple_loss=0.09458, pruned_loss=0.01326, audio_tagging_loss=0.00914, over 16440.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09809, pruned_loss=0.01779, audio_tagging_loss=0.01001, over 3048620.13 frames. ], batch size: 62, lr: 4.02e-03, grad_scale: 16.0 2023-11-21 03:38:04,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1340846.6666666667, ans=0.125 2023-11-21 03:38:07,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1340913.3333333333, ans=0.0 2023-11-21 03:38:08,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1340913.3333333333, ans=0.125 2023-11-21 03:38:14,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1340913.3333333333, ans=0.0 2023-11-21 03:38:21,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201150 2023-11-21 03:38:58,534 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8800, loss[loss=0.05907, simple_loss=0.07302, pruned_loss=0.01222, audio_tagging_loss=0.01035, over 15849.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09955, pruned_loss=0.01787, audio_tagging_loss=0.01, over 3054448.36 frames. ], batch size: 61, lr: 4.02e-03, grad_scale: 32.0 2023-11-21 03:39:09,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1341180.0, ans=0.0 2023-11-21 03:39:21,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1341246.6666666667, ans=0.125 2023-11-21 03:39:22,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1341313.3333333333, ans=0.125 2023-11-21 03:39:24,707 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201200 2023-11-21 03:39:30,376 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.246e+01 8.266e+01 8.872e+01 9.572e+01 1.229e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-21 03:39:46,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1341380.0, ans=0.05 2023-11-21 03:39:54,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=1341446.6666666667, ans=15.0 2023-11-21 03:39:57,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1341446.6666666667, ans=0.0 2023-11-21 03:40:02,359 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8850, loss[loss=0.05867, simple_loss=0.07608, pruned_loss=0.01198, audio_tagging_loss=0.00865, over 14411.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09878, pruned_loss=0.01767, audio_tagging_loss=0.009974, over 3056146.04 frames. ], batch size: 55, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:40:13,254 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 03:40:29,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201250 2023-11-21 03:40:32,393 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.82 vs. limit=15.0 2023-11-21 03:40:44,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1341713.3333333333, ans=0.1 2023-11-21 03:41:05,622 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8900, loss[loss=0.06424, simple_loss=0.09201, pruned_loss=0.009765, audio_tagging_loss=0.008465, over 16452.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09831, pruned_loss=0.01755, audio_tagging_loss=0.009806, over 3059655.13 frames. ], batch size: 62, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:41:22,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.61 vs. limit=15.0 2023-11-21 03:41:24,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1341913.3333333333, ans=0.1 2023-11-21 03:41:33,477 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201300 2023-11-21 03:41:38,118 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.120e+01 8.967e+01 9.567e+01 1.214e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-21 03:41:50,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1342046.6666666667, ans=0.0 2023-11-21 03:42:06,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1342113.3333333333, ans=0.125 2023-11-21 03:42:11,264 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 8950, loss[loss=0.05133, simple_loss=0.0657, pruned_loss=0.00885, audio_tagging_loss=0.009633, over 14730.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09846, pruned_loss=0.01768, audio_tagging_loss=0.009757, over 3053052.01 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:42:12,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1342180.0, ans=0.125 2023-11-21 03:42:20,729 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 03:42:37,328 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201350 2023-11-21 03:43:14,483 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9000, loss[loss=0.05128, simple_loss=0.06014, pruned_loss=0.009456, audio_tagging_loss=0.01175, over 13547.00 frames. ], tot_loss[loss=0.07695, simple_loss=0.09871, pruned_loss=0.01788, audio_tagging_loss=0.009719, over 3054398.94 frames. ], batch size: 53, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:43:14,484 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 03:43:55,720 INFO [train_asr.py:1253] (3/4) Epoch 17, validation: loss=0.06143, simple_loss=0.05268, pruned_loss=0.005433, audio_tagging_loss=0.02966, over 4681554.00 frames. 2023-11-21 03:43:55,721 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 03:44:03,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1342513.3333333333, ans=0.125 2023-11-21 03:44:23,548 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201400 2023-11-21 03:44:28,635 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.286e+01 9.090e+01 9.750e+01 1.340e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-21 03:44:45,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1342713.3333333333, ans=0.125 2023-11-21 03:44:48,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1342780.0, ans=0.1 2023-11-21 03:44:48,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.13 vs. limit=15.0 2023-11-21 03:45:00,651 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9050, loss[loss=0.08022, simple_loss=0.1011, pruned_loss=0.02004, audio_tagging_loss=0.009639, over 15191.00 frames. ], tot_loss[loss=0.07676, simple_loss=0.09869, pruned_loss=0.01777, audio_tagging_loss=0.009646, over 3056338.87 frames. ], batch size: 59, lr: 4.01e-03, grad_scale: 16.0 2023-11-21 03:45:26,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201450 2023-11-21 03:45:47,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1343046.6666666667, ans=0.125 2023-11-21 03:45:54,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1343113.3333333333, ans=0.125 2023-11-21 03:46:04,615 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9100, loss[loss=0.08261, simple_loss=0.1004, pruned_loss=0.02247, audio_tagging_loss=0.009908, over 15775.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09903, pruned_loss=0.01792, audio_tagging_loss=0.00964, over 3066757.38 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 16.0 2023-11-21 03:46:28,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1343246.6666666667, ans=0.0 2023-11-21 03:46:32,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201500 2023-11-21 03:46:36,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1343313.3333333333, ans=0.1 2023-11-21 03:46:38,212 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.193e+01 8.891e+01 9.703e+01 1.287e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-21 03:46:47,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1343380.0, ans=0.0 2023-11-21 03:46:50,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1343380.0, ans=0.0 2023-11-21 03:46:58,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.78 vs. limit=12.0 2023-11-21 03:47:08,811 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9150, loss[loss=0.07254, simple_loss=0.08763, pruned_loss=0.01994, audio_tagging_loss=0.008795, over 14166.00 frames. ], tot_loss[loss=0.076, simple_loss=0.09744, pruned_loss=0.01765, audio_tagging_loss=0.00963, over 3061017.86 frames. ], batch size: 53, lr: 4.01e-03, grad_scale: 16.0 2023-11-21 03:47:14,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1343513.3333333333, ans=0.125 2023-11-21 03:47:23,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.69 vs. limit=10.0 2023-11-21 03:47:27,320 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.70 vs. limit=22.5 2023-11-21 03:47:35,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1343646.6666666667, ans=0.0 2023-11-21 03:47:36,999 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201550 2023-11-21 03:47:48,298 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.65 vs. limit=15.0 2023-11-21 03:47:51,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1343713.3333333333, ans=0.0 2023-11-21 03:47:54,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1343713.3333333333, ans=0.125 2023-11-21 03:47:58,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1343780.0, ans=0.0 2023-11-21 03:48:13,912 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9200, loss[loss=0.04829, simple_loss=0.05502, pruned_loss=0.009931, audio_tagging_loss=0.01084, over 14716.00 frames. ], tot_loss[loss=0.07531, simple_loss=0.09645, pruned_loss=0.0175, audio_tagging_loss=0.009587, over 3056414.97 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:48:20,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1343846.6666666667, ans=0.125 2023-11-21 03:48:28,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1343913.3333333333, ans=0.125 2023-11-21 03:48:30,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.24 vs. limit=22.5 2023-11-21 03:48:33,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1343913.3333333333, ans=0.05 2023-11-21 03:48:40,530 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201600 2023-11-21 03:48:41,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1343980.0, ans=0.0 2023-11-21 03:48:46,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1343980.0, ans=0.0 2023-11-21 03:48:46,815 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.039e+01 8.659e+01 9.327e+01 1.552e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 03:48:52,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1344046.6666666667, ans=0.1 2023-11-21 03:49:10,075 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0 2023-11-21 03:49:17,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1344180.0, ans=0.1 2023-11-21 03:49:18,532 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9250, loss[loss=0.08702, simple_loss=0.1136, pruned_loss=0.02207, audio_tagging_loss=0.008155, over 15992.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.09697, pruned_loss=0.01772, audio_tagging_loss=0.009467, over 3062330.72 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:49:26,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1344180.0, ans=0.125 2023-11-21 03:49:34,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1344246.6666666667, ans=0.0 2023-11-21 03:49:38,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1344246.6666666667, ans=0.125 2023-11-21 03:49:45,606 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201650 2023-11-21 03:49:52,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1344313.3333333333, ans=0.125 2023-11-21 03:50:10,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1344446.6666666667, ans=22.5 2023-11-21 03:50:12,961 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.75 vs. limit=15.0 2023-11-21 03:50:22,233 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9300, loss[loss=0.07047, simple_loss=0.08948, pruned_loss=0.01657, audio_tagging_loss=0.00916, over 15038.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09666, pruned_loss=0.01742, audio_tagging_loss=0.009493, over 3057764.15 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:50:26,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1344513.3333333333, ans=0.125 2023-11-21 03:50:29,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1344513.3333333333, ans=0.125 2023-11-21 03:50:44,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1344580.0, ans=0.0 2023-11-21 03:50:50,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201700 2023-11-21 03:50:56,652 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 7.965e+01 8.483e+01 9.250e+01 1.325e+02, threshold=1.697e+02, percent-clipped=0.0 2023-11-21 03:51:10,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1344713.3333333333, ans=0.125 2023-11-21 03:51:16,304 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.45 vs. limit=15.0 2023-11-21 03:51:27,504 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9350, loss[loss=0.07266, simple_loss=0.08632, pruned_loss=0.017, audio_tagging_loss=0.01251, over 15475.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.09682, pruned_loss=0.01736, audio_tagging_loss=0.009525, over 3055766.07 frames. ], batch size: 61, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:51:28,337 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.93 vs. limit=6.0 2023-11-21 03:51:34,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1344846.6666666667, ans=0.125 2023-11-21 03:51:36,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.80 vs. limit=15.0 2023-11-21 03:51:39,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.49 vs. limit=10.0 2023-11-21 03:51:48,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1344913.3333333333, ans=0.04949747468305833 2023-11-21 03:51:54,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201750 2023-11-21 03:51:56,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1344980.0, ans=0.125 2023-11-21 03:52:02,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.80 vs. limit=15.0 2023-11-21 03:52:07,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1345046.6666666667, ans=0.125 2023-11-21 03:52:32,196 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9400, loss[loss=0.06568, simple_loss=0.07828, pruned_loss=0.0139, audio_tagging_loss=0.01264, over 15472.00 frames. ], tot_loss[loss=0.07575, simple_loss=0.09735, pruned_loss=0.01748, audio_tagging_loss=0.009589, over 3053068.84 frames. ], batch size: 60, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:52:58,700 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201800 2023-11-21 03:53:04,952 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.174e+01 8.028e+01 8.710e+01 9.500e+01 1.230e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-21 03:53:05,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1345313.3333333333, ans=0.1 2023-11-21 03:53:06,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1345313.3333333333, ans=0.125 2023-11-21 03:53:09,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1345380.0, ans=0.125 2023-11-21 03:53:22,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1345446.6666666667, ans=0.125 2023-11-21 03:53:33,229 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 03:53:35,699 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9450, loss[loss=0.08834, simple_loss=0.1116, pruned_loss=0.02274, audio_tagging_loss=0.00982, over 17431.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09789, pruned_loss=0.01758, audio_tagging_loss=0.009745, over 3059545.99 frames. ], batch size: 65, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:53:45,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1345513.3333333333, ans=0.1 2023-11-21 03:53:51,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.19 vs. limit=15.0 2023-11-21 03:54:03,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201850 2023-11-21 03:54:17,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1345713.3333333333, ans=0.125 2023-11-21 03:54:17,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1345713.3333333333, ans=10.0 2023-11-21 03:54:40,236 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9500, loss[loss=0.06256, simple_loss=0.08315, pruned_loss=0.01194, audio_tagging_loss=0.009041, over 14207.00 frames. ], tot_loss[loss=0.07588, simple_loss=0.09746, pruned_loss=0.01739, audio_tagging_loss=0.009759, over 3054693.47 frames. ], batch size: 54, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:54:53,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1345913.3333333333, ans=0.1 2023-11-21 03:55:01,112 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=5.771e-02 2023-11-21 03:55:06,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201900 2023-11-21 03:55:07,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.88 vs. limit=15.0 2023-11-21 03:55:12,970 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.020e+01 8.201e+01 8.880e+01 9.737e+01 1.239e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-21 03:55:14,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1345980.0, ans=0.1 2023-11-21 03:55:33,192 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.59 vs. limit=10.0 2023-11-21 03:55:44,097 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9550, loss[loss=0.05383, simple_loss=0.05643, pruned_loss=0.01193, audio_tagging_loss=0.01369, over 15676.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.0978, pruned_loss=0.01748, audio_tagging_loss=0.009782, over 3049275.85 frames. ], batch size: 61, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:55:46,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1346180.0, ans=0.0 2023-11-21 03:56:01,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1346246.6666666667, ans=0.125 2023-11-21 03:56:03,468 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.01 vs. limit=12.0 2023-11-21 03:56:10,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 201950 2023-11-21 03:56:21,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1346380.0, ans=0.125 2023-11-21 03:56:39,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1346446.6666666667, ans=0.2 2023-11-21 03:56:42,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1346446.6666666667, ans=0.125 2023-11-21 03:56:43,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1346446.6666666667, ans=0.125 2023-11-21 03:56:48,199 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9600, loss[loss=0.0544, simple_loss=0.07065, pruned_loss=0.009896, audio_tagging_loss=0.00918, over 15332.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09829, pruned_loss=0.0175, audio_tagging_loss=0.009766, over 3050108.10 frames. ], batch size: 59, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:57:04,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.24 vs. limit=15.0 2023-11-21 03:57:15,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202000 2023-11-21 03:57:17,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.74 vs. limit=12.0 2023-11-21 03:57:20,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1346646.6666666667, ans=0.1 2023-11-21 03:57:21,642 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.303e+01 7.767e+01 8.568e+01 9.180e+01 1.180e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 03:57:53,448 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9650, loss[loss=0.07597, simple_loss=0.09866, pruned_loss=0.01745, audio_tagging_loss=0.009184, over 16476.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09854, pruned_loss=0.0177, audio_tagging_loss=0.009833, over 3048800.34 frames. ], batch size: 60, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:58:03,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=26.75 vs. limit=22.5 2023-11-21 03:58:04,639 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 03:58:04,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1346913.3333333333, ans=0.1 2023-11-21 03:58:08,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1346913.3333333333, ans=0.0 2023-11-21 03:58:20,078 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202050 2023-11-21 03:58:27,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1346980.0, ans=0.125 2023-11-21 03:58:33,652 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 03:58:36,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1347046.6666666667, ans=0.125 2023-11-21 03:58:41,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1347046.6666666667, ans=0.0 2023-11-21 03:58:47,343 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 03:58:49,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.67 vs. limit=22.5 2023-11-21 03:58:52,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1347113.3333333333, ans=0.125 2023-11-21 03:58:57,084 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9700, loss[loss=0.09017, simple_loss=0.1181, pruned_loss=0.02016, audio_tagging_loss=0.01095, over 15810.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.09799, pruned_loss=0.01758, audio_tagging_loss=0.009714, over 3055584.73 frames. ], batch size: 60, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 03:59:02,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1347180.0, ans=0.0 2023-11-21 03:59:04,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1347180.0, ans=0.0 2023-11-21 03:59:24,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202100 2023-11-21 03:59:31,476 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.071e+01 8.852e+01 9.589e+01 1.175e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-21 03:59:36,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1347380.0, ans=0.125 2023-11-21 03:59:49,197 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.01 vs. limit=22.5 2023-11-21 03:59:54,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1347446.6666666667, ans=0.1 2023-11-21 04:00:01,935 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9750, loss[loss=0.06907, simple_loss=0.08892, pruned_loss=0.01391, audio_tagging_loss=0.01069, over 15256.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09792, pruned_loss=0.01758, audio_tagging_loss=0.009609, over 3048083.74 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 04:00:09,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1347513.3333333333, ans=0.035 2023-11-21 04:00:13,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1347580.0, ans=0.125 2023-11-21 04:00:21,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1347580.0, ans=0.125 2023-11-21 04:00:28,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202150 2023-11-21 04:00:29,152 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:01:05,909 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9800, loss[loss=0.06848, simple_loss=0.0737, pruned_loss=0.01921, audio_tagging_loss=0.01242, over 14408.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.09785, pruned_loss=0.01757, audio_tagging_loss=0.009596, over 3049441.56 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0 2023-11-21 04:01:08,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.24 vs. limit=15.0 2023-11-21 04:01:10,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1347846.6666666667, ans=0.125 2023-11-21 04:01:33,223 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202200 2023-11-21 04:01:40,217 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.143e+01 8.218e+01 8.911e+01 9.582e+01 1.351e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-21 04:02:02,316 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 04:02:10,983 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9850, loss[loss=0.09764, simple_loss=0.1264, pruned_loss=0.0236, audio_tagging_loss=0.01086, over 14593.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.0983, pruned_loss=0.01779, audio_tagging_loss=0.009597, over 3043006.81 frames. ], batch size: 53, lr: 4.00e-03, grad_scale: 32.0 2023-11-21 04:02:38,484 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202250 2023-11-21 04:03:00,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1348380.0, ans=0.125 2023-11-21 04:03:15,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1348513.3333333333, ans=0.0 2023-11-21 04:03:16,205 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9900, loss[loss=0.08429, simple_loss=0.11, pruned_loss=0.01965, audio_tagging_loss=0.009649, over 14647.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.09824, pruned_loss=0.01788, audio_tagging_loss=0.009544, over 3037518.47 frames. ], batch size: 53, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:03:17,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1348513.3333333333, ans=0.0 2023-11-21 04:03:44,143 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202300 2023-11-21 04:03:51,364 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.722e+01 8.036e+01 8.846e+01 9.540e+01 1.221e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 04:04:21,071 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 9950, loss[loss=0.06883, simple_loss=0.08613, pruned_loss=0.01443, audio_tagging_loss=0.01134, over 15153.00 frames. ], tot_loss[loss=0.07569, simple_loss=0.09742, pruned_loss=0.01749, audio_tagging_loss=0.009489, over 3045585.80 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:04:35,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1348913.3333333333, ans=0.125 2023-11-21 04:04:48,432 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202350 2023-11-21 04:05:06,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1349046.6666666667, ans=15.0 2023-11-21 04:05:08,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1349046.6666666667, ans=0.0 2023-11-21 04:05:22,491 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0 2023-11-21 04:05:25,842 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10000, loss[loss=0.08752, simple_loss=0.1207, pruned_loss=0.02041, audio_tagging_loss=0.00674, over 15512.00 frames. ], tot_loss[loss=0.07597, simple_loss=0.09778, pruned_loss=0.01763, audio_tagging_loss=0.009449, over 3049662.26 frames. ], batch size: 55, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:05:39,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1349246.6666666667, ans=0.125 2023-11-21 04:05:41,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1349246.6666666667, ans=0.1 2023-11-21 04:05:49,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1349246.6666666667, ans=0.2 2023-11-21 04:05:52,691 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202400 2023-11-21 04:06:02,541 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.043e+01 8.738e+01 9.707e+01 1.240e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 04:06:02,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1349313.3333333333, ans=0.125 2023-11-21 04:06:25,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1349446.6666666667, ans=0.125 2023-11-21 04:06:30,496 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10050, loss[loss=0.07692, simple_loss=0.09126, pruned_loss=0.02001, audio_tagging_loss=0.01128, over 15249.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09794, pruned_loss=0.01775, audio_tagging_loss=0.009451, over 3046566.31 frames. ], batch size: 59, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:06:46,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1349580.0, ans=0.0 2023-11-21 04:06:54,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1349580.0, ans=0.125 2023-11-21 04:06:58,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202450 2023-11-21 04:07:08,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1349713.3333333333, ans=0.125 2023-11-21 04:07:13,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1349713.3333333333, ans=0.0 2023-11-21 04:07:24,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1349780.0, ans=0.125 2023-11-21 04:07:34,518 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10100, loss[loss=0.08345, simple_loss=0.1051, pruned_loss=0.01928, audio_tagging_loss=0.01159, over 15874.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09854, pruned_loss=0.0177, audio_tagging_loss=0.009461, over 3051985.48 frames. ], batch size: 59, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:07:40,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1349846.6666666667, ans=0.0 2023-11-21 04:08:02,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202500 2023-11-21 04:08:11,187 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.354e+01 8.956e+01 9.533e+01 1.205e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-21 04:08:24,729 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 04:08:39,847 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10150, loss[loss=0.06146, simple_loss=0.07876, pruned_loss=0.008727, audio_tagging_loss=0.01335, over 15345.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.09813, pruned_loss=0.01759, audio_tagging_loss=0.009596, over 3046161.36 frames. ], batch size: 58, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:08:47,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1350180.0, ans=0.125 2023-11-21 04:08:57,917 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:09:06,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202550 2023-11-21 04:09:07,627 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 04:09:11,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1350313.3333333333, ans=0.1 2023-11-21 04:09:12,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1350313.3333333333, ans=0.0 2023-11-21 04:09:21,055 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.70 vs. limit=6.0 2023-11-21 04:09:28,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1350380.0, ans=0.125 2023-11-21 04:09:34,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1350446.6666666667, ans=0.1 2023-11-21 04:09:44,014 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10200, loss[loss=0.1183, simple_loss=0.1498, pruned_loss=0.03656, audio_tagging_loss=0.006878, over 14577.00 frames. ], tot_loss[loss=0.07649, simple_loss=0.09839, pruned_loss=0.01766, audio_tagging_loss=0.009632, over 3043979.00 frames. ], batch size: 57, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:10:04,732 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 04:10:10,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202600 2023-11-21 04:10:19,591 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.093e+01 8.508e+01 9.534e+01 1.524e+02, threshold=1.702e+02, percent-clipped=0.0 2023-11-21 04:10:31,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1350713.3333333333, ans=10.0 2023-11-21 04:10:47,479 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10250, loss[loss=0.08044, simple_loss=0.09122, pruned_loss=0.02193, audio_tagging_loss=0.0129, over 15836.00 frames. ], tot_loss[loss=0.07665, simple_loss=0.0986, pruned_loss=0.01766, audio_tagging_loss=0.009689, over 3043788.12 frames. ], batch size: 63, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:10:57,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1350846.6666666667, ans=0.125 2023-11-21 04:11:03,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1350913.3333333333, ans=0.04949747468305833 2023-11-21 04:11:07,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1350913.3333333333, ans=0.125 2023-11-21 04:11:09,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1350913.3333333333, ans=0.05 2023-11-21 04:11:14,699 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202650 2023-11-21 04:11:23,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0 2023-11-21 04:11:39,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1351113.3333333333, ans=0.0 2023-11-21 04:11:43,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.52 vs. limit=15.0 2023-11-21 04:11:49,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1351113.3333333333, ans=0.0 2023-11-21 04:11:52,702 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10300, loss[loss=0.06145, simple_loss=0.06721, pruned_loss=0.01172, audio_tagging_loss=0.01612, over 14576.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09759, pruned_loss=0.01753, audio_tagging_loss=0.009778, over 3045646.62 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:12:19,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202700 2023-11-21 04:12:20,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1351313.3333333333, ans=0.125 2023-11-21 04:12:25,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1351313.3333333333, ans=0.1 2023-11-21 04:12:27,748 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.129e+01 8.690e+01 9.404e+01 1.408e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-21 04:12:30,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1351380.0, ans=0.2 2023-11-21 04:12:33,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1351380.0, ans=0.125 2023-11-21 04:12:54,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1351446.6666666667, ans=0.125 2023-11-21 04:12:57,291 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10350, loss[loss=0.06594, simple_loss=0.0785, pruned_loss=0.01346, audio_tagging_loss=0.01323, over 15403.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09804, pruned_loss=0.0176, audio_tagging_loss=0.009888, over 3048432.24 frames. ], batch size: 58, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:13:24,366 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202750 2023-11-21 04:13:28,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1351646.6666666667, ans=0.125 2023-11-21 04:13:30,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1351646.6666666667, ans=0.125 2023-11-21 04:13:33,024 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.49 vs. limit=15.0 2023-11-21 04:13:39,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1351713.3333333333, ans=0.125 2023-11-21 04:13:49,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1351780.0, ans=0.125 2023-11-21 04:14:01,499 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10400, loss[loss=0.06627, simple_loss=0.08108, pruned_loss=0.01252, audio_tagging_loss=0.01322, over 16272.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09742, pruned_loss=0.01743, audio_tagging_loss=0.01002, over 3042269.06 frames. ], batch size: 62, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:14:03,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1351846.6666666667, ans=0.2 2023-11-21 04:14:11,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1351846.6666666667, ans=0.2 2023-11-21 04:14:25,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1351913.3333333333, ans=0.0 2023-11-21 04:14:29,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202800 2023-11-21 04:14:39,608 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.428e+01 7.927e+01 8.563e+01 9.335e+01 1.364e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-21 04:14:54,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1352113.3333333333, ans=0.0 2023-11-21 04:14:54,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1352113.3333333333, ans=0.04949747468305833 2023-11-21 04:15:06,473 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10450, loss[loss=0.06782, simple_loss=0.08561, pruned_loss=0.01415, audio_tagging_loss=0.01087, over 14827.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09687, pruned_loss=0.01736, audio_tagging_loss=0.009985, over 3045809.14 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:15:33,449 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202850 2023-11-21 04:15:48,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1352380.0, ans=0.0 2023-11-21 04:15:50,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1352380.0, ans=0.2 2023-11-21 04:15:51,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1352380.0, ans=0.1 2023-11-21 04:16:08,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.71 vs. limit=6.0 2023-11-21 04:16:10,739 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10500, loss[loss=0.05419, simple_loss=0.06678, pruned_loss=0.01016, audio_tagging_loss=0.01064, over 15611.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09682, pruned_loss=0.01749, audio_tagging_loss=0.009762, over 3044600.84 frames. ], batch size: 61, lr: 4.00e-03, grad_scale: 8.0 2023-11-21 04:16:31,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1352580.0, ans=0.125 2023-11-21 04:16:37,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202900 2023-11-21 04:16:43,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=11.36 vs. limit=12.0 2023-11-21 04:16:46,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1352646.6666666667, ans=0.2 2023-11-21 04:16:50,053 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.399e+01 8.004e+01 8.567e+01 9.229e+01 1.198e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-21 04:16:52,852 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:17:03,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1352780.0, ans=0.125 2023-11-21 04:17:07,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1352780.0, ans=0.0 2023-11-21 04:17:15,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1352846.6666666667, ans=0.1 2023-11-21 04:17:16,041 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10550, loss[loss=0.08307, simple_loss=0.1116, pruned_loss=0.02025, audio_tagging_loss=0.007012, over 15598.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09699, pruned_loss=0.01728, audio_tagging_loss=0.009633, over 3046858.18 frames. ], batch size: 57, lr: 4.00e-03, grad_scale: 8.0 2023-11-21 04:17:16,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1352846.6666666667, ans=0.05 2023-11-21 04:17:38,772 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.07 vs. limit=15.0 2023-11-21 04:17:43,216 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 202950 2023-11-21 04:18:00,685 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=2.537e-03 2023-11-21 04:18:08,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0 2023-11-21 04:18:21,420 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10600, loss[loss=0.07434, simple_loss=0.08813, pruned_loss=0.01699, audio_tagging_loss=0.01328, over 15959.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09668, pruned_loss=0.01727, audio_tagging_loss=0.009674, over 3047394.89 frames. ], batch size: 60, lr: 4.00e-03, grad_scale: 8.0 2023-11-21 04:18:22,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1353180.0, ans=0.0 2023-11-21 04:18:46,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1353313.3333333333, ans=0.1 2023-11-21 04:18:48,696 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203000 2023-11-21 04:18:51,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.24 vs. limit=15.0 2023-11-21 04:18:59,886 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.372e+01 8.133e+01 8.810e+01 9.778e+01 1.215e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-21 04:19:02,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1353380.0, ans=0.025 2023-11-21 04:19:07,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1353380.0, ans=0.125 2023-11-21 04:19:22,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1353446.6666666667, ans=0.0 2023-11-21 04:19:26,444 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10650, loss[loss=0.06389, simple_loss=0.08377, pruned_loss=0.01272, audio_tagging_loss=0.009286, over 14904.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09705, pruned_loss=0.01736, audio_tagging_loss=0.009578, over 3045893.21 frames. ], batch size: 55, lr: 4.00e-03, grad_scale: 8.0 2023-11-21 04:19:26,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1353513.3333333333, ans=0.125 2023-11-21 04:19:53,001 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203050 2023-11-21 04:20:31,169 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10700, loss[loss=0.07915, simple_loss=0.1018, pruned_loss=0.01829, audio_tagging_loss=0.009951, over 15209.00 frames. ], tot_loss[loss=0.07569, simple_loss=0.09757, pruned_loss=0.01739, audio_tagging_loss=0.009514, over 3046861.47 frames. ], batch size: 55, lr: 4.00e-03, grad_scale: 8.0 2023-11-21 04:20:57,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1353980.0, ans=0.05 2023-11-21 04:20:58,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203100 2023-11-21 04:21:06,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.08 vs. limit=6.0 2023-11-21 04:21:10,480 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.882e+01 7.991e+01 8.733e+01 9.483e+01 1.206e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 04:21:12,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1354046.6666666667, ans=0.05 2023-11-21 04:21:30,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1354113.3333333333, ans=0.125 2023-11-21 04:21:35,723 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10750, loss[loss=0.07666, simple_loss=0.1004, pruned_loss=0.01835, audio_tagging_loss=0.008105, over 16296.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09736, pruned_loss=0.01725, audio_tagging_loss=0.009597, over 3051529.44 frames. ], batch size: 60, lr: 4.00e-03, grad_scale: 8.0 2023-11-21 04:21:48,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1354246.6666666667, ans=0.125 2023-11-21 04:22:03,925 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203150 2023-11-21 04:22:05,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1354313.3333333333, ans=0.0 2023-11-21 04:22:14,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1354380.0, ans=0.125 2023-11-21 04:22:27,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1354446.6666666667, ans=0.125 2023-11-21 04:22:41,287 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10800, loss[loss=0.07911, simple_loss=0.09896, pruned_loss=0.01895, audio_tagging_loss=0.01068, over 15667.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09659, pruned_loss=0.01713, audio_tagging_loss=0.009685, over 3050763.16 frames. ], batch size: 58, lr: 4.00e-03, grad_scale: 16.0 2023-11-21 04:22:48,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1354513.3333333333, ans=0.125 2023-11-21 04:23:09,417 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203200 2023-11-21 04:23:13,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.14 vs. limit=10.0 2023-11-21 04:23:16,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1354646.6666666667, ans=0.125 2023-11-21 04:23:21,928 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.183e+01 7.867e+01 8.488e+01 9.244e+01 1.126e+02, threshold=1.698e+02, percent-clipped=0.0 2023-11-21 04:23:22,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1354713.3333333333, ans=0.0 2023-11-21 04:23:32,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1354713.3333333333, ans=0.0 2023-11-21 04:23:40,236 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.28 vs. limit=15.0 2023-11-21 04:23:41,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1354780.0, ans=0.125 2023-11-21 04:23:47,609 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10850, loss[loss=0.06213, simple_loss=0.0735, pruned_loss=0.01259, audio_tagging_loss=0.01279, over 14286.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09722, pruned_loss=0.01741, audio_tagging_loss=0.009714, over 3048471.48 frames. ], batch size: 55, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:23:52,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=15.0 2023-11-21 04:24:14,727 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203250 2023-11-21 04:24:41,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1355113.3333333333, ans=0.125 2023-11-21 04:24:45,939 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 04:24:47,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1355113.3333333333, ans=0.125 2023-11-21 04:24:52,231 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10900, loss[loss=0.07876, simple_loss=0.1003, pruned_loss=0.01861, audio_tagging_loss=0.009991, over 16432.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09768, pruned_loss=0.01743, audio_tagging_loss=0.00974, over 3053284.52 frames. ], batch size: 61, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:25:01,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.08 vs. limit=15.0 2023-11-21 04:25:08,496 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:25:12,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1355246.6666666667, ans=0.125 2023-11-21 04:25:16,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1355246.6666666667, ans=0.125 2023-11-21 04:25:16,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1355246.6666666667, ans=0.125 2023-11-21 04:25:19,299 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.84 vs. limit=15.0 2023-11-21 04:25:20,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203300 2023-11-21 04:25:20,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1355313.3333333333, ans=0.125 2023-11-21 04:25:21,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1355313.3333333333, ans=0.0 2023-11-21 04:25:31,857 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.236e+01 9.164e+01 1.009e+02 1.942e+02, threshold=1.833e+02, percent-clipped=1.0 2023-11-21 04:25:34,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1355380.0, ans=0.2 2023-11-21 04:25:39,688 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.11 vs. limit=15.0 2023-11-21 04:25:44,298 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.92 vs. limit=15.0 2023-11-21 04:25:57,473 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 10950, loss[loss=0.05887, simple_loss=0.07833, pruned_loss=0.01313, audio_tagging_loss=0.006573, over 14563.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09755, pruned_loss=0.01752, audio_tagging_loss=0.009814, over 3052714.15 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:26:20,380 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.37 vs. limit=22.5 2023-11-21 04:26:24,781 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203350 2023-11-21 04:26:39,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1355713.3333333333, ans=0.125 2023-11-21 04:26:50,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.28 vs. limit=6.0 2023-11-21 04:26:50,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1355780.0, ans=0.125 2023-11-21 04:26:55,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1355780.0, ans=0.0 2023-11-21 04:26:55,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1355780.0, ans=0.125 2023-11-21 04:26:56,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1355780.0, ans=0.0 2023-11-21 04:27:02,254 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11000, loss[loss=0.06872, simple_loss=0.0828, pruned_loss=0.01613, audio_tagging_loss=0.01118, over 14826.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09735, pruned_loss=0.01757, audio_tagging_loss=0.009857, over 3050412.17 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:27:05,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1355846.6666666667, ans=0.0 2023-11-21 04:27:10,225 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 04:27:25,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.35 vs. limit=5.0 2023-11-21 04:27:27,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1355980.0, ans=0.2 2023-11-21 04:27:28,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1355980.0, ans=0.035 2023-11-21 04:27:29,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203400 2023-11-21 04:27:40,974 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.079e+01 8.776e+01 9.579e+01 1.135e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 04:27:45,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1356046.6666666667, ans=0.125 2023-11-21 04:27:50,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1356046.6666666667, ans=0.125 2023-11-21 04:27:57,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1356113.3333333333, ans=0.0 2023-11-21 04:28:06,821 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11050, loss[loss=0.08426, simple_loss=0.1109, pruned_loss=0.02095, audio_tagging_loss=0.007846, over 16734.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09763, pruned_loss=0.0177, audio_tagging_loss=0.009889, over 3049205.34 frames. ], batch size: 63, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:28:34,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203450 2023-11-21 04:28:47,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5 2023-11-21 04:29:00,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1356446.6666666667, ans=0.125 2023-11-21 04:29:04,528 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:29:11,781 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11100, loss[loss=0.06328, simple_loss=0.08135, pruned_loss=0.01089, audio_tagging_loss=0.01172, over 13846.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09657, pruned_loss=0.01741, audio_tagging_loss=0.01003, over 3049540.58 frames. ], batch size: 53, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:29:19,935 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.56 vs. limit=22.5 2023-11-21 04:29:34,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=22.5 2023-11-21 04:29:39,520 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203500 2023-11-21 04:29:51,161 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.645e+01 8.437e+01 9.142e+01 9.871e+01 1.258e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-21 04:29:51,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1356713.3333333333, ans=0.1 2023-11-21 04:30:05,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1356780.0, ans=0.2 2023-11-21 04:30:14,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1356780.0, ans=0.125 2023-11-21 04:30:16,974 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11150, loss[loss=0.08139, simple_loss=0.09896, pruned_loss=0.01947, audio_tagging_loss=0.01244, over 14610.00 frames. ], tot_loss[loss=0.07659, simple_loss=0.09751, pruned_loss=0.01777, audio_tagging_loss=0.01006, over 3047722.00 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:30:17,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1356846.6666666667, ans=0.125 2023-11-21 04:30:45,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203550 2023-11-21 04:30:47,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1356980.0, ans=0.0 2023-11-21 04:30:54,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.13 vs. limit=15.0 2023-11-21 04:31:04,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1357046.6666666667, ans=0.0 2023-11-21 04:31:23,081 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11200, loss[loss=0.07617, simple_loss=0.1005, pruned_loss=0.01794, audio_tagging_loss=0.00798, over 15764.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09673, pruned_loss=0.01748, audio_tagging_loss=0.01009, over 3049381.10 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:31:28,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1357180.0, ans=0.0 2023-11-21 04:31:51,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203600 2023-11-21 04:31:55,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1357313.3333333333, ans=0.125 2023-11-21 04:31:56,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1357313.3333333333, ans=0.125 2023-11-21 04:32:02,366 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.264e+01 8.069e+01 8.575e+01 9.412e+01 1.593e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-21 04:32:10,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1357380.0, ans=0.1 2023-11-21 04:32:29,575 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11250, loss[loss=0.07606, simple_loss=0.09993, pruned_loss=0.0172, audio_tagging_loss=0.00889, over 14838.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09603, pruned_loss=0.01729, audio_tagging_loss=0.01004, over 3045772.95 frames. ], batch size: 55, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:32:39,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1357513.3333333333, ans=0.2 2023-11-21 04:32:44,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1357580.0, ans=0.0 2023-11-21 04:32:56,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203650 2023-11-21 04:32:57,128 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.28 vs. limit=22.5 2023-11-21 04:33:03,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.84 vs. limit=6.0 2023-11-21 04:33:07,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1357713.3333333333, ans=0.125 2023-11-21 04:33:25,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1357780.0, ans=0.125 2023-11-21 04:33:35,050 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11300, loss[loss=0.06686, simple_loss=0.08139, pruned_loss=0.01223, audio_tagging_loss=0.01393, over 17056.00 frames. ], tot_loss[loss=0.07631, simple_loss=0.0976, pruned_loss=0.0176, audio_tagging_loss=0.009907, over 3043205.00 frames. ], batch size: 64, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:33:41,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1357846.6666666667, ans=0.0 2023-11-21 04:33:52,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1357913.3333333333, ans=0.125 2023-11-21 04:33:55,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.04 vs. limit=22.5 2023-11-21 04:34:02,641 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203700 2023-11-21 04:34:13,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1358046.6666666667, ans=0.1 2023-11-21 04:34:14,174 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.294e+01 8.174e+01 8.923e+01 9.788e+01 1.326e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-21 04:34:39,494 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11350, loss[loss=0.06316, simple_loss=0.08503, pruned_loss=0.01375, audio_tagging_loss=0.006895, over 14427.00 frames. ], tot_loss[loss=0.07575, simple_loss=0.09679, pruned_loss=0.0175, audio_tagging_loss=0.009861, over 3043229.69 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:35:00,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1358246.6666666667, ans=0.1 2023-11-21 04:35:05,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1358313.3333333333, ans=15.0 2023-11-21 04:35:07,482 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203750 2023-11-21 04:35:08,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.88 vs. limit=22.5 2023-11-21 04:35:23,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1358380.0, ans=0.09899494936611666 2023-11-21 04:35:46,324 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11400, loss[loss=0.07283, simple_loss=0.07813, pruned_loss=0.02173, audio_tagging_loss=0.01203, over 15346.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09621, pruned_loss=0.01734, audio_tagging_loss=0.009771, over 3042963.82 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:35:50,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1358513.3333333333, ans=0.5 2023-11-21 04:36:13,233 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203800 2023-11-21 04:36:24,722 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.066e+01 8.848e+01 9.491e+01 1.619e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-21 04:36:30,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1358713.3333333333, ans=0.0 2023-11-21 04:36:52,319 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11450, loss[loss=0.08188, simple_loss=0.1074, pruned_loss=0.01698, audio_tagging_loss=0.01121, over 15501.00 frames. ], tot_loss[loss=0.07564, simple_loss=0.09689, pruned_loss=0.01755, audio_tagging_loss=0.009636, over 3047362.81 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:36:56,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1358846.6666666667, ans=0.125 2023-11-21 04:36:57,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1358846.6666666667, ans=0.125 2023-11-21 04:37:13,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1358913.3333333333, ans=0.1 2023-11-21 04:37:16,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1358980.0, ans=0.125 2023-11-21 04:37:18,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203850 2023-11-21 04:37:22,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1358980.0, ans=0.0 2023-11-21 04:37:34,209 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.88 vs. limit=15.0 2023-11-21 04:37:44,746 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.19 vs. limit=22.5 2023-11-21 04:37:56,309 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11500, loss[loss=0.08761, simple_loss=0.1234, pruned_loss=0.01859, audio_tagging_loss=0.00734, over 15167.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09608, pruned_loss=0.01745, audio_tagging_loss=0.009626, over 3034611.26 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:38:07,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1359246.6666666667, ans=0.0 2023-11-21 04:38:12,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1359246.6666666667, ans=0.125 2023-11-21 04:38:23,971 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203900 2023-11-21 04:38:27,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.28 vs. limit=6.0 2023-11-21 04:38:28,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.15 vs. limit=22.5 2023-11-21 04:38:35,520 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.753e+01 8.131e+01 8.864e+01 9.970e+01 1.435e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-21 04:39:01,257 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11550, loss[loss=0.0772, simple_loss=0.09814, pruned_loss=0.01812, audio_tagging_loss=0.01001, over 14816.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09576, pruned_loss=0.01738, audio_tagging_loss=0.009739, over 3039241.54 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:39:02,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.83 vs. limit=15.0 2023-11-21 04:39:28,150 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 203950 2023-11-21 04:39:37,822 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 04:39:44,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1359713.3333333333, ans=0.125 2023-11-21 04:39:53,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1359780.0, ans=0.0 2023-11-21 04:40:05,781 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11600, loss[loss=0.06587, simple_loss=0.08471, pruned_loss=0.01506, audio_tagging_loss=0.008449, over 16033.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09659, pruned_loss=0.01751, audio_tagging_loss=0.009568, over 3035441.26 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 32.0 2023-11-21 04:40:12,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1359846.6666666667, ans=0.125 2023-11-21 04:40:29,232 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.12 vs. limit=15.0 2023-11-21 04:40:32,512 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204000 2023-11-21 04:40:33,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1359980.0, ans=0.0 2023-11-21 04:40:51,358 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.095e+01 8.795e+01 9.597e+01 1.214e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 04:41:14,563 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11650, loss[loss=0.08512, simple_loss=0.1129, pruned_loss=0.02109, audio_tagging_loss=0.007567, over 14532.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09704, pruned_loss=0.01767, audio_tagging_loss=0.009591, over 3028639.21 frames. ], batch size: 53, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:41:22,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1360180.0, ans=0.0 2023-11-21 04:41:41,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1360313.3333333333, ans=0.1 2023-11-21 04:41:42,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204050 2023-11-21 04:42:12,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1360446.6666666667, ans=0.0 2023-11-21 04:42:14,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1360446.6666666667, ans=0.1 2023-11-21 04:42:15,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1360446.6666666667, ans=0.125 2023-11-21 04:42:16,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1360446.6666666667, ans=0.0 2023-11-21 04:42:19,567 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11700, loss[loss=0.06255, simple_loss=0.07945, pruned_loss=0.01147, audio_tagging_loss=0.01136, over 15624.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09662, pruned_loss=0.01741, audio_tagging_loss=0.009694, over 3032658.11 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:42:19,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1360513.3333333333, ans=0.0 2023-11-21 04:42:33,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1360580.0, ans=0.125 2023-11-21 04:42:36,797 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:42:38,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1360580.0, ans=0.04949747468305833 2023-11-21 04:42:47,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204100 2023-11-21 04:42:53,610 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:43:00,701 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.797e+01 8.480e+01 9.195e+01 1.315e+02, threshold=1.696e+02, percent-clipped=0.0 2023-11-21 04:43:13,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1360780.0, ans=0.125 2023-11-21 04:43:13,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0 2023-11-21 04:43:24,813 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11750, loss[loss=0.07546, simple_loss=0.1035, pruned_loss=0.01284, audio_tagging_loss=0.01086, over 15328.00 frames. ], tot_loss[loss=0.07513, simple_loss=0.09633, pruned_loss=0.0173, audio_tagging_loss=0.009662, over 3035914.02 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:43:42,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1360913.3333333333, ans=0.07 2023-11-21 04:43:44,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1360913.3333333333, ans=0.0 2023-11-21 04:43:46,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1360913.3333333333, ans=0.1 2023-11-21 04:43:48,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1360913.3333333333, ans=0.0 2023-11-21 04:43:49,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1360980.0, ans=0.0 2023-11-21 04:43:51,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204150 2023-11-21 04:43:59,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1360980.0, ans=0.2 2023-11-21 04:44:01,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1360980.0, ans=0.125 2023-11-21 04:44:16,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1361113.3333333333, ans=0.125 2023-11-21 04:44:21,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1361113.3333333333, ans=0.1 2023-11-21 04:44:29,367 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11800, loss[loss=0.08577, simple_loss=0.1201, pruned_loss=0.01629, audio_tagging_loss=0.009406, over 15214.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09611, pruned_loss=0.01725, audio_tagging_loss=0.009712, over 3031143.56 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:44:41,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1361246.6666666667, ans=0.125 2023-11-21 04:44:56,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204200 2023-11-21 04:45:01,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.20 vs. limit=12.0 2023-11-21 04:45:11,504 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.252e+01 9.002e+01 9.752e+01 1.410e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-21 04:45:18,078 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:45:23,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1361446.6666666667, ans=0.0 2023-11-21 04:45:32,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1361513.3333333333, ans=0.035 2023-11-21 04:45:33,870 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11850, loss[loss=0.07195, simple_loss=0.08799, pruned_loss=0.01584, audio_tagging_loss=0.01212, over 14892.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09639, pruned_loss=0.01743, audio_tagging_loss=0.009809, over 3036574.36 frames. ], batch size: 54, lr: 3.99e-03, grad_scale: 16.0 2023-11-21 04:45:46,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1361580.0, ans=0.2 2023-11-21 04:45:53,576 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:45:58,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1361580.0, ans=0.125 2023-11-21 04:46:01,801 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.56 vs. limit=10.0 2023-11-21 04:46:02,523 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204250 2023-11-21 04:46:03,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1361646.6666666667, ans=0.125 2023-11-21 04:46:37,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1361780.0, ans=0.0 2023-11-21 04:46:39,223 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11900, loss[loss=0.05357, simple_loss=0.06673, pruned_loss=0.008951, audio_tagging_loss=0.01125, over 15200.00 frames. ], tot_loss[loss=0.07575, simple_loss=0.09662, pruned_loss=0.01754, audio_tagging_loss=0.009901, over 3033595.87 frames. ], batch size: 58, lr: 3.98e-03, grad_scale: 16.0 2023-11-21 04:46:48,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1361846.6666666667, ans=0.0 2023-11-21 04:46:49,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.31 vs. limit=10.0 2023-11-21 04:46:51,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.31 vs. limit=15.0 2023-11-21 04:47:06,330 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204300 2023-11-21 04:47:16,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1362046.6666666667, ans=0.125 2023-11-21 04:47:20,910 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.041e+01 8.628e+01 9.158e+01 1.228e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-21 04:47:22,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1362046.6666666667, ans=0.2 2023-11-21 04:47:23,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.04 vs. limit=15.0 2023-11-21 04:47:44,169 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 11950, loss[loss=0.08452, simple_loss=0.1088, pruned_loss=0.02014, audio_tagging_loss=0.009978, over 15494.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.0961, pruned_loss=0.01737, audio_tagging_loss=0.009975, over 3041238.21 frames. ], batch size: 59, lr: 3.98e-03, grad_scale: 16.0 2023-11-21 04:47:45,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1362180.0, ans=0.125 2023-11-21 04:47:52,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1362180.0, ans=0.0 2023-11-21 04:47:57,871 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.24 vs. limit=12.0 2023-11-21 04:48:02,858 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.90 vs. limit=15.0 2023-11-21 04:48:10,681 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204350 2023-11-21 04:48:32,469 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.272e-01 2023-11-21 04:48:35,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1362446.6666666667, ans=0.1 2023-11-21 04:48:36,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1362446.6666666667, ans=0.0 2023-11-21 04:48:45,540 INFO [train_asr.py:1221] (3/4) Epoch 17, batch 12000, loss[loss=0.08533, simple_loss=0.104, pruned_loss=0.02109, audio_tagging_loss=0.01223, over 15397.00 frames. ], tot_loss[loss=0.07552, simple_loss=0.09628, pruned_loss=0.01738, audio_tagging_loss=0.01, over 3044217.85 frames. ], batch size: 57, lr: 3.98e-03, grad_scale: 32.0 2023-11-21 04:48:45,541 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 04:49:13,230 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.9684, 5.8926, 5.7026, 5.6229], device='cuda:3') 2023-11-21 04:49:26,091 INFO [train_asr.py:1253] (3/4) Epoch 17, validation: loss=0.06069, simple_loss=0.05267, pruned_loss=0.005387, audio_tagging_loss=0.02896, over 4681554.00 frames. 2023-11-21 04:49:26,092 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 04:49:40,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.75 vs. limit=22.5 2023-11-21 04:49:41,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1362580.0, ans=0.125 2023-11-21 04:49:51,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204400 2023-11-21 04:50:33,270 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 0, loss[loss=0.08058, simple_loss=0.08874, pruned_loss=0.01442, audio_tagging_loss=0.02179, over 15667.00 frames. ], tot_loss[loss=0.08058, simple_loss=0.08874, pruned_loss=0.01442, audio_tagging_loss=0.02179, over 15667.00 frames. ], batch size: 58, lr: 3.87e-03, grad_scale: 32.0 2023-11-21 04:50:33,271 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 04:50:54,361 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7727, 5.7830, 5.8546, 5.8533], device='cuda:3') 2023-11-21 04:51:08,561 INFO [train_asr.py:1253] (3/4) Epoch 18, validation: loss=0.05959, simple_loss=0.05266, pruned_loss=0.005405, audio_tagging_loss=0.02786, over 4681554.00 frames. 2023-11-21 04:51:08,562 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 04:51:19,302 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.498e+01 8.070e+01 8.803e+01 9.675e+01 1.246e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-21 04:51:54,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1362866.6666666667, ans=0.125 2023-11-21 04:52:02,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0 2023-11-21 04:52:09,450 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204450 2023-11-21 04:52:11,802 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 50, loss[loss=0.09678, simple_loss=0.1186, pruned_loss=0.02256, audio_tagging_loss=0.01491, over 15143.00 frames. ], tot_loss[loss=0.0859, simple_loss=0.1003, pruned_loss=0.01769, audio_tagging_loss=0.01803, over 694442.36 frames. ], batch size: 55, lr: 3.87e-03, grad_scale: 32.0 2023-11-21 04:52:15,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1363000.0, ans=0.125 2023-11-21 04:52:21,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1363000.0, ans=0.125 2023-11-21 04:52:29,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1363066.6666666667, ans=0.04949747468305833 2023-11-21 04:53:13,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204500 2023-11-21 04:53:15,775 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 100, loss[loss=0.07293, simple_loss=0.1016, pruned_loss=0.01045, audio_tagging_loss=0.0117, over 14966.00 frames. ], tot_loss[loss=0.08481, simple_loss=0.09928, pruned_loss=0.01761, audio_tagging_loss=0.01756, over 1219904.95 frames. ], batch size: 55, lr: 3.87e-03, grad_scale: 32.0 2023-11-21 04:53:28,005 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.544e+01 9.245e+01 1.008e+02 1.490e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-21 04:53:31,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1363400.0, ans=0.125 2023-11-21 04:53:33,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1363400.0, ans=0.0 2023-11-21 04:53:36,697 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.14 vs. limit=15.0 2023-11-21 04:53:39,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1363400.0, ans=0.125 2023-11-21 04:54:18,650 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204550 2023-11-21 04:54:21,055 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 150, loss[loss=0.08019, simple_loss=0.09783, pruned_loss=0.01893, audio_tagging_loss=0.01235, over 16509.00 frames. ], tot_loss[loss=0.08093, simple_loss=0.0956, pruned_loss=0.01701, audio_tagging_loss=0.01612, over 1616175.16 frames. ], batch size: 62, lr: 3.87e-03, grad_scale: 32.0 2023-11-21 04:54:21,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1363666.6666666667, ans=0.125 2023-11-21 04:54:42,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1363733.3333333333, ans=0.0 2023-11-21 04:54:43,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1363733.3333333333, ans=0.125 2023-11-21 04:54:44,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1363733.3333333333, ans=0.0 2023-11-21 04:55:02,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.42 vs. limit=5.0 2023-11-21 04:55:23,538 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204600 2023-11-21 04:55:26,216 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 200, loss[loss=0.06669, simple_loss=0.08651, pruned_loss=0.01329, audio_tagging_loss=0.01013, over 15398.00 frames. ], tot_loss[loss=0.07959, simple_loss=0.0965, pruned_loss=0.01708, audio_tagging_loss=0.01425, over 1930720.88 frames. ], batch size: 57, lr: 3.87e-03, grad_scale: 32.0 2023-11-21 04:55:28,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1364000.0, ans=0.125 2023-11-21 04:55:37,063 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.028e+01 8.648e+01 9.287e+01 1.143e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-21 04:55:46,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1364066.6666666667, ans=0.1 2023-11-21 04:56:03,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1364200.0, ans=0.125 2023-11-21 04:56:03,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1364200.0, ans=0.125 2023-11-21 04:56:03,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1364200.0, ans=0.125 2023-11-21 04:56:17,256 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:56:26,745 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204650 2023-11-21 04:56:29,166 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 250, loss[loss=0.08126, simple_loss=0.1013, pruned_loss=0.02154, audio_tagging_loss=0.009086, over 14644.00 frames. ], tot_loss[loss=0.07932, simple_loss=0.09851, pruned_loss=0.01738, audio_tagging_loss=0.01268, over 2177934.58 frames. ], batch size: 55, lr: 3.87e-03, grad_scale: 16.0 2023-11-21 04:56:40,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1364333.3333333333, ans=0.0 2023-11-21 04:57:08,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=1364533.3333333333, ans=0.02 2023-11-21 04:57:08,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1364533.3333333333, ans=0.0 2023-11-21 04:57:10,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1364533.3333333333, ans=0.0 2023-11-21 04:57:31,696 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204700 2023-11-21 04:57:34,634 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 300, loss[loss=0.06833, simple_loss=0.08776, pruned_loss=0.01678, audio_tagging_loss=0.007664, over 15137.00 frames. ], tot_loss[loss=0.07903, simple_loss=0.09925, pruned_loss=0.01767, audio_tagging_loss=0.01173, over 2377403.91 frames. ], batch size: 57, lr: 3.87e-03, grad_scale: 16.0 2023-11-21 04:57:43,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1364666.6666666667, ans=0.125 2023-11-21 04:57:43,477 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 04:57:46,743 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.596e+01 8.107e+01 8.931e+01 9.351e+01 1.204e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-21 04:57:54,376 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=15.0 2023-11-21 04:58:03,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.85 vs. limit=15.0 2023-11-21 04:58:25,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1364933.3333333333, ans=0.125 2023-11-21 04:58:35,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204750 2023-11-21 04:58:37,494 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 350, loss[loss=0.06438, simple_loss=0.07527, pruned_loss=0.01515, audio_tagging_loss=0.0116, over 14063.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.09902, pruned_loss=0.01751, audio_tagging_loss=0.01116, over 2521295.07 frames. ], batch size: 53, lr: 3.87e-03, grad_scale: 16.0 2023-11-21 04:59:09,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1365133.3333333333, ans=0.1 2023-11-21 04:59:26,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1365200.0, ans=0.125 2023-11-21 04:59:29,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1365266.6666666667, ans=0.2 2023-11-21 04:59:39,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204800 2023-11-21 04:59:42,262 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 400, loss[loss=0.08488, simple_loss=0.1099, pruned_loss=0.02033, audio_tagging_loss=0.009596, over 15849.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09838, pruned_loss=0.0175, audio_tagging_loss=0.01082, over 2641036.75 frames. ], batch size: 57, lr: 3.87e-03, grad_scale: 32.0 2023-11-21 04:59:47,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1365333.3333333333, ans=0.125 2023-11-21 04:59:55,644 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.688e+01 8.042e+01 8.695e+01 9.547e+01 1.167e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 05:00:08,204 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.23 vs. limit=5.0 2023-11-21 05:00:37,046 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.78 vs. limit=15.0 2023-11-21 05:00:44,998 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204850 2023-11-21 05:00:45,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1365600.0, ans=0.125 2023-11-21 05:00:47,471 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 450, loss[loss=0.07782, simple_loss=0.1012, pruned_loss=0.01886, audio_tagging_loss=0.008342, over 14877.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09882, pruned_loss=0.0177, audio_tagging_loss=0.01046, over 2729691.21 frames. ], batch size: 54, lr: 3.87e-03, grad_scale: 32.0 2023-11-21 05:01:22,201 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.50 vs. limit=15.0 2023-11-21 05:01:23,359 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=11.06 vs. limit=12.0 2023-11-21 05:01:27,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1365866.6666666667, ans=0.0 2023-11-21 05:01:50,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204900 2023-11-21 05:01:51,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1366000.0, ans=0.125 2023-11-21 05:01:52,752 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 500, loss[loss=0.06523, simple_loss=0.07385, pruned_loss=0.01265, audio_tagging_loss=0.01565, over 15494.00 frames. ], tot_loss[loss=0.07687, simple_loss=0.09802, pruned_loss=0.01756, audio_tagging_loss=0.01029, over 2805013.13 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:02:01,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1366000.0, ans=0.1 2023-11-21 05:02:06,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1366066.6666666667, ans=0.125 2023-11-21 05:02:07,169 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.144e+01 8.900e+01 9.756e+01 1.828e+02, threshold=1.780e+02, percent-clipped=1.0 2023-11-21 05:02:13,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1366066.6666666667, ans=0.0 2023-11-21 05:02:27,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1366133.3333333333, ans=0.1 2023-11-21 05:02:55,391 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 204950 2023-11-21 05:02:57,725 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 550, loss[loss=0.06424, simple_loss=0.07367, pruned_loss=0.01605, audio_tagging_loss=0.01135, over 14326.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.09701, pruned_loss=0.01739, audio_tagging_loss=0.01022, over 2848481.84 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:03:20,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1366400.0, ans=0.125 2023-11-21 05:03:25,789 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.39 vs. limit=12.0 2023-11-21 05:03:31,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1366466.6666666667, ans=0.125 2023-11-21 05:03:35,332 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=9.337e-02 2023-11-21 05:03:51,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1366600.0, ans=0.125 2023-11-21 05:03:58,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1366600.0, ans=0.125 2023-11-21 05:03:59,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205000 2023-11-21 05:04:02,521 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 600, loss[loss=0.05832, simple_loss=0.07914, pruned_loss=0.009884, audio_tagging_loss=0.008868, over 14530.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09717, pruned_loss=0.01743, audio_tagging_loss=0.01013, over 2892778.01 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:04:06,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1366666.6666666667, ans=0.125 2023-11-21 05:04:16,851 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.071e+01 8.678e+01 9.459e+01 1.222e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-21 05:04:19,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1366733.3333333333, ans=0.5 2023-11-21 05:04:47,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1366866.6666666667, ans=0.125 2023-11-21 05:04:59,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.18 vs. limit=15.0 2023-11-21 05:05:05,024 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205050 2023-11-21 05:05:07,390 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 650, loss[loss=0.08104, simple_loss=0.1135, pruned_loss=0.01449, audio_tagging_loss=0.009815, over 15264.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09655, pruned_loss=0.01719, audio_tagging_loss=0.01015, over 2922373.42 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:05:52,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1367200.0, ans=0.125 2023-11-21 05:06:01,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1367266.6666666667, ans=0.125 2023-11-21 05:06:10,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205100 2023-11-21 05:06:12,495 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 700, loss[loss=0.0755, simple_loss=0.1005, pruned_loss=0.01601, audio_tagging_loss=0.009249, over 16273.00 frames. ], tot_loss[loss=0.07596, simple_loss=0.0974, pruned_loss=0.01728, audio_tagging_loss=0.009984, over 2956273.90 frames. ], batch size: 59, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:06:27,560 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.155e+01 8.827e+01 9.700e+01 1.144e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-21 05:06:51,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1367533.3333333333, ans=0.1 2023-11-21 05:07:07,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1367600.0, ans=0.0 2023-11-21 05:07:16,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205150 2023-11-21 05:07:19,613 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 750, loss[loss=0.06508, simple_loss=0.08208, pruned_loss=0.01403, audio_tagging_loss=0.01, over 14048.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09746, pruned_loss=0.01724, audio_tagging_loss=0.01002, over 2982030.45 frames. ], batch size: 53, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:07:23,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1367666.6666666667, ans=0.125 2023-11-21 05:07:31,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1367733.3333333333, ans=0.125 2023-11-21 05:07:35,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1367733.3333333333, ans=0.2 2023-11-21 05:07:40,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1367733.3333333333, ans=0.125 2023-11-21 05:07:54,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1367800.0, ans=0.125 2023-11-21 05:08:04,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1367866.6666666667, ans=0.2 2023-11-21 05:08:22,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205200 2023-11-21 05:08:25,691 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 800, loss[loss=0.05876, simple_loss=0.06778, pruned_loss=0.01453, audio_tagging_loss=0.01034, over 15244.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09829, pruned_loss=0.0175, audio_tagging_loss=0.009988, over 2991017.46 frames. ], batch size: 58, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:08:33,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1368000.0, ans=0.125 2023-11-21 05:08:39,575 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.688e+01 8.168e+01 8.753e+01 9.823e+01 1.363e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 05:08:57,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1368133.3333333333, ans=0.125 2023-11-21 05:09:23,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1368266.6666666667, ans=0.125 2023-11-21 05:09:28,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205250 2023-11-21 05:09:30,516 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 850, loss[loss=0.06666, simple_loss=0.08714, pruned_loss=0.01422, audio_tagging_loss=0.008864, over 14880.00 frames. ], tot_loss[loss=0.07732, simple_loss=0.09885, pruned_loss=0.01787, audio_tagging_loss=0.01003, over 3005255.17 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:09:33,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1368333.3333333333, ans=0.125 2023-11-21 05:10:08,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1368533.3333333333, ans=0.1 2023-11-21 05:10:17,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1368533.3333333333, ans=0.125 2023-11-21 05:10:24,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1368600.0, ans=0.0 2023-11-21 05:10:32,139 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205300 2023-11-21 05:10:35,131 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 900, loss[loss=0.06139, simple_loss=0.07067, pruned_loss=0.01526, audio_tagging_loss=0.0108, over 14765.00 frames. ], tot_loss[loss=0.07686, simple_loss=0.09816, pruned_loss=0.01771, audio_tagging_loss=0.01008, over 3015324.23 frames. ], batch size: 58, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:10:35,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1368666.6666666667, ans=0.035 2023-11-21 05:10:39,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1368666.6666666667, ans=0.0 2023-11-21 05:10:40,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1368666.6666666667, ans=0.2 2023-11-21 05:10:47,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.44 vs. limit=22.5 2023-11-21 05:10:50,737 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.117e+01 8.694e+01 9.614e+01 1.237e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 05:11:04,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.40 vs. limit=6.0 2023-11-21 05:11:19,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1368866.6666666667, ans=0.0 2023-11-21 05:11:31,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1368933.3333333333, ans=0.09899494936611666 2023-11-21 05:11:37,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1368933.3333333333, ans=0.125 2023-11-21 05:11:39,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205350 2023-11-21 05:11:41,857 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 950, loss[loss=0.06757, simple_loss=0.08277, pruned_loss=0.01814, audio_tagging_loss=0.00805, over 15746.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09829, pruned_loss=0.01778, audio_tagging_loss=0.009919, over 3020384.51 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:11:42,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1369000.0, ans=0.125 2023-11-21 05:11:53,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1369066.6666666667, ans=0.125 2023-11-21 05:12:00,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1369066.6666666667, ans=0.1 2023-11-21 05:12:30,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1369200.0, ans=0.125 2023-11-21 05:12:38,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1369266.6666666667, ans=0.2 2023-11-21 05:12:43,710 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205400 2023-11-21 05:12:46,439 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1000, loss[loss=0.08634, simple_loss=0.1163, pruned_loss=0.02265, audio_tagging_loss=0.005542, over 14551.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.09868, pruned_loss=0.01799, audio_tagging_loss=0.009647, over 3029385.85 frames. ], batch size: 53, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:12:46,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1369333.3333333333, ans=0.0 2023-11-21 05:12:54,242 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:13:01,197 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.289e+01 9.122e+01 9.845e+01 1.226e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-21 05:13:13,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1369466.6666666667, ans=0.125 2023-11-21 05:13:14,392 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 05:13:29,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.10 vs. limit=15.0 2023-11-21 05:13:33,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1369533.3333333333, ans=0.0 2023-11-21 05:13:48,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205450 2023-11-21 05:13:48,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1369600.0, ans=0.2 2023-11-21 05:13:50,689 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1050, loss[loss=0.08382, simple_loss=0.1115, pruned_loss=0.02164, audio_tagging_loss=0.006417, over 13945.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09774, pruned_loss=0.01778, audio_tagging_loss=0.009626, over 3029529.53 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:13:55,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1369666.6666666667, ans=0.125 2023-11-21 05:13:55,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1369666.6666666667, ans=10.0 2023-11-21 05:14:04,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1369733.3333333333, ans=0.2 2023-11-21 05:14:13,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1369733.3333333333, ans=0.5 2023-11-21 05:14:26,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.64 vs. limit=15.0 2023-11-21 05:14:37,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1369866.6666666667, ans=0.125 2023-11-21 05:14:55,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205500 2023-11-21 05:14:57,790 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1100, loss[loss=0.07083, simple_loss=0.0929, pruned_loss=0.01465, audio_tagging_loss=0.009727, over 14679.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09724, pruned_loss=0.01778, audio_tagging_loss=0.009508, over 3037201.25 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:15:01,472 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 05:15:07,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1370000.0, ans=0.125 2023-11-21 05:15:07,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1370000.0, ans=0.0 2023-11-21 05:15:12,509 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 8.447e+01 9.123e+01 9.824e+01 1.304e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-21 05:15:15,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1370066.6666666667, ans=0.0 2023-11-21 05:15:53,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1370266.6666666667, ans=0.125 2023-11-21 05:15:59,771 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205550 2023-11-21 05:16:02,106 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1150, loss[loss=0.08748, simple_loss=0.1113, pruned_loss=0.02579, audio_tagging_loss=0.006042, over 15492.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09744, pruned_loss=0.01779, audio_tagging_loss=0.009483, over 3039192.39 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0 2023-11-21 05:16:29,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1370466.6666666667, ans=0.0 2023-11-21 05:16:30,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1370466.6666666667, ans=0.125 2023-11-21 05:16:35,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.22 vs. limit=15.0 2023-11-21 05:16:42,401 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.74 vs. limit=15.0 2023-11-21 05:16:44,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1370533.3333333333, ans=0.125 2023-11-21 05:17:04,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205600 2023-11-21 05:17:07,020 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1200, loss[loss=0.09021, simple_loss=0.1154, pruned_loss=0.02538, audio_tagging_loss=0.007156, over 14002.00 frames. ], tot_loss[loss=0.07607, simple_loss=0.09752, pruned_loss=0.01782, audio_tagging_loss=0.009484, over 3035841.72 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:17:23,715 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.058e+01 8.835e+01 9.727e+01 1.276e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 05:17:25,928 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.24 vs. limit=22.5 2023-11-21 05:17:40,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1370800.0, ans=0.125 2023-11-21 05:17:56,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.43 vs. limit=22.5 2023-11-21 05:18:09,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1370933.3333333333, ans=0.0 2023-11-21 05:18:10,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205650 2023-11-21 05:18:12,814 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1250, loss[loss=0.06651, simple_loss=0.08058, pruned_loss=0.0125, audio_tagging_loss=0.01371, over 14349.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09787, pruned_loss=0.01787, audio_tagging_loss=0.009443, over 3044070.31 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:18:13,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1371000.0, ans=0.125 2023-11-21 05:18:13,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1371000.0, ans=0.125 2023-11-21 05:18:15,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1371000.0, ans=0.0 2023-11-21 05:18:29,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1371066.6666666667, ans=0.0 2023-11-21 05:18:43,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1371133.3333333333, ans=0.1 2023-11-21 05:19:13,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1371266.6666666667, ans=0.1 2023-11-21 05:19:16,721 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205700 2023-11-21 05:19:16,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1371266.6666666667, ans=0.125 2023-11-21 05:19:19,128 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1300, loss[loss=0.08097, simple_loss=0.1091, pruned_loss=0.01758, audio_tagging_loss=0.008837, over 15045.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09786, pruned_loss=0.01769, audio_tagging_loss=0.009483, over 3047766.83 frames. ], batch size: 58, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:19:28,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1371333.3333333333, ans=0.2 2023-11-21 05:19:29,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1371333.3333333333, ans=0.1 2023-11-21 05:19:33,945 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.156e+01 8.683e+01 9.710e+01 1.341e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 05:19:45,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.96 vs. limit=15.0 2023-11-21 05:20:04,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1371533.3333333333, ans=0.0 2023-11-21 05:20:20,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1371600.0, ans=0.0 2023-11-21 05:20:21,704 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205750 2023-11-21 05:20:24,020 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1350, loss[loss=0.07054, simple_loss=0.07962, pruned_loss=0.01879, audio_tagging_loss=0.01194, over 14773.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09749, pruned_loss=0.01753, audio_tagging_loss=0.009584, over 3050582.90 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:20:25,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1371666.6666666667, ans=0.0 2023-11-21 05:20:36,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1371733.3333333333, ans=0.07 2023-11-21 05:20:40,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1371733.3333333333, ans=0.125 2023-11-21 05:21:11,037 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 05:21:26,971 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205800 2023-11-21 05:21:29,766 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1400, loss[loss=0.07365, simple_loss=0.09793, pruned_loss=0.01491, audio_tagging_loss=0.009776, over 14623.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09726, pruned_loss=0.01744, audio_tagging_loss=0.009579, over 3054021.32 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:21:41,711 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:21:45,817 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.942e+01 8.095e+01 8.796e+01 9.675e+01 1.234e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 05:21:47,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.62 vs. limit=15.0 2023-11-21 05:22:01,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1372133.3333333333, ans=0.125 2023-11-21 05:22:32,393 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205850 2023-11-21 05:22:35,546 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1450, loss[loss=0.06764, simple_loss=0.07825, pruned_loss=0.01599, audio_tagging_loss=0.01253, over 14223.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09736, pruned_loss=0.01745, audio_tagging_loss=0.009602, over 3045822.85 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:22:38,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1372333.3333333333, ans=0.2 2023-11-21 05:22:38,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1372333.3333333333, ans=0.125 2023-11-21 05:22:48,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.34 vs. limit=12.0 2023-11-21 05:22:49,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1372400.0, ans=0.1 2023-11-21 05:22:54,560 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:23:02,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1372466.6666666667, ans=0.0 2023-11-21 05:23:11,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1372466.6666666667, ans=0.125 2023-11-21 05:23:13,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1372533.3333333333, ans=0.2 2023-11-21 05:23:18,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1372533.3333333333, ans=0.1 2023-11-21 05:23:33,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1372600.0, ans=0.0 2023-11-21 05:23:35,646 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.57 vs. limit=22.5 2023-11-21 05:23:35,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.52 vs. limit=15.0 2023-11-21 05:23:37,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205900 2023-11-21 05:23:40,178 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1500, loss[loss=0.07573, simple_loss=0.08943, pruned_loss=0.02033, audio_tagging_loss=0.01068, over 14798.00 frames. ], tot_loss[loss=0.07608, simple_loss=0.09785, pruned_loss=0.01752, audio_tagging_loss=0.009627, over 3052732.68 frames. ], batch size: 59, lr: 3.86e-03, grad_scale: 32.0 2023-11-21 05:23:50,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1372666.6666666667, ans=0.2 2023-11-21 05:23:51,998 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:23:52,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1372733.3333333333, ans=0.125 2023-11-21 05:23:55,891 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 8.193e+01 8.878e+01 9.448e+01 1.243e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-21 05:24:05,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.47 vs. limit=15.0 2023-11-21 05:24:07,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1372800.0, ans=0.125 2023-11-21 05:24:08,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.37 vs. limit=15.0 2023-11-21 05:24:11,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1372800.0, ans=0.125 2023-11-21 05:24:12,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1372800.0, ans=0.0 2023-11-21 05:24:14,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1372800.0, ans=0.125 2023-11-21 05:24:21,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1372866.6666666667, ans=0.0 2023-11-21 05:24:34,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.43 vs. limit=15.0 2023-11-21 05:24:43,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 205950 2023-11-21 05:24:45,771 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1550, loss[loss=0.06802, simple_loss=0.09738, pruned_loss=0.01049, audio_tagging_loss=0.008834, over 15912.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09692, pruned_loss=0.01735, audio_tagging_loss=0.009748, over 3050996.87 frames. ], batch size: 58, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:24:54,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1373000.0, ans=0.125 2023-11-21 05:25:08,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1373066.6666666667, ans=0.0 2023-11-21 05:25:23,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1373200.0, ans=0.0 2023-11-21 05:25:23,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.09 vs. limit=15.0 2023-11-21 05:25:27,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1373200.0, ans=0.1 2023-11-21 05:25:28,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1373200.0, ans=0.0 2023-11-21 05:25:33,140 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:25:46,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1373266.6666666667, ans=0.0 2023-11-21 05:25:48,344 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206000 2023-11-21 05:25:51,193 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1600, loss[loss=0.04321, simple_loss=0.04821, pruned_loss=0.008823, audio_tagging_loss=0.01028, over 14503.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09745, pruned_loss=0.01758, audio_tagging_loss=0.009791, over 3041114.64 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:26:07,359 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.284e+01 8.965e+01 9.783e+01 1.317e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-21 05:26:12,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1373400.0, ans=0.0 2023-11-21 05:26:12,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1373400.0, ans=0.125 2023-11-21 05:26:37,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=11.03 vs. limit=15.0 2023-11-21 05:26:44,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1373600.0, ans=0.5 2023-11-21 05:26:55,253 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206050 2023-11-21 05:26:57,586 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1650, loss[loss=0.08365, simple_loss=0.115, pruned_loss=0.01573, audio_tagging_loss=0.01043, over 16062.00 frames. ], tot_loss[loss=0.07661, simple_loss=0.09806, pruned_loss=0.01776, audio_tagging_loss=0.009822, over 3044029.48 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:27:06,938 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.13 vs. limit=15.0 2023-11-21 05:27:13,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1373733.3333333333, ans=0.125 2023-11-21 05:27:29,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1373800.0, ans=0.125 2023-11-21 05:27:54,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1373933.3333333333, ans=0.0 2023-11-21 05:28:00,636 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206100 2023-11-21 05:28:03,618 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1700, loss[loss=0.1047, simple_loss=0.1413, pruned_loss=0.02725, audio_tagging_loss=0.006784, over 15895.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.09884, pruned_loss=0.0177, audio_tagging_loss=0.009847, over 3043567.07 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:28:04,394 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.24 vs. limit=15.0 2023-11-21 05:28:08,162 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.97 vs. limit=6.0 2023-11-21 05:28:14,607 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=15.0 2023-11-21 05:28:18,889 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.074e+01 8.560e+01 9.664e+01 1.224e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-21 05:28:53,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1374200.0, ans=0.125 2023-11-21 05:29:06,276 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206150 2023-11-21 05:29:08,604 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1750, loss[loss=0.06542, simple_loss=0.08268, pruned_loss=0.01563, audio_tagging_loss=0.008451, over 14435.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09883, pruned_loss=0.01779, audio_tagging_loss=0.009814, over 3045707.07 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:29:08,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1374333.3333333333, ans=0.0 2023-11-21 05:29:10,778 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.38 vs. limit=6.0 2023-11-21 05:29:15,508 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.43 vs. limit=15.0 2023-11-21 05:29:24,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1374400.0, ans=0.04949747468305833 2023-11-21 05:29:25,766 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.27 vs. limit=15.0 2023-11-21 05:29:30,066 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.08 vs. limit=15.0 2023-11-21 05:29:57,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1374533.3333333333, ans=0.1 2023-11-21 05:29:59,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1374600.0, ans=0.5 2023-11-21 05:30:03,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1374600.0, ans=0.0 2023-11-21 05:30:03,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.50 vs. limit=22.5 2023-11-21 05:30:10,784 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206200 2023-11-21 05:30:13,607 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1800, loss[loss=0.07213, simple_loss=0.0861, pruned_loss=0.01812, audio_tagging_loss=0.01096, over 15762.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09905, pruned_loss=0.01778, audio_tagging_loss=0.009715, over 3050827.14 frames. ], batch size: 60, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:30:23,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1374666.6666666667, ans=0.125 2023-11-21 05:30:29,695 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.350e+01 8.891e+01 9.779e+01 1.328e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-21 05:30:40,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1374800.0, ans=0.125 2023-11-21 05:30:52,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1374866.6666666667, ans=0.1 2023-11-21 05:31:04,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1374933.3333333333, ans=0.125 2023-11-21 05:31:05,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1374933.3333333333, ans=0.0 2023-11-21 05:31:07,361 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.13 vs. limit=15.0 2023-11-21 05:31:12,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1374933.3333333333, ans=0.1 2023-11-21 05:31:16,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206250 2023-11-21 05:31:18,494 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1850, loss[loss=0.09655, simple_loss=0.1215, pruned_loss=0.02383, audio_tagging_loss=0.01196, over 15825.00 frames. ], tot_loss[loss=0.07712, simple_loss=0.0991, pruned_loss=0.01788, audio_tagging_loss=0.009681, over 3055852.47 frames. ], batch size: 58, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:31:22,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1375000.0, ans=0.125 2023-11-21 05:31:31,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1375066.6666666667, ans=0.1 2023-11-21 05:31:53,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1375133.3333333333, ans=0.015 2023-11-21 05:32:22,526 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206300 2023-11-21 05:32:22,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1375266.6666666667, ans=0.1 2023-11-21 05:32:24,923 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1900, loss[loss=0.06106, simple_loss=0.07437, pruned_loss=0.01281, audio_tagging_loss=0.01107, over 15257.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09811, pruned_loss=0.01776, audio_tagging_loss=0.009588, over 3053075.08 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:32:27,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1375333.3333333333, ans=0.125 2023-11-21 05:32:31,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1375333.3333333333, ans=0.125 2023-11-21 05:32:38,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1375400.0, ans=0.125 2023-11-21 05:32:40,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.312e+01 7.949e+01 8.733e+01 9.558e+01 1.328e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 05:32:57,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1375466.6666666667, ans=0.125 2023-11-21 05:33:23,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1375600.0, ans=0.125 2023-11-21 05:33:27,135 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206350 2023-11-21 05:33:29,456 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 1950, loss[loss=0.06506, simple_loss=0.088, pruned_loss=0.009931, audio_tagging_loss=0.01113, over 14492.00 frames. ], tot_loss[loss=0.07673, simple_loss=0.09874, pruned_loss=0.01784, audio_tagging_loss=0.009525, over 3050673.19 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:33:38,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1375666.6666666667, ans=0.2 2023-11-21 05:33:38,548 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0 2023-11-21 05:33:48,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1375733.3333333333, ans=0.125 2023-11-21 05:34:01,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1375800.0, ans=0.0 2023-11-21 05:34:18,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1375866.6666666667, ans=0.0 2023-11-21 05:34:26,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1375933.3333333333, ans=0.2 2023-11-21 05:34:30,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206400 2023-11-21 05:34:34,498 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2000, loss[loss=0.08131, simple_loss=0.1029, pruned_loss=0.02082, audio_tagging_loss=0.009054, over 14905.00 frames. ], tot_loss[loss=0.07678, simple_loss=0.09862, pruned_loss=0.01788, audio_tagging_loss=0.009596, over 3054728.91 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:34:53,484 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.010e+01 8.880e+01 9.739e+01 1.562e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-21 05:34:53,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1376066.6666666667, ans=0.0 2023-11-21 05:34:56,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1376066.6666666667, ans=0.125 2023-11-21 05:35:21,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.46 vs. limit=12.0 2023-11-21 05:35:35,493 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:35:37,847 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206450 2023-11-21 05:35:40,809 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2050, loss[loss=0.07081, simple_loss=0.08354, pruned_loss=0.01585, audio_tagging_loss=0.01319, over 14452.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09703, pruned_loss=0.01755, audio_tagging_loss=0.009587, over 3046084.05 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:35:48,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1376333.3333333333, ans=0.0 2023-11-21 05:36:35,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1376600.0, ans=0.2 2023-11-21 05:36:42,071 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206500 2023-11-21 05:36:44,382 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2100, loss[loss=0.0801, simple_loss=0.1013, pruned_loss=0.02098, audio_tagging_loss=0.008462, over 14882.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.0972, pruned_loss=0.01763, audio_tagging_loss=0.009671, over 3050472.42 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:36:59,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.31 vs. limit=15.0 2023-11-21 05:37:01,375 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 8.008e+01 8.892e+01 9.733e+01 1.759e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-21 05:37:10,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1376800.0, ans=0.1 2023-11-21 05:37:10,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1376800.0, ans=0.95 2023-11-21 05:37:14,239 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.06 vs. limit=15.0 2023-11-21 05:37:18,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1376800.0, ans=0.1 2023-11-21 05:37:19,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1376800.0, ans=0.125 2023-11-21 05:37:24,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1376866.6666666667, ans=0.125 2023-11-21 05:37:44,475 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206550 2023-11-21 05:37:46,820 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2150, loss[loss=0.07577, simple_loss=0.09206, pruned_loss=0.0204, audio_tagging_loss=0.009341, over 15141.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09786, pruned_loss=0.01767, audio_tagging_loss=0.009647, over 3054123.44 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:38:01,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1377066.6666666667, ans=0.0 2023-11-21 05:38:01,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1377066.6666666667, ans=0.04949747468305833 2023-11-21 05:38:07,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1377066.6666666667, ans=0.125 2023-11-21 05:38:12,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1377133.3333333333, ans=0.125 2023-11-21 05:38:13,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1377133.3333333333, ans=0.125 2023-11-21 05:38:26,401 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 05:38:30,181 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:38:33,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1377200.0, ans=0.1 2023-11-21 05:38:38,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1377266.6666666667, ans=0.125 2023-11-21 05:38:48,795 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206600 2023-11-21 05:38:51,618 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2200, loss[loss=0.06326, simple_loss=0.07832, pruned_loss=0.01557, audio_tagging_loss=0.008525, over 15144.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.09859, pruned_loss=0.01773, audio_tagging_loss=0.009514, over 3056831.52 frames. ], batch size: 59, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:38:57,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1377333.3333333333, ans=0.125 2023-11-21 05:39:04,283 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.28 vs. limit=15.0 2023-11-21 05:39:09,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.13 vs. limit=15.0 2023-11-21 05:39:09,737 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.508e+01 7.959e+01 8.463e+01 9.337e+01 1.181e+02, threshold=1.693e+02, percent-clipped=0.0 2023-11-21 05:39:54,657 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206650 2023-11-21 05:39:54,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1377600.0, ans=0.2 2023-11-21 05:39:57,054 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2250, loss[loss=0.09029, simple_loss=0.1112, pruned_loss=0.02217, audio_tagging_loss=0.01253, over 15988.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09863, pruned_loss=0.01796, audio_tagging_loss=0.009649, over 3057092.37 frames. ], batch size: 59, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:39:58,979 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.01 vs. limit=10.0 2023-11-21 05:39:58,984 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.40 vs. limit=6.0 2023-11-21 05:40:09,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1377733.3333333333, ans=0.125 2023-11-21 05:40:10,024 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.56 vs. limit=15.0 2023-11-21 05:40:31,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1377800.0, ans=0.0 2023-11-21 05:40:54,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1377933.3333333333, ans=0.015 2023-11-21 05:40:56,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1377933.3333333333, ans=0.125 2023-11-21 05:40:59,041 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206700 2023-11-21 05:41:01,595 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2300, loss[loss=0.07742, simple_loss=0.105, pruned_loss=0.01576, audio_tagging_loss=0.009168, over 15190.00 frames. ], tot_loss[loss=0.07658, simple_loss=0.0983, pruned_loss=0.01774, audio_tagging_loss=0.009696, over 3049995.22 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:41:14,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1378066.6666666667, ans=0.1 2023-11-21 05:41:20,808 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.169e+01 8.944e+01 9.700e+01 1.275e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-21 05:41:42,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.27 vs. limit=15.0 2023-11-21 05:42:00,183 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 05:42:04,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206750 2023-11-21 05:42:07,018 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2350, loss[loss=0.08859, simple_loss=0.119, pruned_loss=0.02203, audio_tagging_loss=0.007041, over 16031.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09798, pruned_loss=0.01757, audio_tagging_loss=0.009803, over 3053703.15 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:42:19,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1378400.0, ans=0.125 2023-11-21 05:42:19,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1378400.0, ans=0.2 2023-11-21 05:42:27,571 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.19 vs. limit=15.0 2023-11-21 05:42:28,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1378400.0, ans=0.125 2023-11-21 05:42:30,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1378400.0, ans=0.1 2023-11-21 05:42:57,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.22 vs. limit=12.0 2023-11-21 05:43:06,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.90 vs. limit=12.0 2023-11-21 05:43:09,487 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206800 2023-11-21 05:43:10,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.47 vs. limit=22.5 2023-11-21 05:43:12,378 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2400, loss[loss=0.07284, simple_loss=0.09157, pruned_loss=0.01523, audio_tagging_loss=0.01183, over 14775.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.09783, pruned_loss=0.01745, audio_tagging_loss=0.009929, over 3056912.96 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:43:15,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1378666.6666666667, ans=0.0 2023-11-21 05:43:29,495 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.194e+01 8.214e+01 8.717e+01 9.423e+01 1.179e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 05:43:37,849 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.67 vs. limit=15.0 2023-11-21 05:43:41,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1378800.0, ans=0.125 2023-11-21 05:43:42,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1378800.0, ans=0.0 2023-11-21 05:43:47,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1378800.0, ans=0.125 2023-11-21 05:44:13,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206850 2023-11-21 05:44:15,711 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2450, loss[loss=0.05708, simple_loss=0.06948, pruned_loss=0.0132, audio_tagging_loss=0.009139, over 13828.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.0972, pruned_loss=0.0171, audio_tagging_loss=0.009897, over 3047774.70 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:44:26,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1379000.0, ans=0.0 2023-11-21 05:44:37,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1379066.6666666667, ans=0.125 2023-11-21 05:44:44,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1379133.3333333333, ans=0.025 2023-11-21 05:44:57,460 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.53 vs. limit=6.0 2023-11-21 05:45:16,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206900 2023-11-21 05:45:19,799 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2500, loss[loss=0.07305, simple_loss=0.09773, pruned_loss=0.01652, audio_tagging_loss=0.007667, over 15400.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09728, pruned_loss=0.0171, audio_tagging_loss=0.009957, over 3047712.74 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:45:24,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1379333.3333333333, ans=0.125 2023-11-21 05:45:30,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1379333.3333333333, ans=0.125 2023-11-21 05:45:36,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1379400.0, ans=0.0 2023-11-21 05:45:38,427 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.201e+01 8.897e+01 9.730e+01 1.405e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-21 05:45:53,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1379466.6666666667, ans=0.125 2023-11-21 05:46:15,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1379600.0, ans=0.125 2023-11-21 05:46:19,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1379600.0, ans=0.125 2023-11-21 05:46:22,208 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 206950 2023-11-21 05:46:25,215 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2550, loss[loss=0.08147, simple_loss=0.1058, pruned_loss=0.0213, audio_tagging_loss=0.007247, over 15060.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09705, pruned_loss=0.017, audio_tagging_loss=0.009808, over 3048241.16 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 32.0 2023-11-21 05:46:41,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1379733.3333333333, ans=0.125 2023-11-21 05:46:52,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1379800.0, ans=0.04949747468305833 2023-11-21 05:46:54,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1379800.0, ans=0.1 2023-11-21 05:47:02,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1379866.6666666667, ans=0.125 2023-11-21 05:47:04,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1379866.6666666667, ans=0.0 2023-11-21 05:47:20,330 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.59 vs. limit=6.0 2023-11-21 05:47:25,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207000 2023-11-21 05:47:28,465 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2600, loss[loss=0.09398, simple_loss=0.1277, pruned_loss=0.02181, audio_tagging_loss=0.00833, over 14599.00 frames. ], tot_loss[loss=0.07556, simple_loss=0.09748, pruned_loss=0.01719, audio_tagging_loss=0.009624, over 3049823.47 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 16.0 2023-11-21 05:47:30,452 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.22 vs. limit=15.0 2023-11-21 05:47:47,829 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.166e+01 8.159e+01 8.772e+01 9.350e+01 1.184e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 05:47:56,777 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.30 vs. limit=8.0 2023-11-21 05:48:20,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=1380266.6666666667, ans=0.1 2023-11-21 05:48:21,871 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.28 vs. limit=12.0 2023-11-21 05:48:30,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207050 2023-11-21 05:48:32,779 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2650, loss[loss=0.08856, simple_loss=0.1143, pruned_loss=0.0228, audio_tagging_loss=0.008599, over 14528.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09752, pruned_loss=0.01722, audio_tagging_loss=0.009584, over 3054190.26 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:48:47,922 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.11 vs. limit=22.5 2023-11-21 05:48:53,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1380400.0, ans=0.1 2023-11-21 05:49:00,612 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:49:01,047 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.26 vs. limit=15.0 2023-11-21 05:49:16,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1380533.3333333333, ans=0.0 2023-11-21 05:49:34,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207100 2023-11-21 05:49:37,447 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2700, loss[loss=0.05507, simple_loss=0.07142, pruned_loss=0.008801, audio_tagging_loss=0.01055, over 14872.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.0975, pruned_loss=0.01721, audio_tagging_loss=0.009494, over 3046864.75 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 8.0 2023-11-21 05:49:57,496 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 7.943e+01 8.665e+01 9.544e+01 1.191e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 05:50:24,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1380866.6666666667, ans=10.0 2023-11-21 05:50:40,211 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207150 2023-11-21 05:50:42,609 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2750, loss[loss=0.0599, simple_loss=0.07992, pruned_loss=0.01312, audio_tagging_loss=0.006817, over 14929.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09718, pruned_loss=0.01712, audio_tagging_loss=0.009469, over 3051398.41 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 8.0 2023-11-21 05:50:47,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1381000.0, ans=0.125 2023-11-21 05:50:54,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1381066.6666666667, ans=0.125 2023-11-21 05:51:04,694 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.92 vs. limit=10.0 2023-11-21 05:51:20,057 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:51:29,922 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.07 vs. limit=22.5 2023-11-21 05:51:40,454 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 05:51:41,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1381266.6666666667, ans=0.125 2023-11-21 05:51:45,584 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207200 2023-11-21 05:51:46,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.24 vs. limit=15.0 2023-11-21 05:51:48,295 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2800, loss[loss=0.07885, simple_loss=0.1065, pruned_loss=0.0178, audio_tagging_loss=0.007815, over 14189.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09624, pruned_loss=0.01703, audio_tagging_loss=0.009451, over 3046967.58 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:51:54,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1381333.3333333333, ans=0.2 2023-11-21 05:52:09,753 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 7.920e+01 8.497e+01 9.162e+01 1.236e+02, threshold=1.699e+02, percent-clipped=0.0 2023-11-21 05:52:11,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1381400.0, ans=0.125 2023-11-21 05:52:28,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1381533.3333333333, ans=0.1 2023-11-21 05:52:28,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1381533.3333333333, ans=0.125 2023-11-21 05:52:51,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207250 2023-11-21 05:52:54,686 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2850, loss[loss=0.08921, simple_loss=0.1104, pruned_loss=0.02544, audio_tagging_loss=0.008589, over 15406.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.0963, pruned_loss=0.01694, audio_tagging_loss=0.009446, over 3053087.61 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:52:57,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1381666.6666666667, ans=0.125 2023-11-21 05:53:34,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1381866.6666666667, ans=0.125 2023-11-21 05:53:47,746 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.54 vs. limit=15.0 2023-11-21 05:53:57,032 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207300 2023-11-21 05:53:59,454 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2900, loss[loss=0.07731, simple_loss=0.09788, pruned_loss=0.01914, audio_tagging_loss=0.009235, over 15442.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09655, pruned_loss=0.01707, audio_tagging_loss=0.00951, over 3045611.52 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:54:10,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1382000.0, ans=0.2 2023-11-21 05:54:14,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1382066.6666666667, ans=0.125 2023-11-21 05:54:20,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 7.887e+01 8.991e+01 9.698e+01 1.464e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-21 05:54:21,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1382066.6666666667, ans=0.2 2023-11-21 05:54:56,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1382266.6666666667, ans=0.1 2023-11-21 05:54:56,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.86 vs. limit=22.5 2023-11-21 05:55:02,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207350 2023-11-21 05:55:03,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.31 vs. limit=6.0 2023-11-21 05:55:04,465 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 2950, loss[loss=0.106, simple_loss=0.1421, pruned_loss=0.02843, audio_tagging_loss=0.006541, over 15819.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.0971, pruned_loss=0.01728, audio_tagging_loss=0.009594, over 3049204.97 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:55:12,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1382333.3333333333, ans=0.1 2023-11-21 05:55:17,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1382400.0, ans=0.0 2023-11-21 05:55:38,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1382466.6666666667, ans=0.125 2023-11-21 05:55:50,528 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:56:07,540 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207400 2023-11-21 05:56:10,281 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3000, loss[loss=0.07264, simple_loss=0.1023, pruned_loss=0.01242, audio_tagging_loss=0.009045, over 14979.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09832, pruned_loss=0.01735, audio_tagging_loss=0.009533, over 3048873.55 frames. ], batch size: 54, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:56:10,282 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 05:56:45,506 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([1.9702, 3.0065, 3.3714, 3.0091, 3.8155, 3.7885, 3.3514, 3.1871], device='cuda:3') 2023-11-21 05:56:50,263 INFO [train_asr.py:1253] (3/4) Epoch 18, validation: loss=0.06024, simple_loss=0.05252, pruned_loss=0.00529, audio_tagging_loss=0.02869, over 4681554.00 frames. 2023-11-21 05:56:50,264 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 05:57:09,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1382733.3333333333, ans=0.125 2023-11-21 05:57:11,727 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.230e+01 8.309e+01 8.951e+01 9.724e+01 1.389e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-21 05:57:28,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1382866.6666666667, ans=0.125 2023-11-21 05:57:44,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.97 vs. limit=15.0 2023-11-21 05:57:53,467 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207450 2023-11-21 05:57:55,783 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3050, loss[loss=0.05318, simple_loss=0.06108, pruned_loss=0.01029, audio_tagging_loss=0.01235, over 15880.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09753, pruned_loss=0.0173, audio_tagging_loss=0.009608, over 3047007.11 frames. ], batch size: 61, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:58:34,424 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 05:58:39,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1383200.0, ans=0.0 2023-11-21 05:58:51,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1383266.6666666667, ans=0.1 2023-11-21 05:58:57,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1383266.6666666667, ans=0.0 2023-11-21 05:58:58,595 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207500 2023-11-21 05:59:01,644 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3100, loss[loss=0.07655, simple_loss=0.09313, pruned_loss=0.02154, audio_tagging_loss=0.008439, over 15647.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.0982, pruned_loss=0.01751, audio_tagging_loss=0.009693, over 3046256.30 frames. ], batch size: 59, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 05:59:15,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1383400.0, ans=0.125 2023-11-21 05:59:18,997 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 05:59:21,229 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.138e+01 8.768e+01 9.517e+01 1.436e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 05:59:43,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1383533.3333333333, ans=0.1 2023-11-21 05:59:55,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1383600.0, ans=0.1 2023-11-21 05:59:59,747 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.20 vs. limit=8.0 2023-11-21 06:00:03,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207550 2023-11-21 06:00:06,329 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3150, loss[loss=0.0757, simple_loss=0.09237, pruned_loss=0.0196, audio_tagging_loss=0.009906, over 14152.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.09822, pruned_loss=0.01754, audio_tagging_loss=0.009792, over 3042820.22 frames. ], batch size: 54, lr: 3.84e-03, grad_scale: 16.0 2023-11-21 06:00:16,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1383666.6666666667, ans=0.0 2023-11-21 06:00:39,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1383800.0, ans=0.1 2023-11-21 06:00:43,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1383800.0, ans=0.0 2023-11-21 06:00:44,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1383866.6666666667, ans=0.04949747468305833 2023-11-21 06:01:06,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.32 vs. limit=15.0 2023-11-21 06:01:07,821 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207600 2023-11-21 06:01:10,783 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3200, loss[loss=0.07338, simple_loss=0.1002, pruned_loss=0.01378, audio_tagging_loss=0.009478, over 15586.00 frames. ], tot_loss[loss=0.07624, simple_loss=0.098, pruned_loss=0.01744, audio_tagging_loss=0.0098, over 3046601.09 frames. ], batch size: 58, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:01:32,257 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.077e+01 8.904e+01 9.561e+01 1.400e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-21 06:01:54,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1384200.0, ans=0.0 2023-11-21 06:01:55,137 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.18 vs. limit=22.5 2023-11-21 06:01:56,287 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.19 vs. limit=12.0 2023-11-21 06:02:00,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1384200.0, ans=0.1 2023-11-21 06:02:10,501 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.32 vs. limit=12.0 2023-11-21 06:02:13,614 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207650 2023-11-21 06:02:15,885 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3250, loss[loss=0.0852, simple_loss=0.1163, pruned_loss=0.01926, audio_tagging_loss=0.007794, over 16167.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.0975, pruned_loss=0.01714, audio_tagging_loss=0.009899, over 3045494.53 frames. ], batch size: 58, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:02:25,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1384333.3333333333, ans=0.125 2023-11-21 06:02:35,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1384400.0, ans=0.125 2023-11-21 06:02:41,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1384466.6666666667, ans=0.2 2023-11-21 06:02:57,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1384533.3333333333, ans=0.125 2023-11-21 06:03:02,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1384533.3333333333, ans=0.1 2023-11-21 06:03:05,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1384533.3333333333, ans=0.0 2023-11-21 06:03:11,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.51 vs. limit=12.0 2023-11-21 06:03:14,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.62 vs. limit=6.0 2023-11-21 06:03:17,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207700 2023-11-21 06:03:18,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1384600.0, ans=0.1 2023-11-21 06:03:20,339 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3300, loss[loss=0.08535, simple_loss=0.1068, pruned_loss=0.02294, audio_tagging_loss=0.009017, over 17190.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09758, pruned_loss=0.01729, audio_tagging_loss=0.009976, over 3048773.34 frames. ], batch size: 64, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:03:40,719 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.701e+01 8.200e+01 8.829e+01 9.665e+01 1.279e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-21 06:03:50,768 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.34 vs. limit=10.0 2023-11-21 06:03:51,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.00 vs. limit=15.0 2023-11-21 06:04:02,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1384866.6666666667, ans=10.0 2023-11-21 06:04:11,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1384933.3333333333, ans=0.125 2023-11-21 06:04:11,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1384933.3333333333, ans=0.1 2023-11-21 06:04:21,618 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207750 2023-11-21 06:04:24,072 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3350, loss[loss=0.06679, simple_loss=0.08703, pruned_loss=0.01346, audio_tagging_loss=0.009807, over 15592.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09773, pruned_loss=0.01729, audio_tagging_loss=0.009899, over 3052491.65 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:05:10,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1385200.0, ans=10.0 2023-11-21 06:05:18,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1385266.6666666667, ans=0.2 2023-11-21 06:05:27,907 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207800 2023-11-21 06:05:30,667 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3400, loss[loss=0.09416, simple_loss=0.1319, pruned_loss=0.02304, audio_tagging_loss=0.005173, over 16403.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.0984, pruned_loss=0.01739, audio_tagging_loss=0.009713, over 3057197.03 frames. ], batch size: 59, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:05:43,173 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.18 vs. limit=15.0 2023-11-21 06:05:46,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1385400.0, ans=0.1 2023-11-21 06:05:50,869 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 8.344e+01 9.058e+01 9.925e+01 1.172e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-21 06:05:58,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1385466.6666666667, ans=0.0 2023-11-21 06:06:01,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.39 vs. limit=12.0 2023-11-21 06:06:06,709 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:06:24,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1385600.0, ans=0.125 2023-11-21 06:06:30,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1385600.0, ans=0.2 2023-11-21 06:06:33,010 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207850 2023-11-21 06:06:34,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1385666.6666666667, ans=0.125 2023-11-21 06:06:35,417 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3450, loss[loss=0.07982, simple_loss=0.1042, pruned_loss=0.01882, audio_tagging_loss=0.008903, over 15624.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09811, pruned_loss=0.01739, audio_tagging_loss=0.009615, over 3053535.80 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:06:36,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1385666.6666666667, ans=0.1 2023-11-21 06:06:50,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1385733.3333333333, ans=0.95 2023-11-21 06:07:37,346 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207900 2023-11-21 06:07:39,823 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3500, loss[loss=0.06714, simple_loss=0.08834, pruned_loss=0.01522, audio_tagging_loss=0.007752, over 16070.00 frames. ], tot_loss[loss=0.07631, simple_loss=0.09886, pruned_loss=0.01738, audio_tagging_loss=0.009502, over 3051878.44 frames. ], batch size: 59, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:08:02,180 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.378e+01 8.049e+01 8.836e+01 9.881e+01 1.284e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 06:08:07,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1386133.3333333333, ans=0.025 2023-11-21 06:08:14,498 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 06:08:27,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1386200.0, ans=0.2 2023-11-21 06:08:28,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1386200.0, ans=0.125 2023-11-21 06:08:28,883 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.73 vs. limit=22.5 2023-11-21 06:08:38,608 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=15.0 2023-11-21 06:08:43,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 207950 2023-11-21 06:08:45,809 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3550, loss[loss=0.05923, simple_loss=0.06826, pruned_loss=0.0147, audio_tagging_loss=0.01039, over 14833.00 frames. ], tot_loss[loss=0.07561, simple_loss=0.09766, pruned_loss=0.01733, audio_tagging_loss=0.009449, over 3050886.32 frames. ], batch size: 60, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:08:50,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1386333.3333333333, ans=0.125 2023-11-21 06:08:52,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1386333.3333333333, ans=0.0 2023-11-21 06:09:36,310 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.57 vs. limit=15.0 2023-11-21 06:09:43,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.13 vs. limit=15.0 2023-11-21 06:09:49,146 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208000 2023-11-21 06:09:54,991 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3600, loss[loss=0.06926, simple_loss=0.08696, pruned_loss=0.01498, audio_tagging_loss=0.0108, over 14234.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.0962, pruned_loss=0.01681, audio_tagging_loss=0.009412, over 3051481.61 frames. ], batch size: 52, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:09:55,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1386666.6666666667, ans=0.125 2023-11-21 06:10:12,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1386733.3333333333, ans=0.0 2023-11-21 06:10:14,771 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.348e+01 7.918e+01 8.568e+01 9.474e+01 1.185e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 06:10:56,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208050 2023-11-21 06:10:58,457 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3650, loss[loss=0.0934, simple_loss=0.1281, pruned_loss=0.0214, audio_tagging_loss=0.007946, over 16040.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09572, pruned_loss=0.01672, audio_tagging_loss=0.009487, over 3046594.98 frames. ], batch size: 58, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:11:06,625 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:11:32,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.40 vs. limit=22.5 2023-11-21 06:11:33,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1387133.3333333333, ans=0.0 2023-11-21 06:11:38,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1387200.0, ans=0.125 2023-11-21 06:11:54,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1387266.6666666667, ans=0.1 2023-11-21 06:12:01,532 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208100 2023-11-21 06:12:03,993 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3700, loss[loss=0.0751, simple_loss=0.09431, pruned_loss=0.01861, audio_tagging_loss=0.009337, over 14735.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.0976, pruned_loss=0.01693, audio_tagging_loss=0.009366, over 3050154.75 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 32.0 2023-11-21 06:12:04,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1387333.3333333333, ans=0.125 2023-11-21 06:12:06,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1387333.3333333333, ans=0.0 2023-11-21 06:12:11,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.87 vs. limit=22.5 2023-11-21 06:12:25,064 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.253e+01 8.797e+01 9.684e+01 1.136e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 06:12:32,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1387466.6666666667, ans=0.2 2023-11-21 06:12:39,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1387466.6666666667, ans=0.0 2023-11-21 06:12:47,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1387533.3333333333, ans=0.1 2023-11-21 06:12:51,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1387533.3333333333, ans=0.125 2023-11-21 06:12:57,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1387600.0, ans=0.2 2023-11-21 06:13:07,352 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208150 2023-11-21 06:13:09,753 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3750, loss[loss=0.06803, simple_loss=0.08913, pruned_loss=0.01611, audio_tagging_loss=0.007355, over 15413.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09841, pruned_loss=0.01708, audio_tagging_loss=0.009415, over 3057736.58 frames. ], batch size: 58, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:13:10,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1387666.6666666667, ans=0.125 2023-11-21 06:13:36,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1387800.0, ans=0.125 2023-11-21 06:13:55,205 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 06:14:11,925 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208200 2023-11-21 06:14:14,591 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3800, loss[loss=0.08954, simple_loss=0.1127, pruned_loss=0.02205, audio_tagging_loss=0.01116, over 14754.00 frames. ], tot_loss[loss=0.07515, simple_loss=0.09743, pruned_loss=0.01696, audio_tagging_loss=0.009475, over 3050028.27 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:14:14,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1388000.0, ans=0.125 2023-11-21 06:14:17,898 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.88 vs. limit=6.0 2023-11-21 06:14:23,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1388000.0, ans=0.05 2023-11-21 06:14:34,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1388066.6666666667, ans=0.0 2023-11-21 06:14:35,463 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.099e+01 8.755e+01 9.581e+01 1.226e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 06:14:49,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1388133.3333333333, ans=0.0 2023-11-21 06:14:52,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1388200.0, ans=0.125 2023-11-21 06:14:53,224 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:15:05,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1388266.6666666667, ans=0.07 2023-11-21 06:15:16,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208250 2023-11-21 06:15:19,765 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3850, loss[loss=0.09193, simple_loss=0.1118, pruned_loss=0.02287, audio_tagging_loss=0.01316, over 14302.00 frames. ], tot_loss[loss=0.07515, simple_loss=0.09704, pruned_loss=0.01697, audio_tagging_loss=0.009664, over 3050094.59 frames. ], batch size: 54, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:15:31,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1388400.0, ans=0.0 2023-11-21 06:15:33,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1388400.0, ans=0.125 2023-11-21 06:15:34,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0 2023-11-21 06:15:53,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1388466.6666666667, ans=0.2 2023-11-21 06:16:05,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1388533.3333333333, ans=0.0 2023-11-21 06:16:22,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208300 2023-11-21 06:16:24,947 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3900, loss[loss=0.07306, simple_loss=0.09917, pruned_loss=0.01486, audio_tagging_loss=0.008606, over 14523.00 frames. ], tot_loss[loss=0.07531, simple_loss=0.09697, pruned_loss=0.01709, audio_tagging_loss=0.009732, over 3042785.45 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:16:44,539 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.39 vs. limit=15.0 2023-11-21 06:16:46,307 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.288e+01 8.904e+01 9.716e+01 1.197e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-21 06:17:19,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1388933.3333333333, ans=0.125 2023-11-21 06:17:27,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208350 2023-11-21 06:17:27,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1388933.3333333333, ans=0.125 2023-11-21 06:17:29,818 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 3950, loss[loss=0.0655, simple_loss=0.07795, pruned_loss=0.01619, audio_tagging_loss=0.01033, over 14564.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09685, pruned_loss=0.01719, audio_tagging_loss=0.009943, over 3048555.79 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:17:30,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1389000.0, ans=0.0 2023-11-21 06:17:30,527 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0 2023-11-21 06:17:51,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1389066.6666666667, ans=0.1 2023-11-21 06:17:52,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.77 vs. limit=10.0 2023-11-21 06:18:26,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1389266.6666666667, ans=0.1 2023-11-21 06:18:31,479 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208400 2023-11-21 06:18:34,777 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4000, loss[loss=0.08528, simple_loss=0.1104, pruned_loss=0.0201, audio_tagging_loss=0.00996, over 14985.00 frames. ], tot_loss[loss=0.07607, simple_loss=0.09728, pruned_loss=0.01747, audio_tagging_loss=0.009957, over 3045839.23 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:18:56,796 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.265e+01 8.108e+01 8.819e+01 9.792e+01 1.460e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-21 06:18:57,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1389400.0, ans=0.125 2023-11-21 06:18:57,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0 2023-11-21 06:19:04,484 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.13 vs. limit=10.0 2023-11-21 06:19:05,532 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.73 vs. limit=15.0 2023-11-21 06:19:13,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1389533.3333333333, ans=0.0 2023-11-21 06:19:25,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1389600.0, ans=0.1 2023-11-21 06:19:37,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208450 2023-11-21 06:19:40,000 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4050, loss[loss=0.06765, simple_loss=0.08641, pruned_loss=0.01516, audio_tagging_loss=0.009288, over 14219.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09826, pruned_loss=0.01765, audio_tagging_loss=0.009896, over 3046240.82 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:19:41,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.83 vs. limit=22.5 2023-11-21 06:19:43,792 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 06:19:46,661 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.09 vs. limit=22.5 2023-11-21 06:19:55,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1389733.3333333333, ans=0.125 2023-11-21 06:20:07,644 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:20:08,015 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.80 vs. limit=15.0 2023-11-21 06:20:23,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1389866.6666666667, ans=0.1 2023-11-21 06:20:39,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1389933.3333333333, ans=0.125 2023-11-21 06:20:39,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1389933.3333333333, ans=0.95 2023-11-21 06:20:41,785 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208500 2023-11-21 06:20:43,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1390000.0, ans=0.125 2023-11-21 06:20:44,109 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4100, loss[loss=0.0338, simple_loss=0.03689, pruned_loss=0.00372, audio_tagging_loss=0.01163, over 14713.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09877, pruned_loss=0.01783, audio_tagging_loss=0.009865, over 3039544.11 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:21:08,097 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.154e+01 8.779e+01 9.471e+01 2.037e+02, threshold=1.756e+02, percent-clipped=1.0 2023-11-21 06:21:10,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1390133.3333333333, ans=0.2 2023-11-21 06:21:46,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208550 2023-11-21 06:21:49,126 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4150, loss[loss=0.05019, simple_loss=0.05723, pruned_loss=0.01004, audio_tagging_loss=0.01154, over 15598.00 frames. ], tot_loss[loss=0.07683, simple_loss=0.0983, pruned_loss=0.01783, audio_tagging_loss=0.009847, over 3039891.09 frames. ], batch size: 61, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:21:54,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1390333.3333333333, ans=0.125 2023-11-21 06:22:27,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1390533.3333333333, ans=0.1 2023-11-21 06:22:31,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1390533.3333333333, ans=0.125 2023-11-21 06:22:37,662 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 06:22:47,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1390600.0, ans=0.125 2023-11-21 06:22:52,579 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208600 2023-11-21 06:22:52,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1390600.0, ans=0.125 2023-11-21 06:22:55,643 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4200, loss[loss=0.08606, simple_loss=0.1024, pruned_loss=0.02494, audio_tagging_loss=0.009947, over 14474.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09796, pruned_loss=0.0178, audio_tagging_loss=0.009724, over 3039345.48 frames. ], batch size: 58, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:23:03,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1390666.6666666667, ans=0.125 2023-11-21 06:23:17,911 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 7.993e+01 8.905e+01 1.014e+02 1.723e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-21 06:23:19,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1390800.0, ans=0.0 2023-11-21 06:23:29,701 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.74 vs. limit=15.0 2023-11-21 06:23:33,175 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.58 vs. limit=22.5 2023-11-21 06:23:40,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1390866.6666666667, ans=0.125 2023-11-21 06:23:46,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1390933.3333333333, ans=0.125 2023-11-21 06:23:51,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0 2023-11-21 06:23:52,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1390933.3333333333, ans=0.0 2023-11-21 06:23:57,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208650 2023-11-21 06:23:59,945 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4250, loss[loss=0.09268, simple_loss=0.1274, pruned_loss=0.02111, audio_tagging_loss=0.007867, over 15884.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09789, pruned_loss=0.01777, audio_tagging_loss=0.009693, over 3040062.28 frames. ], batch size: 58, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:24:04,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1391000.0, ans=0.125 2023-11-21 06:24:12,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1391066.6666666667, ans=0.125 2023-11-21 06:24:29,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1391133.3333333333, ans=0.125 2023-11-21 06:24:34,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1391133.3333333333, ans=0.125 2023-11-21 06:24:36,931 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.77 vs. limit=22.5 2023-11-21 06:24:47,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1391200.0, ans=0.2 2023-11-21 06:24:50,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1391200.0, ans=0.2 2023-11-21 06:25:02,586 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208700 2023-11-21 06:25:05,629 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4300, loss[loss=0.08979, simple_loss=0.1156, pruned_loss=0.02091, audio_tagging_loss=0.01109, over 17289.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09895, pruned_loss=0.01792, audio_tagging_loss=0.009615, over 3053469.26 frames. ], batch size: 66, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:25:21,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1391400.0, ans=0.2 2023-11-21 06:25:30,098 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.185e+01 9.001e+01 9.859e+01 1.198e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-21 06:25:55,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1391533.3333333333, ans=0.0 2023-11-21 06:26:09,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208750 2023-11-21 06:26:12,088 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4350, loss[loss=0.07538, simple_loss=0.09577, pruned_loss=0.016, audio_tagging_loss=0.0115, over 15520.00 frames. ], tot_loss[loss=0.07688, simple_loss=0.0987, pruned_loss=0.01791, audio_tagging_loss=0.009622, over 3047765.34 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:26:36,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1391800.0, ans=0.1 2023-11-21 06:26:46,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1391800.0, ans=0.2 2023-11-21 06:27:14,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208800 2023-11-21 06:27:16,876 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4400, loss[loss=0.0664, simple_loss=0.09145, pruned_loss=0.01344, audio_tagging_loss=0.007238, over 14948.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09923, pruned_loss=0.0179, audio_tagging_loss=0.009524, over 3047527.56 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:27:18,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1392000.0, ans=0.1 2023-11-21 06:27:40,618 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.458e+01 8.121e+01 8.675e+01 9.482e+01 1.551e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 06:27:52,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1392133.3333333333, ans=0.125 2023-11-21 06:27:58,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1392200.0, ans=0.125 2023-11-21 06:28:09,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1392266.6666666667, ans=0.125 2023-11-21 06:28:18,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208850 2023-11-21 06:28:21,399 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4450, loss[loss=0.09205, simple_loss=0.1192, pruned_loss=0.02115, audio_tagging_loss=0.01131, over 14481.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.09932, pruned_loss=0.01808, audio_tagging_loss=0.00949, over 3046128.93 frames. ], batch size: 52, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:28:25,834 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.68 vs. limit=15.0 2023-11-21 06:28:54,030 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.13 vs. limit=15.0 2023-11-21 06:29:05,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1392533.3333333333, ans=0.2 2023-11-21 06:29:12,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1392600.0, ans=0.95 2023-11-21 06:29:23,784 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208900 2023-11-21 06:29:25,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1392666.6666666667, ans=0.0 2023-11-21 06:29:26,811 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4500, loss[loss=0.07304, simple_loss=0.09127, pruned_loss=0.01668, audio_tagging_loss=0.01072, over 15608.00 frames. ], tot_loss[loss=0.07686, simple_loss=0.099, pruned_loss=0.01793, audio_tagging_loss=0.009428, over 3047690.17 frames. ], batch size: 60, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:29:49,969 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.149e+01 8.678e+01 9.442e+01 1.327e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-21 06:29:51,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1392800.0, ans=0.125 2023-11-21 06:29:54,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1392800.0, ans=0.1 2023-11-21 06:30:03,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1392866.6666666667, ans=0.0 2023-11-21 06:30:04,062 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.60 vs. limit=15.0 2023-11-21 06:30:06,201 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.93 vs. limit=22.5 2023-11-21 06:30:08,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1392866.6666666667, ans=0.0 2023-11-21 06:30:20,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1392933.3333333333, ans=0.0 2023-11-21 06:30:29,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 208950 2023-11-21 06:30:32,272 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4550, loss[loss=0.08257, simple_loss=0.1008, pruned_loss=0.02202, audio_tagging_loss=0.01015, over 14920.00 frames. ], tot_loss[loss=0.07689, simple_loss=0.09922, pruned_loss=0.01791, audio_tagging_loss=0.009371, over 3043324.80 frames. ], batch size: 54, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:30:32,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1393000.0, ans=0.0 2023-11-21 06:31:15,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1393200.0, ans=0.0 2023-11-21 06:31:20,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1393200.0, ans=0.125 2023-11-21 06:31:23,159 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 06:31:34,222 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209000 2023-11-21 06:31:36,994 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4600, loss[loss=0.09568, simple_loss=0.1112, pruned_loss=0.02921, audio_tagging_loss=0.01088, over 14655.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09913, pruned_loss=0.01803, audio_tagging_loss=0.009491, over 3037933.10 frames. ], batch size: 53, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:31:44,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1393333.3333333333, ans=0.125 2023-11-21 06:31:48,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1393333.3333333333, ans=0.1 2023-11-21 06:32:01,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.053e+01 8.667e+01 9.436e+01 2.145e+02, threshold=1.733e+02, percent-clipped=1.0 2023-11-21 06:32:01,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1393400.0, ans=0.2 2023-11-21 06:32:30,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1393600.0, ans=0.125 2023-11-21 06:32:33,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.34 vs. limit=6.0 2023-11-21 06:32:39,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209050 2023-11-21 06:32:41,484 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4650, loss[loss=0.07409, simple_loss=0.09208, pruned_loss=0.01515, audio_tagging_loss=0.01289, over 15660.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.0989, pruned_loss=0.01795, audio_tagging_loss=0.009568, over 3043115.85 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:32:42,095 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.24 vs. limit=15.0 2023-11-21 06:32:47,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1393666.6666666667, ans=0.09899494936611666 2023-11-21 06:33:05,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1393733.3333333333, ans=0.125 2023-11-21 06:33:07,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1393800.0, ans=0.0 2023-11-21 06:33:14,480 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:33:23,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1393866.6666666667, ans=0.2 2023-11-21 06:33:44,829 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209100 2023-11-21 06:33:47,225 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4700, loss[loss=0.1008, simple_loss=0.1284, pruned_loss=0.02672, audio_tagging_loss=0.009865, over 15243.00 frames. ], tot_loss[loss=0.07676, simple_loss=0.09848, pruned_loss=0.01777, audio_tagging_loss=0.009747, over 3052902.86 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 32.0 2023-11-21 06:33:48,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1394000.0, ans=0.09899494936611666 2023-11-21 06:33:53,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.22 vs. limit=10.0 2023-11-21 06:34:03,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1394066.6666666667, ans=0.125 2023-11-21 06:34:08,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1394066.6666666667, ans=0.0 2023-11-21 06:34:10,863 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.149e+01 8.884e+01 9.885e+01 2.074e+02, threshold=1.777e+02, percent-clipped=1.0 2023-11-21 06:34:15,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1394133.3333333333, ans=0.0 2023-11-21 06:34:27,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1394200.0, ans=0.125 2023-11-21 06:34:28,450 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.44 vs. limit=15.0 2023-11-21 06:34:34,620 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.99 vs. limit=15.0 2023-11-21 06:34:48,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209150 2023-11-21 06:34:51,272 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4750, loss[loss=0.06765, simple_loss=0.08741, pruned_loss=0.01333, audio_tagging_loss=0.01062, over 14836.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09761, pruned_loss=0.01736, audio_tagging_loss=0.009855, over 3050069.44 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 16.0 2023-11-21 06:35:23,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1394466.6666666667, ans=0.0 2023-11-21 06:35:54,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209200 2023-11-21 06:35:57,081 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4800, loss[loss=0.06508, simple_loss=0.08347, pruned_loss=0.01479, audio_tagging_loss=0.008557, over 15168.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09776, pruned_loss=0.0173, audio_tagging_loss=0.009847, over 3052840.22 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:36:01,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1394666.6666666667, ans=0.0 2023-11-21 06:36:07,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1394666.6666666667, ans=0.2 2023-11-21 06:36:24,236 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.238e+01 8.873e+01 9.665e+01 1.246e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-21 06:36:48,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1394933.3333333333, ans=0.2 2023-11-21 06:36:55,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1394933.3333333333, ans=0.125 2023-11-21 06:36:56,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1394933.3333333333, ans=0.125 2023-11-21 06:37:01,188 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209250 2023-11-21 06:37:04,145 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4850, loss[loss=0.07006, simple_loss=0.09012, pruned_loss=0.0152, audio_tagging_loss=0.009794, over 15023.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09675, pruned_loss=0.01715, audio_tagging_loss=0.009937, over 3042441.58 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:37:11,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1395000.0, ans=0.1 2023-11-21 06:37:20,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.99 vs. limit=15.0 2023-11-21 06:37:25,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1395066.6666666667, ans=0.0 2023-11-21 06:37:26,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.49 vs. limit=22.5 2023-11-21 06:37:30,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1395133.3333333333, ans=0.2 2023-11-21 06:37:38,749 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.70 vs. limit=15.0 2023-11-21 06:37:46,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.68 vs. limit=15.0 2023-11-21 06:37:54,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1395200.0, ans=0.0 2023-11-21 06:38:06,302 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209300 2023-11-21 06:38:08,798 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4900, loss[loss=0.1065, simple_loss=0.1279, pruned_loss=0.0344, audio_tagging_loss=0.008177, over 16282.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09661, pruned_loss=0.01717, audio_tagging_loss=0.009948, over 3039674.62 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:38:19,120 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.36 vs. limit=22.5 2023-11-21 06:38:26,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1395400.0, ans=0.0 2023-11-21 06:38:32,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1395400.0, ans=0.125 2023-11-21 06:38:34,567 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.171e+01 8.744e+01 9.651e+01 1.565e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-21 06:38:36,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1395466.6666666667, ans=0.09899494936611666 2023-11-21 06:38:47,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1395533.3333333333, ans=0.125 2023-11-21 06:38:57,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1395533.3333333333, ans=0.125 2023-11-21 06:39:10,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209350 2023-11-21 06:39:13,600 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 4950, loss[loss=0.08117, simple_loss=0.11, pruned_loss=0.0158, audio_tagging_loss=0.01037, over 16177.00 frames. ], tot_loss[loss=0.07577, simple_loss=0.09754, pruned_loss=0.01735, audio_tagging_loss=0.009643, over 3041810.55 frames. ], batch size: 60, lr: 3.82e-03, grad_scale: 8.0 2023-11-21 06:40:05,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1395933.3333333333, ans=0.125 2023-11-21 06:40:17,536 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209400 2023-11-21 06:40:20,297 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5000, loss[loss=0.07739, simple_loss=0.09683, pruned_loss=0.01871, audio_tagging_loss=0.01027, over 13969.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09733, pruned_loss=0.01726, audio_tagging_loss=0.009551, over 3041898.52 frames. ], batch size: 54, lr: 3.82e-03, grad_scale: 8.0 2023-11-21 06:40:43,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1396066.6666666667, ans=0.2 2023-11-21 06:40:47,082 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.108e+01 8.729e+01 9.428e+01 1.080e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 06:40:50,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1396133.3333333333, ans=0.2 2023-11-21 06:40:50,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1396133.3333333333, ans=0.125 2023-11-21 06:40:57,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1396200.0, ans=0.1 2023-11-21 06:41:04,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1396200.0, ans=0.125 2023-11-21 06:41:23,022 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209450 2023-11-21 06:41:25,443 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5050, loss[loss=0.06286, simple_loss=0.07752, pruned_loss=0.01285, audio_tagging_loss=0.01125, over 15447.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.0971, pruned_loss=0.01707, audio_tagging_loss=0.009486, over 3039150.25 frames. ], batch size: 63, lr: 3.82e-03, grad_scale: 8.0 2023-11-21 06:41:29,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1396333.3333333333, ans=0.0 2023-11-21 06:41:43,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1396400.0, ans=0.1 2023-11-21 06:42:11,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1396533.3333333333, ans=0.125 2023-11-21 06:42:14,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1396533.3333333333, ans=0.125 2023-11-21 06:42:17,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1396600.0, ans=0.0 2023-11-21 06:42:26,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.12 vs. limit=15.0 2023-11-21 06:42:27,035 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209500 2023-11-21 06:42:29,381 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5100, loss[loss=0.0562, simple_loss=0.06739, pruned_loss=0.01098, audio_tagging_loss=0.01153, over 14397.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09666, pruned_loss=0.01704, audio_tagging_loss=0.009486, over 3053867.36 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 8.0 2023-11-21 06:42:56,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1396800.0, ans=0.0 2023-11-21 06:42:57,150 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.138e+01 8.643e+01 9.208e+01 2.160e+02, threshold=1.729e+02, percent-clipped=1.0 2023-11-21 06:43:02,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1396800.0, ans=0.1 2023-11-21 06:43:05,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1396800.0, ans=0.1 2023-11-21 06:43:07,860 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:43:07,944 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:43:32,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209550 2023-11-21 06:43:35,667 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5150, loss[loss=0.06804, simple_loss=0.0848, pruned_loss=0.01501, audio_tagging_loss=0.01063, over 15712.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09668, pruned_loss=0.01706, audio_tagging_loss=0.00954, over 3045052.10 frames. ], batch size: 61, lr: 3.82e-03, grad_scale: 8.0 2023-11-21 06:43:35,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1397000.0, ans=0.125 2023-11-21 06:43:38,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1397000.0, ans=0.0 2023-11-21 06:43:42,808 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.65 vs. limit=15.0 2023-11-21 06:43:44,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1397000.0, ans=0.025 2023-11-21 06:43:52,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1397066.6666666667, ans=0.1 2023-11-21 06:44:37,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209600 2023-11-21 06:44:39,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1397333.3333333333, ans=0.125 2023-11-21 06:44:40,766 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5200, loss[loss=0.08314, simple_loss=0.1107, pruned_loss=0.01883, audio_tagging_loss=0.008941, over 15482.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.09694, pruned_loss=0.01711, audio_tagging_loss=0.009627, over 3036450.98 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:45:01,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1397400.0, ans=0.0 2023-11-21 06:45:02,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.27 vs. limit=22.5 2023-11-21 06:45:07,617 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.125e+01 8.190e+01 8.762e+01 9.315e+01 1.275e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-21 06:45:15,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1397466.6666666667, ans=0.1 2023-11-21 06:45:18,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1397466.6666666667, ans=0.1 2023-11-21 06:45:18,215 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.35 vs. limit=15.0 2023-11-21 06:45:19,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1397533.3333333333, ans=0.1 2023-11-21 06:45:42,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209650 2023-11-21 06:45:44,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1397666.6666666667, ans=0.07 2023-11-21 06:45:45,302 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5250, loss[loss=0.09298, simple_loss=0.1276, pruned_loss=0.02127, audio_tagging_loss=0.007883, over 15468.00 frames. ], tot_loss[loss=0.07552, simple_loss=0.09745, pruned_loss=0.01726, audio_tagging_loss=0.009536, over 3040257.06 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:45:52,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1397666.6666666667, ans=0.09899494936611666 2023-11-21 06:45:55,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1397666.6666666667, ans=0.0 2023-11-21 06:45:59,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1397733.3333333333, ans=0.0 2023-11-21 06:46:14,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1397800.0, ans=0.125 2023-11-21 06:46:15,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.92 vs. limit=22.5 2023-11-21 06:46:39,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1397933.3333333333, ans=0.1 2023-11-21 06:46:39,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1397933.3333333333, ans=0.1 2023-11-21 06:46:39,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.71 vs. limit=22.5 2023-11-21 06:46:47,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209700 2023-11-21 06:46:49,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1398000.0, ans=0.125 2023-11-21 06:46:50,775 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5300, loss[loss=0.05803, simple_loss=0.07134, pruned_loss=0.01075, audio_tagging_loss=0.0116, over 15044.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.0977, pruned_loss=0.01729, audio_tagging_loss=0.009485, over 3041560.81 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:46:56,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1398000.0, ans=0.1 2023-11-21 06:47:10,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1398066.6666666667, ans=0.125 2023-11-21 06:47:12,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1398066.6666666667, ans=0.0 2023-11-21 06:47:17,467 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.345e+01 8.152e+01 8.842e+01 9.695e+01 1.214e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 06:47:18,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.89 vs. limit=12.0 2023-11-21 06:47:39,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1398200.0, ans=0.0 2023-11-21 06:47:46,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1398266.6666666667, ans=0.125 2023-11-21 06:47:52,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209750 2023-11-21 06:47:55,000 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5350, loss[loss=0.07826, simple_loss=0.1025, pruned_loss=0.02035, audio_tagging_loss=0.006657, over 14654.00 frames. ], tot_loss[loss=0.07556, simple_loss=0.09741, pruned_loss=0.01731, audio_tagging_loss=0.009547, over 3040703.13 frames. ], batch size: 54, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:48:12,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1398400.0, ans=0.125 2023-11-21 06:48:14,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1398400.0, ans=0.125 2023-11-21 06:48:14,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1398400.0, ans=0.125 2023-11-21 06:48:15,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.29 vs. limit=22.5 2023-11-21 06:48:29,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1398466.6666666667, ans=0.125 2023-11-21 06:48:38,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1398533.3333333333, ans=0.125 2023-11-21 06:48:57,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209800 2023-11-21 06:48:58,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1398600.0, ans=0.04949747468305833 2023-11-21 06:49:00,611 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5400, loss[loss=0.09364, simple_loss=0.1228, pruned_loss=0.02086, audio_tagging_loss=0.0114, over 14891.00 frames. ], tot_loss[loss=0.07552, simple_loss=0.09746, pruned_loss=0.0172, audio_tagging_loss=0.009588, over 3045189.94 frames. ], batch size: 55, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:49:14,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.49 vs. limit=15.0 2023-11-21 06:49:22,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.00 vs. limit=15.0 2023-11-21 06:49:28,200 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.049e+01 8.809e+01 9.277e+01 1.117e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-21 06:49:55,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1398933.3333333333, ans=0.0 2023-11-21 06:49:58,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0 2023-11-21 06:50:02,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209850 2023-11-21 06:50:04,909 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5450, loss[loss=0.07873, simple_loss=0.09686, pruned_loss=0.01826, audio_tagging_loss=0.01204, over 16174.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09806, pruned_loss=0.01741, audio_tagging_loss=0.009565, over 3044395.59 frames. ], batch size: 62, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:50:21,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1399066.6666666667, ans=0.0 2023-11-21 06:50:37,873 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.89 vs. limit=15.0 2023-11-21 06:51:05,832 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:51:08,080 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209900 2023-11-21 06:51:10,482 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5500, loss[loss=0.07558, simple_loss=0.08488, pruned_loss=0.01971, audio_tagging_loss=0.01344, over 14260.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09817, pruned_loss=0.01741, audio_tagging_loss=0.009641, over 3043588.29 frames. ], batch size: 53, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:51:21,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1399400.0, ans=0.0 2023-11-21 06:51:37,451 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 7.977e+01 8.798e+01 9.625e+01 1.160e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-21 06:51:37,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1399466.6666666667, ans=0.04949747468305833 2023-11-21 06:51:49,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1399533.3333333333, ans=0.125 2023-11-21 06:51:49,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1399533.3333333333, ans=0.125 2023-11-21 06:51:55,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.01 vs. limit=15.0 2023-11-21 06:52:04,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1399600.0, ans=0.125 2023-11-21 06:52:08,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1399600.0, ans=0.125 2023-11-21 06:52:11,943 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 209950 2023-11-21 06:52:14,292 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5550, loss[loss=0.06768, simple_loss=0.06998, pruned_loss=0.01628, audio_tagging_loss=0.01642, over 14449.00 frames. ], tot_loss[loss=0.07636, simple_loss=0.09835, pruned_loss=0.01751, audio_tagging_loss=0.009677, over 3047215.69 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 16.0 2023-11-21 06:52:14,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1399666.6666666667, ans=0.0 2023-11-21 06:52:18,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1399666.6666666667, ans=0.125 2023-11-21 06:52:29,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1399733.3333333333, ans=0.1 2023-11-21 06:52:46,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.79 vs. limit=22.5 2023-11-21 06:52:48,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1399800.0, ans=0.2 2023-11-21 06:53:08,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.40 vs. limit=22.5 2023-11-21 06:53:10,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1399933.3333333333, ans=0.125 2023-11-21 06:53:17,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210000 2023-11-21 06:53:20,257 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5600, loss[loss=0.06334, simple_loss=0.08061, pruned_loss=0.01168, audio_tagging_loss=0.01135, over 14130.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09756, pruned_loss=0.01726, audio_tagging_loss=0.009832, over 3050271.52 frames. ], batch size: 54, lr: 3.82e-03, grad_scale: 32.0 2023-11-21 06:53:23,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1400000.0, ans=0.125 2023-11-21 06:53:45,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1400133.3333333333, ans=0.125 2023-11-21 06:53:47,686 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.103e+01 7.991e+01 8.686e+01 9.530e+01 1.132e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 06:53:51,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.73 vs. limit=12.0 2023-11-21 06:53:59,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1400200.0, ans=0.2 2023-11-21 06:54:07,365 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 06:54:23,408 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210050 2023-11-21 06:54:25,748 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5650, loss[loss=0.07288, simple_loss=0.09518, pruned_loss=0.01596, audio_tagging_loss=0.009332, over 16508.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09686, pruned_loss=0.01713, audio_tagging_loss=0.009968, over 3048198.54 frames. ], batch size: 62, lr: 3.82e-03, grad_scale: 32.0 2023-11-21 06:54:33,343 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 06:54:36,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1400400.0, ans=0.125 2023-11-21 06:54:40,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1400400.0, ans=0.0 2023-11-21 06:54:58,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1400466.6666666667, ans=0.0 2023-11-21 06:54:58,535 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=9.613e-02 2023-11-21 06:54:58,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1400466.6666666667, ans=0.2 2023-11-21 06:55:19,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.52 vs. limit=15.0 2023-11-21 06:55:22,341 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.20 vs. limit=8.0 2023-11-21 06:55:24,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1400600.0, ans=0.0 2023-11-21 06:55:25,602 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.39 vs. limit=15.0 2023-11-21 06:55:26,363 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210100 2023-11-21 06:55:28,800 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5700, loss[loss=0.06094, simple_loss=0.06518, pruned_loss=0.01529, audio_tagging_loss=0.01306, over 14900.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09697, pruned_loss=0.01726, audio_tagging_loss=0.009987, over 3046021.82 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 32.0 2023-11-21 06:55:42,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=11.31 vs. limit=12.0 2023-11-21 06:55:45,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1400733.3333333333, ans=0.5 2023-11-21 06:55:46,732 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.54 vs. limit=15.0 2023-11-21 06:55:47,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1400733.3333333333, ans=0.125 2023-11-21 06:55:48,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1400733.3333333333, ans=0.125 2023-11-21 06:55:57,065 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.246e+01 8.120e+01 8.868e+01 9.513e+01 1.273e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-21 06:56:03,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1400800.0, ans=0.125 2023-11-21 06:56:06,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1400800.0, ans=0.2 2023-11-21 06:56:08,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1400866.6666666667, ans=0.125 2023-11-21 06:56:22,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0 2023-11-21 06:56:28,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1400933.3333333333, ans=0.125 2023-11-21 06:56:30,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210150 2023-11-21 06:56:33,734 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5750, loss[loss=0.07589, simple_loss=0.09895, pruned_loss=0.01826, audio_tagging_loss=0.008149, over 16329.00 frames. ], tot_loss[loss=0.07576, simple_loss=0.0971, pruned_loss=0.01736, audio_tagging_loss=0.009851, over 3051578.79 frames. ], batch size: 61, lr: 3.82e-03, grad_scale: 32.0 2023-11-21 06:56:41,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1401000.0, ans=0.125 2023-11-21 06:57:05,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.40 vs. limit=22.5 2023-11-21 06:57:19,039 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.40 vs. limit=22.5 2023-11-21 06:57:30,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1401266.6666666667, ans=0.05 2023-11-21 06:57:32,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1401266.6666666667, ans=0.1 2023-11-21 06:57:36,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210200 2023-11-21 06:57:39,679 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5800, loss[loss=0.07264, simple_loss=0.0903, pruned_loss=0.01587, audio_tagging_loss=0.01162, over 15368.00 frames. ], tot_loss[loss=0.07638, simple_loss=0.09809, pruned_loss=0.01759, audio_tagging_loss=0.009748, over 3053920.99 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 32.0 2023-11-21 06:57:46,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1401333.3333333333, ans=0.09899494936611666 2023-11-21 06:57:54,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1401400.0, ans=0.125 2023-11-21 06:57:58,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1401400.0, ans=0.2 2023-11-21 06:58:04,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1401466.6666666667, ans=0.125 2023-11-21 06:58:05,604 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.333e+01 8.998e+01 9.692e+01 1.329e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-21 06:58:16,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1401533.3333333333, ans=0.0 2023-11-21 06:58:35,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1401600.0, ans=0.04949747468305833 2023-11-21 06:58:41,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210250 2023-11-21 06:58:44,343 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5850, loss[loss=0.05711, simple_loss=0.06529, pruned_loss=0.01255, audio_tagging_loss=0.01191, over 15852.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09802, pruned_loss=0.01762, audio_tagging_loss=0.009619, over 3047188.29 frames. ], batch size: 62, lr: 3.82e-03, grad_scale: 32.0 2023-11-21 06:58:59,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1401733.3333333333, ans=0.125 2023-11-21 06:59:12,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1401800.0, ans=0.0 2023-11-21 06:59:20,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1401800.0, ans=0.1 2023-11-21 06:59:22,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1401866.6666666667, ans=0.0 2023-11-21 06:59:23,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1401866.6666666667, ans=0.125 2023-11-21 06:59:28,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0 2023-11-21 06:59:42,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.54 vs. limit=15.0 2023-11-21 06:59:45,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210300 2023-11-21 06:59:47,879 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5900, loss[loss=0.07476, simple_loss=0.1042, pruned_loss=0.01646, audio_tagging_loss=0.006219, over 14453.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09781, pruned_loss=0.01758, audio_tagging_loss=0.009529, over 3049259.37 frames. ], batch size: 54, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 06:59:48,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.35 vs. limit=15.0 2023-11-21 06:59:51,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1402000.0, ans=0.125 2023-11-21 06:59:54,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.43 vs. limit=15.0 2023-11-21 07:00:17,474 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.132e+01 8.306e+01 8.999e+01 9.965e+01 1.195e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-21 07:00:37,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1402200.0, ans=0.2 2023-11-21 07:00:51,591 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210350 2023-11-21 07:00:51,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1402266.6666666667, ans=0.2 2023-11-21 07:00:54,591 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 5950, loss[loss=0.09184, simple_loss=0.1197, pruned_loss=0.02202, audio_tagging_loss=0.009951, over 15195.00 frames. ], tot_loss[loss=0.07618, simple_loss=0.09803, pruned_loss=0.01754, audio_tagging_loss=0.009618, over 3045978.47 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:01:05,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1402400.0, ans=0.125 2023-11-21 07:01:08,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1402400.0, ans=10.0 2023-11-21 07:01:38,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1402533.3333333333, ans=0.0 2023-11-21 07:01:47,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0 2023-11-21 07:01:49,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1402600.0, ans=0.0 2023-11-21 07:01:49,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1402600.0, ans=0.125 2023-11-21 07:01:55,448 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210400 2023-11-21 07:01:58,242 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6000, loss[loss=0.07789, simple_loss=0.1108, pruned_loss=0.0151, audio_tagging_loss=0.007394, over 16095.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09749, pruned_loss=0.01734, audio_tagging_loss=0.009578, over 3048931.20 frames. ], batch size: 59, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:01:58,243 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 07:02:30,389 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.1492, 3.1544, 3.4070, 2.8944, 3.8230, 3.8012, 3.3807, 3.1783], device='cuda:3') 2023-11-21 07:02:40,594 INFO [train_asr.py:1253] (3/4) Epoch 18, validation: loss=0.0604, simple_loss=0.05257, pruned_loss=0.00537, audio_tagging_loss=0.02874, over 4681554.00 frames. 2023-11-21 07:02:40,594 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 07:03:09,956 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 7.991e+01 8.851e+01 9.572e+01 1.607e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-21 07:03:21,778 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.40 vs. limit=15.0 2023-11-21 07:03:27,308 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 07:03:29,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1402866.6666666667, ans=0.1 2023-11-21 07:03:37,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1402933.3333333333, ans=0.125 2023-11-21 07:03:37,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1402933.3333333333, ans=0.125 2023-11-21 07:03:44,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210450 2023-11-21 07:03:47,031 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6050, loss[loss=0.1099, simple_loss=0.1436, pruned_loss=0.03096, audio_tagging_loss=0.00713, over 15715.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09719, pruned_loss=0.01744, audio_tagging_loss=0.009593, over 3049909.47 frames. ], batch size: 54, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:03:48,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1403000.0, ans=10.0 2023-11-21 07:03:59,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1403066.6666666667, ans=0.125 2023-11-21 07:04:21,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1403133.3333333333, ans=0.2 2023-11-21 07:04:21,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1403133.3333333333, ans=0.125 2023-11-21 07:04:24,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1403200.0, ans=0.1 2023-11-21 07:04:32,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1403200.0, ans=0.0 2023-11-21 07:04:48,584 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210500 2023-11-21 07:04:51,000 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6100, loss[loss=0.07872, simple_loss=0.11, pruned_loss=0.01531, audio_tagging_loss=0.008405, over 16295.00 frames. ], tot_loss[loss=0.07624, simple_loss=0.09797, pruned_loss=0.01765, audio_tagging_loss=0.009606, over 3045070.23 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:05:04,172 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.07 vs. limit=15.0 2023-11-21 07:05:16,614 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.76 vs. limit=15.0 2023-11-21 07:05:20,538 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.632e+01 8.223e+01 8.598e+01 9.128e+01 4.094e+02, threshold=1.720e+02, percent-clipped=1.0 2023-11-21 07:05:35,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1403533.3333333333, ans=0.0 2023-11-21 07:05:51,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1403600.0, ans=0.125 2023-11-21 07:05:53,444 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210550 2023-11-21 07:05:54,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1403666.6666666667, ans=0.125 2023-11-21 07:05:55,857 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6150, loss[loss=0.06791, simple_loss=0.08946, pruned_loss=0.01461, audio_tagging_loss=0.00857, over 15866.00 frames. ], tot_loss[loss=0.07575, simple_loss=0.09709, pruned_loss=0.01747, audio_tagging_loss=0.009733, over 3042764.29 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:05:57,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1403666.6666666667, ans=0.95 2023-11-21 07:06:16,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1403733.3333333333, ans=0.0 2023-11-21 07:06:19,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1403733.3333333333, ans=0.0 2023-11-21 07:06:34,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1403866.6666666667, ans=0.125 2023-11-21 07:06:42,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1403866.6666666667, ans=0.0 2023-11-21 07:06:49,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.34 vs. limit=15.0 2023-11-21 07:06:50,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1403933.3333333333, ans=0.0 2023-11-21 07:07:00,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210600 2023-11-21 07:07:03,393 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6200, loss[loss=0.07523, simple_loss=0.1063, pruned_loss=0.01449, audio_tagging_loss=0.007608, over 14788.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09644, pruned_loss=0.0173, audio_tagging_loss=0.009895, over 3041795.86 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:07:04,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1404000.0, ans=0.125 2023-11-21 07:07:18,472 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0 2023-11-21 07:07:26,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.96 vs. limit=15.0 2023-11-21 07:07:29,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1404133.3333333333, ans=0.0 2023-11-21 07:07:31,492 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.071e+01 8.570e+01 9.490e+01 1.156e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 07:07:43,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1404200.0, ans=0.0 2023-11-21 07:07:49,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1404200.0, ans=0.0 2023-11-21 07:08:06,186 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210650 2023-11-21 07:08:08,599 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6250, loss[loss=0.06522, simple_loss=0.08174, pruned_loss=0.01344, audio_tagging_loss=0.01091, over 16611.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09628, pruned_loss=0.01722, audio_tagging_loss=0.009987, over 3032839.67 frames. ], batch size: 63, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:08:26,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1404400.0, ans=0.1 2023-11-21 07:08:27,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1404400.0, ans=0.125 2023-11-21 07:08:31,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1404400.0, ans=0.1 2023-11-21 07:08:32,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1404466.6666666667, ans=0.125 2023-11-21 07:09:00,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1404600.0, ans=0.125 2023-11-21 07:09:10,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210700 2023-11-21 07:09:12,482 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6300, loss[loss=0.05876, simple_loss=0.06511, pruned_loss=0.01272, audio_tagging_loss=0.01348, over 15001.00 frames. ], tot_loss[loss=0.07589, simple_loss=0.09711, pruned_loss=0.01731, audio_tagging_loss=0.01003, over 3034068.52 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:09:22,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1404666.6666666667, ans=0.0 2023-11-21 07:09:25,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1404733.3333333333, ans=0.07 2023-11-21 07:09:33,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0 2023-11-21 07:09:41,466 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.208e+01 8.088e+01 8.800e+01 9.638e+01 1.138e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-21 07:09:43,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1404800.0, ans=0.125 2023-11-21 07:09:58,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1404866.6666666667, ans=0.0 2023-11-21 07:10:13,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1404933.3333333333, ans=0.125 2023-11-21 07:10:13,875 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.13 vs. limit=6.0 2023-11-21 07:10:15,424 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210750 2023-11-21 07:10:18,531 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6350, loss[loss=0.06413, simple_loss=0.08477, pruned_loss=0.01151, audio_tagging_loss=0.01023, over 15489.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09731, pruned_loss=0.01728, audio_tagging_loss=0.0101, over 3039457.73 frames. ], batch size: 59, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:10:44,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1405133.3333333333, ans=0.125 2023-11-21 07:10:44,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1405133.3333333333, ans=0.125 2023-11-21 07:10:57,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1405200.0, ans=0.125 2023-11-21 07:11:20,627 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210800 2023-11-21 07:11:23,320 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6400, loss[loss=0.04911, simple_loss=0.06148, pruned_loss=0.006675, audio_tagging_loss=0.0117, over 16060.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09771, pruned_loss=0.01728, audio_tagging_loss=0.01012, over 3049132.86 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:11:45,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1405400.0, ans=0.125 2023-11-21 07:11:55,152 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.546e+01 8.103e+01 8.679e+01 9.633e+01 1.173e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-21 07:11:58,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1405466.6666666667, ans=0.125 2023-11-21 07:12:08,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1405533.3333333333, ans=0.125 2023-11-21 07:12:25,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210850 2023-11-21 07:12:28,231 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6450, loss[loss=0.05931, simple_loss=0.07275, pruned_loss=0.009843, audio_tagging_loss=0.01309, over 15232.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.0987, pruned_loss=0.01753, audio_tagging_loss=0.01009, over 3043211.82 frames. ], batch size: 59, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:12:35,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0 2023-11-21 07:12:35,295 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.83 vs. limit=15.0 2023-11-21 07:12:42,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1405733.3333333333, ans=0.125 2023-11-21 07:12:48,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.69 vs. limit=10.0 2023-11-21 07:12:58,239 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.99 vs. limit=15.0 2023-11-21 07:13:32,037 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210900 2023-11-21 07:13:33,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1406000.0, ans=0.125 2023-11-21 07:13:34,484 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6500, loss[loss=0.08963, simple_loss=0.1225, pruned_loss=0.02188, audio_tagging_loss=0.006512, over 14756.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09779, pruned_loss=0.01734, audio_tagging_loss=0.01007, over 3045231.60 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:14:05,534 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.017e+01 8.785e+01 9.599e+01 1.242e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-21 07:14:15,174 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.386e-01 2023-11-21 07:14:37,179 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 210950 2023-11-21 07:14:39,584 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6550, loss[loss=0.07209, simple_loss=0.09648, pruned_loss=0.0155, audio_tagging_loss=0.008355, over 16001.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09719, pruned_loss=0.01728, audio_tagging_loss=0.009907, over 3049491.06 frames. ], batch size: 58, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:14:39,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1406333.3333333333, ans=0.09899494936611666 2023-11-21 07:14:51,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1406400.0, ans=0.2 2023-11-21 07:15:41,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211000 2023-11-21 07:15:42,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1406600.0, ans=0.2 2023-11-21 07:15:44,654 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6600, loss[loss=0.09507, simple_loss=0.1246, pruned_loss=0.02257, audio_tagging_loss=0.0102, over 15893.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.0972, pruned_loss=0.01724, audio_tagging_loss=0.009761, over 3049235.71 frames. ], batch size: 55, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:15:49,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1406666.6666666667, ans=0.125 2023-11-21 07:16:16,225 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 8.176e+01 8.712e+01 9.397e+01 1.234e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-21 07:16:34,713 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.93 vs. limit=15.0 2023-11-21 07:16:36,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1406933.3333333333, ans=0.1 2023-11-21 07:16:47,268 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211050 2023-11-21 07:16:48,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1407000.0, ans=0.125 2023-11-21 07:16:50,694 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6650, loss[loss=0.0888, simple_loss=0.1069, pruned_loss=0.02373, audio_tagging_loss=0.01162, over 14884.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09686, pruned_loss=0.01727, audio_tagging_loss=0.00972, over 3047641.54 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:17:00,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1407000.0, ans=0.5 2023-11-21 07:17:25,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1407133.3333333333, ans=0.125 2023-11-21 07:17:37,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1407200.0, ans=0.0 2023-11-21 07:17:39,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1407200.0, ans=0.125 2023-11-21 07:17:50,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1407266.6666666667, ans=0.125 2023-11-21 07:17:53,715 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211100 2023-11-21 07:17:56,166 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6700, loss[loss=0.08846, simple_loss=0.1123, pruned_loss=0.02284, audio_tagging_loss=0.009481, over 17235.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.09731, pruned_loss=0.01749, audio_tagging_loss=0.00965, over 3050266.23 frames. ], batch size: 64, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:18:07,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1407400.0, ans=0.035 2023-11-21 07:18:15,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1407400.0, ans=0.0 2023-11-21 07:18:17,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=22.59 vs. limit=22.5 2023-11-21 07:18:21,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1407466.6666666667, ans=0.125 2023-11-21 07:18:24,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1407466.6666666667, ans=0.125 2023-11-21 07:18:27,197 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.113e+01 8.605e+01 9.333e+01 1.253e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-21 07:18:38,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1407533.3333333333, ans=0.0 2023-11-21 07:18:58,334 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211150 2023-11-21 07:19:00,731 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6750, loss[loss=0.04968, simple_loss=0.05582, pruned_loss=0.01193, audio_tagging_loss=0.009844, over 15290.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09783, pruned_loss=0.01759, audio_tagging_loss=0.00963, over 3046867.60 frames. ], batch size: 59, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:19:21,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1407733.3333333333, ans=0.0 2023-11-21 07:19:56,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1407933.3333333333, ans=0.0 2023-11-21 07:20:03,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211200 2023-11-21 07:20:06,273 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6800, loss[loss=0.09324, simple_loss=0.1201, pruned_loss=0.02546, audio_tagging_loss=0.007733, over 14699.00 frames. ], tot_loss[loss=0.07635, simple_loss=0.09843, pruned_loss=0.01761, audio_tagging_loss=0.009519, over 3044965.10 frames. ], batch size: 55, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:20:08,189 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.55 vs. limit=10.0 2023-11-21 07:20:10,736 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.90 vs. limit=10.0 2023-11-21 07:20:33,445 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.67 vs. limit=15.0 2023-11-21 07:20:36,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1408133.3333333333, ans=0.2 2023-11-21 07:20:37,398 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.078e+01 8.657e+01 9.285e+01 1.100e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 07:20:41,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1408133.3333333333, ans=0.0 2023-11-21 07:20:50,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1408200.0, ans=0.0 2023-11-21 07:20:58,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1408266.6666666667, ans=0.125 2023-11-21 07:21:03,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1408266.6666666667, ans=0.2 2023-11-21 07:21:09,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211250 2023-11-21 07:21:11,851 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6850, loss[loss=0.08418, simple_loss=0.1023, pruned_loss=0.02262, audio_tagging_loss=0.01039, over 14990.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09807, pruned_loss=0.01746, audio_tagging_loss=0.009518, over 3051654.70 frames. ], batch size: 54, lr: 3.81e-03, grad_scale: 32.0 2023-11-21 07:21:24,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1408400.0, ans=0.07 2023-11-21 07:21:28,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1408400.0, ans=0.1 2023-11-21 07:21:55,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1408533.3333333333, ans=0.125 2023-11-21 07:22:14,390 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211300 2023-11-21 07:22:16,870 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6900, loss[loss=0.07326, simple_loss=0.09363, pruned_loss=0.01663, audio_tagging_loss=0.00982, over 14996.00 frames. ], tot_loss[loss=0.07597, simple_loss=0.0983, pruned_loss=0.01738, audio_tagging_loss=0.009441, over 3049649.80 frames. ], batch size: 58, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:22:22,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1408666.6666666667, ans=0.125 2023-11-21 07:22:27,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1408666.6666666667, ans=0.0 2023-11-21 07:22:28,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1408733.3333333333, ans=0.0 2023-11-21 07:22:38,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.19 vs. limit=15.0 2023-11-21 07:22:44,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.20 vs. limit=15.0 2023-11-21 07:22:44,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1408800.0, ans=0.0 2023-11-21 07:22:50,579 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.236e+01 8.770e+01 9.429e+01 1.247e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 07:22:50,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1408800.0, ans=0.025 2023-11-21 07:23:06,767 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 07:23:10,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1408933.3333333333, ans=0.0 2023-11-21 07:23:18,948 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211350 2023-11-21 07:23:21,804 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 6950, loss[loss=0.09048, simple_loss=0.1186, pruned_loss=0.02132, audio_tagging_loss=0.009878, over 15518.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09853, pruned_loss=0.01747, audio_tagging_loss=0.009465, over 3053570.89 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:23:26,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1409000.0, ans=0.2 2023-11-21 07:23:29,341 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.10 vs. limit=22.5 2023-11-21 07:23:59,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0 2023-11-21 07:24:06,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1409200.0, ans=0.125 2023-11-21 07:24:17,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1409266.6666666667, ans=0.125 2023-11-21 07:24:25,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211400 2023-11-21 07:24:27,844 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7000, loss[loss=0.05654, simple_loss=0.07327, pruned_loss=0.01099, audio_tagging_loss=0.008915, over 14099.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.0972, pruned_loss=0.01713, audio_tagging_loss=0.009574, over 3051076.51 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0 2023-11-21 07:24:40,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1409400.0, ans=0.1 2023-11-21 07:24:41,090 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.59 vs. limit=15.0 2023-11-21 07:24:43,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1409400.0, ans=0.1 2023-11-21 07:24:45,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1409400.0, ans=0.1 2023-11-21 07:24:57,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1409466.6666666667, ans=0.025 2023-11-21 07:24:58,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1409466.6666666667, ans=0.025 2023-11-21 07:24:59,166 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 7.925e+01 8.553e+01 9.217e+01 1.576e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-21 07:25:01,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1409466.6666666667, ans=0.125 2023-11-21 07:25:30,548 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211450 2023-11-21 07:25:32,863 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7050, loss[loss=0.05555, simple_loss=0.0684, pruned_loss=0.01143, audio_tagging_loss=0.00992, over 15508.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09685, pruned_loss=0.01723, audio_tagging_loss=0.009726, over 3062830.67 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 16.0 2023-11-21 07:25:34,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1409666.6666666667, ans=0.05 2023-11-21 07:25:35,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1409666.6666666667, ans=0.1 2023-11-21 07:25:56,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1409733.3333333333, ans=0.0 2023-11-21 07:26:06,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1409800.0, ans=0.025 2023-11-21 07:26:34,565 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211500 2023-11-21 07:26:36,964 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7100, loss[loss=0.07811, simple_loss=0.1033, pruned_loss=0.01684, audio_tagging_loss=0.009631, over 14833.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09683, pruned_loss=0.01714, audio_tagging_loss=0.00984, over 3067612.22 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 16.0 2023-11-21 07:26:37,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1410000.0, ans=0.125 2023-11-21 07:26:41,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1410000.0, ans=0.125 2023-11-21 07:26:50,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1410066.6666666667, ans=0.125 2023-11-21 07:27:06,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1410133.3333333333, ans=0.2 2023-11-21 07:27:10,196 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.252e+01 8.974e+01 1.013e+02 1.228e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-21 07:27:31,414 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.57 vs. limit=15.0 2023-11-21 07:27:41,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211550 2023-11-21 07:27:43,609 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7150, loss[loss=0.0878, simple_loss=0.1118, pruned_loss=0.01984, audio_tagging_loss=0.01206, over 14700.00 frames. ], tot_loss[loss=0.07528, simple_loss=0.09652, pruned_loss=0.01712, audio_tagging_loss=0.009899, over 3065209.20 frames. ], batch size: 54, lr: 3.80e-03, grad_scale: 16.0 2023-11-21 07:28:01,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1410400.0, ans=0.2 2023-11-21 07:28:02,792 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=15.0 2023-11-21 07:28:30,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1410533.3333333333, ans=0.125 2023-11-21 07:28:45,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211600 2023-11-21 07:28:48,051 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7200, loss[loss=0.08359, simple_loss=0.1139, pruned_loss=0.01741, audio_tagging_loss=0.009223, over 15762.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09702, pruned_loss=0.01714, audio_tagging_loss=0.009892, over 3067957.07 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:29:07,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1410733.3333333333, ans=0.125 2023-11-21 07:29:20,722 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 7.997e+01 8.820e+01 9.312e+01 1.213e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-21 07:29:25,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1410800.0, ans=0.0 2023-11-21 07:29:26,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1410866.6666666667, ans=0.0 2023-11-21 07:29:37,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.91 vs. limit=15.0 2023-11-21 07:29:50,147 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211650 2023-11-21 07:29:52,665 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7250, loss[loss=0.07028, simple_loss=0.08518, pruned_loss=0.01375, audio_tagging_loss=0.01394, over 15872.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09715, pruned_loss=0.01714, audio_tagging_loss=0.009883, over 3056187.96 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:29:57,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1411000.0, ans=0.0 2023-11-21 07:30:05,254 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.57 vs. limit=10.0 2023-11-21 07:30:25,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.84 vs. limit=15.0 2023-11-21 07:30:33,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1411200.0, ans=0.125 2023-11-21 07:30:43,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1411266.6666666667, ans=0.125 2023-11-21 07:30:46,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1411266.6666666667, ans=0.125 2023-11-21 07:30:49,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.19 vs. limit=15.0 2023-11-21 07:30:55,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211700 2023-11-21 07:30:58,969 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7300, loss[loss=0.08263, simple_loss=0.108, pruned_loss=0.01793, audio_tagging_loss=0.0107, over 14471.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.0965, pruned_loss=0.01702, audio_tagging_loss=0.009791, over 3049540.39 frames. ], batch size: 53, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:31:05,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1411333.3333333333, ans=0.0 2023-11-21 07:31:14,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1411400.0, ans=0.035 2023-11-21 07:31:24,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1411466.6666666667, ans=0.025 2023-11-21 07:31:30,106 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.648e+01 8.326e+01 8.844e+01 9.526e+01 1.717e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 07:31:32,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.90 vs. limit=15.0 2023-11-21 07:32:01,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211750 2023-11-21 07:32:03,696 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7350, loss[loss=0.09057, simple_loss=0.1144, pruned_loss=0.02423, audio_tagging_loss=0.009129, over 14875.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09738, pruned_loss=0.01728, audio_tagging_loss=0.00965, over 3058709.96 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:32:43,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1411866.6666666667, ans=0.125 2023-11-21 07:33:02,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1411933.3333333333, ans=0.125 2023-11-21 07:33:04,757 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211800 2023-11-21 07:33:07,572 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7400, loss[loss=0.08734, simple_loss=0.1136, pruned_loss=0.01922, audio_tagging_loss=0.01133, over 15304.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09734, pruned_loss=0.01721, audio_tagging_loss=0.009467, over 3051285.66 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:33:13,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.63 vs. limit=15.0 2023-11-21 07:33:15,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1412000.0, ans=0.09899494936611666 2023-11-21 07:33:22,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1412066.6666666667, ans=0.2 2023-11-21 07:33:40,826 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 7.896e+01 8.684e+01 9.314e+01 1.199e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 07:34:09,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211850 2023-11-21 07:34:12,029 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7450, loss[loss=0.07205, simple_loss=0.1051, pruned_loss=0.01282, audio_tagging_loss=0.006678, over 15209.00 frames. ], tot_loss[loss=0.07461, simple_loss=0.09619, pruned_loss=0.01705, audio_tagging_loss=0.009466, over 3047842.06 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:34:14,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1412333.3333333333, ans=0.2 2023-11-21 07:34:17,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1412333.3333333333, ans=0.0 2023-11-21 07:34:29,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1412400.0, ans=0.125 2023-11-21 07:34:59,706 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=15.50 vs. limit=15.0 2023-11-21 07:35:13,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211900 2023-11-21 07:35:13,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1412600.0, ans=0.125 2023-11-21 07:35:15,931 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7500, loss[loss=0.08587, simple_loss=0.1146, pruned_loss=0.02007, audio_tagging_loss=0.008515, over 14951.00 frames. ], tot_loss[loss=0.07508, simple_loss=0.09707, pruned_loss=0.0171, audio_tagging_loss=0.009447, over 3052056.35 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:35:16,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1412666.6666666667, ans=0.1 2023-11-21 07:35:25,925 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.82 vs. limit=15.0 2023-11-21 07:35:44,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1412800.0, ans=0.0 2023-11-21 07:35:47,745 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.064e+01 8.532e+01 9.428e+01 1.202e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-21 07:36:17,907 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 211950 2023-11-21 07:36:20,300 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7550, loss[loss=0.06861, simple_loss=0.08678, pruned_loss=0.01624, audio_tagging_loss=0.008987, over 15038.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09688, pruned_loss=0.01688, audio_tagging_loss=0.009425, over 3052570.18 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:36:50,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1413133.3333333333, ans=0.0 2023-11-21 07:37:22,404 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212000 2023-11-21 07:37:28,034 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7600, loss[loss=0.09399, simple_loss=0.1223, pruned_loss=0.02559, audio_tagging_loss=0.007254, over 15621.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09627, pruned_loss=0.01699, audio_tagging_loss=0.009462, over 3041468.79 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:38:00,162 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.299e+01 8.757e+01 9.275e+01 1.161e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 07:38:03,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.24 vs. limit=15.0 2023-11-21 07:38:30,228 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212050 2023-11-21 07:38:32,597 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7650, loss[loss=0.07771, simple_loss=0.1006, pruned_loss=0.01862, audio_tagging_loss=0.008811, over 15330.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09715, pruned_loss=0.01731, audio_tagging_loss=0.009494, over 3043084.11 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:38:41,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1413666.6666666667, ans=0.0 2023-11-21 07:38:41,700 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.75 vs. limit=12.0 2023-11-21 07:39:21,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1413866.6666666667, ans=0.0 2023-11-21 07:39:34,867 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212100 2023-11-21 07:39:36,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.19 vs. limit=22.5 2023-11-21 07:39:37,893 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7700, loss[loss=0.07627, simple_loss=0.0998, pruned_loss=0.0165, audio_tagging_loss=0.009866, over 15898.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09665, pruned_loss=0.01712, audio_tagging_loss=0.009516, over 3044363.00 frames. ], batch size: 60, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:39:48,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1414000.0, ans=0.0 2023-11-21 07:39:56,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1414066.6666666667, ans=0.0 2023-11-21 07:40:10,450 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.127e+01 8.808e+01 9.594e+01 1.135e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-21 07:40:14,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1414133.3333333333, ans=0.2 2023-11-21 07:40:23,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1414200.0, ans=0.125 2023-11-21 07:40:34,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1414266.6666666667, ans=0.0 2023-11-21 07:40:40,138 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212150 2023-11-21 07:40:42,654 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7750, loss[loss=0.07654, simple_loss=0.09624, pruned_loss=0.01995, audio_tagging_loss=0.008474, over 14383.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09599, pruned_loss=0.01698, audio_tagging_loss=0.0096, over 3032332.39 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:40:42,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1414333.3333333333, ans=0.125 2023-11-21 07:41:37,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.20 vs. limit=15.0 2023-11-21 07:41:46,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212200 2023-11-21 07:41:48,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1414666.6666666667, ans=0.125 2023-11-21 07:41:49,587 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7800, loss[loss=0.08355, simple_loss=0.1087, pruned_loss=0.01998, audio_tagging_loss=0.009208, over 16172.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09641, pruned_loss=0.01701, audio_tagging_loss=0.009595, over 3033488.56 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:42:02,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1414733.3333333333, ans=0.125 2023-11-21 07:42:17,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1414800.0, ans=0.125 2023-11-21 07:42:21,723 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.253e+01 8.940e+01 9.769e+01 1.668e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-21 07:42:24,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1414800.0, ans=0.07 2023-11-21 07:42:33,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1414866.6666666667, ans=0.0 2023-11-21 07:42:35,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1414866.6666666667, ans=0.2 2023-11-21 07:42:44,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1414933.3333333333, ans=0.125 2023-11-21 07:42:51,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212250 2023-11-21 07:42:53,714 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7850, loss[loss=0.06307, simple_loss=0.07715, pruned_loss=0.01246, audio_tagging_loss=0.01203, over 16549.00 frames. ], tot_loss[loss=0.07469, simple_loss=0.0959, pruned_loss=0.01701, audio_tagging_loss=0.009728, over 3031140.19 frames. ], batch size: 62, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:43:05,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1415066.6666666667, ans=0.2 2023-11-21 07:43:13,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1415066.6666666667, ans=0.0 2023-11-21 07:43:18,088 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.49 vs. limit=6.0 2023-11-21 07:43:23,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1415133.3333333333, ans=0.125 2023-11-21 07:43:56,275 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212300 2023-11-21 07:43:59,268 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7900, loss[loss=0.09659, simple_loss=0.1191, pruned_loss=0.02851, audio_tagging_loss=0.008555, over 15128.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09682, pruned_loss=0.01722, audio_tagging_loss=0.009779, over 3041661.34 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 16.0 2023-11-21 07:44:24,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1415466.6666666667, ans=0.125 2023-11-21 07:44:31,963 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.051e+01 8.734e+01 9.440e+01 1.312e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 07:44:39,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1415533.3333333333, ans=0.2 2023-11-21 07:44:42,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1415533.3333333333, ans=0.2 2023-11-21 07:44:42,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.25 vs. limit=15.0 2023-11-21 07:44:48,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1415533.3333333333, ans=0.1 2023-11-21 07:44:48,455 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 07:44:57,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1415600.0, ans=0.125 2023-11-21 07:45:01,041 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212350 2023-11-21 07:45:03,393 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 7950, loss[loss=0.07922, simple_loss=0.09721, pruned_loss=0.0182, audio_tagging_loss=0.01242, over 16941.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09668, pruned_loss=0.01743, audio_tagging_loss=0.009828, over 3043092.07 frames. ], batch size: 64, lr: 3.80e-03, grad_scale: 16.0 2023-11-21 07:45:17,042 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 07:45:33,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1415800.0, ans=0.09899494936611666 2023-11-21 07:45:45,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=1415866.6666666667, ans=10.0 2023-11-21 07:45:46,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1415866.6666666667, ans=0.125 2023-11-21 07:46:00,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1415933.3333333333, ans=0.0 2023-11-21 07:46:02,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.22 vs. limit=15.0 2023-11-21 07:46:04,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212400 2023-11-21 07:46:06,782 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8000, loss[loss=0.08648, simple_loss=0.1111, pruned_loss=0.02019, audio_tagging_loss=0.01072, over 14859.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09616, pruned_loss=0.01719, audio_tagging_loss=0.009928, over 3041112.95 frames. ], batch size: 54, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:46:09,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1416000.0, ans=0.0 2023-11-21 07:46:40,022 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 7.990e+01 8.488e+01 9.256e+01 1.198e+02, threshold=1.698e+02, percent-clipped=0.0 2023-11-21 07:47:02,964 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.89 vs. limit=12.0 2023-11-21 07:47:07,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212450 2023-11-21 07:47:09,802 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8050, loss[loss=0.08484, simple_loss=0.1095, pruned_loss=0.02085, audio_tagging_loss=0.009232, over 16001.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09666, pruned_loss=0.01722, audio_tagging_loss=0.009887, over 3049833.61 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:47:55,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1416533.3333333333, ans=0.07 2023-11-21 07:48:14,158 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212500 2023-11-21 07:48:16,506 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8100, loss[loss=0.08346, simple_loss=0.1057, pruned_loss=0.02181, audio_tagging_loss=0.008774, over 15805.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09658, pruned_loss=0.01707, audio_tagging_loss=0.009823, over 3048525.38 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0 2023-11-21 07:48:18,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1416666.6666666667, ans=0.0 2023-11-21 07:48:25,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1416666.6666666667, ans=0.125 2023-11-21 07:48:38,207 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=15.0 2023-11-21 07:48:48,750 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.164e+01 8.831e+01 9.512e+01 1.309e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-21 07:48:50,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1416800.0, ans=0.125 2023-11-21 07:49:09,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.52 vs. limit=15.0 2023-11-21 07:49:18,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212550 2023-11-21 07:49:19,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1417000.0, ans=0.125 2023-11-21 07:49:20,113 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.68 vs. limit=12.0 2023-11-21 07:49:20,836 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8150, loss[loss=0.0812, simple_loss=0.09555, pruned_loss=0.02252, audio_tagging_loss=0.0109, over 15292.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.09656, pruned_loss=0.01698, audio_tagging_loss=0.00966, over 3046713.98 frames. ], batch size: 59, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:49:39,470 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.75 vs. limit=15.0 2023-11-21 07:50:08,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1417200.0, ans=0.0 2023-11-21 07:50:10,689 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 07:50:21,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212600 2023-11-21 07:50:24,232 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8200, loss[loss=0.07867, simple_loss=0.09785, pruned_loss=0.01701, audio_tagging_loss=0.01274, over 14965.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09638, pruned_loss=0.01695, audio_tagging_loss=0.009666, over 3035761.98 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:50:24,286 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 07:50:27,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1417333.3333333333, ans=0.0 2023-11-21 07:50:58,022 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.391e+01 9.092e+01 9.919e+01 1.241e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-21 07:51:06,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1417533.3333333333, ans=0.1 2023-11-21 07:51:12,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1417533.3333333333, ans=0.2 2023-11-21 07:51:13,614 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.75 vs. limit=15.0 2023-11-21 07:51:23,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1417600.0, ans=0.1 2023-11-21 07:51:26,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212650 2023-11-21 07:51:29,788 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8250, loss[loss=0.08919, simple_loss=0.1189, pruned_loss=0.02215, audio_tagging_loss=0.007603, over 15456.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09589, pruned_loss=0.01683, audio_tagging_loss=0.009581, over 3037707.34 frames. ], batch size: 57, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:51:51,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0 2023-11-21 07:51:56,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1417800.0, ans=0.2 2023-11-21 07:52:32,622 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212700 2023-11-21 07:52:32,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1417933.3333333333, ans=0.1 2023-11-21 07:52:35,018 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8300, loss[loss=0.07364, simple_loss=0.0854, pruned_loss=0.01825, audio_tagging_loss=0.01268, over 14699.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09588, pruned_loss=0.017, audio_tagging_loss=0.009629, over 3033708.82 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:52:36,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.36 vs. limit=15.0 2023-11-21 07:53:10,528 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.350e+01 9.034e+01 9.751e+01 1.328e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-21 07:53:34,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1418266.6666666667, ans=0.0 2023-11-21 07:53:37,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212750 2023-11-21 07:53:37,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1418266.6666666667, ans=0.1 2023-11-21 07:53:39,737 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8350, loss[loss=0.04307, simple_loss=0.05282, pruned_loss=0.005949, audio_tagging_loss=0.01071, over 15541.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09535, pruned_loss=0.0168, audio_tagging_loss=0.009644, over 3030302.78 frames. ], batch size: 59, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 07:53:48,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.52 vs. limit=22.5 2023-11-21 07:53:48,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.70 vs. limit=10.0 2023-11-21 07:53:49,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1418333.3333333333, ans=0.0 2023-11-21 07:53:53,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1418400.0, ans=0.125 2023-11-21 07:54:10,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1418466.6666666667, ans=0.125 2023-11-21 07:54:16,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1418466.6666666667, ans=0.1 2023-11-21 07:54:42,458 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212800 2023-11-21 07:54:45,216 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8400, loss[loss=0.07731, simple_loss=0.08479, pruned_loss=0.02057, audio_tagging_loss=0.01434, over 14212.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09552, pruned_loss=0.01675, audio_tagging_loss=0.009564, over 3025394.66 frames. ], batch size: 57, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:54:47,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1418666.6666666667, ans=0.125 2023-11-21 07:54:54,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1418666.6666666667, ans=0.0 2023-11-21 07:55:01,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1418733.3333333333, ans=0.1 2023-11-21 07:55:09,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1418733.3333333333, ans=0.125 2023-11-21 07:55:21,066 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.837e+01 7.810e+01 8.454e+01 9.398e+01 1.177e+02, threshold=1.691e+02, percent-clipped=0.0 2023-11-21 07:55:37,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1418933.3333333333, ans=0.0 2023-11-21 07:55:46,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1418933.3333333333, ans=0.1 2023-11-21 07:55:49,250 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212850 2023-11-21 07:55:51,653 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8450, loss[loss=0.07885, simple_loss=0.09887, pruned_loss=0.01831, audio_tagging_loss=0.0111, over 15380.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09579, pruned_loss=0.01672, audio_tagging_loss=0.009592, over 3018986.66 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:55:52,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1419000.0, ans=0.125 2023-11-21 07:56:06,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1419066.6666666667, ans=0.0 2023-11-21 07:56:12,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1419066.6666666667, ans=0.0 2023-11-21 07:56:35,284 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.21 vs. limit=15.0 2023-11-21 07:56:49,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1419266.6666666667, ans=0.1 2023-11-21 07:56:53,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212900 2023-11-21 07:56:55,737 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8500, loss[loss=0.05375, simple_loss=0.07344, pruned_loss=0.005914, audio_tagging_loss=0.01112, over 14047.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.0962, pruned_loss=0.01692, audio_tagging_loss=0.009561, over 3021631.77 frames. ], batch size: 54, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:56:59,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1419333.3333333333, ans=0.2 2023-11-21 07:57:02,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1419333.3333333333, ans=0.1 2023-11-21 07:57:19,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1419400.0, ans=0.125 2023-11-21 07:57:31,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1419466.6666666667, ans=0.125 2023-11-21 07:57:32,201 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.883e+01 8.044e+01 8.694e+01 9.368e+01 1.136e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 07:57:36,287 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 07:57:58,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 212950 2023-11-21 07:58:01,242 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8550, loss[loss=0.07529, simple_loss=0.1046, pruned_loss=0.01562, audio_tagging_loss=0.007376, over 15146.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09604, pruned_loss=0.01683, audio_tagging_loss=0.009617, over 3016756.71 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:58:05,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1419666.6666666667, ans=0.0 2023-11-21 07:58:10,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1419666.6666666667, ans=0.0 2023-11-21 07:58:45,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1419866.6666666667, ans=0.2 2023-11-21 07:58:52,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1419933.3333333333, ans=0.0 2023-11-21 07:58:55,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1419933.3333333333, ans=0.125 2023-11-21 07:59:03,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213000 2023-11-21 07:59:06,332 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8600, loss[loss=0.05598, simple_loss=0.06841, pruned_loss=0.01022, audio_tagging_loss=0.01155, over 15126.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09497, pruned_loss=0.01675, audio_tagging_loss=0.009706, over 3030147.94 frames. ], batch size: 59, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 07:59:37,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1420133.3333333333, ans=0.0 2023-11-21 07:59:40,782 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.296e+01 8.926e+01 9.744e+01 1.392e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-21 07:59:58,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1420266.6666666667, ans=0.125 2023-11-21 08:00:03,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.32 vs. limit=8.0 2023-11-21 08:00:06,309 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.38 vs. limit=15.0 2023-11-21 08:00:08,163 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213050 2023-11-21 08:00:10,607 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8650, loss[loss=0.0616, simple_loss=0.07954, pruned_loss=0.01055, audio_tagging_loss=0.01128, over 15315.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09475, pruned_loss=0.01676, audio_tagging_loss=0.009808, over 3028001.93 frames. ], batch size: 58, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:00:43,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1420466.6666666667, ans=0.2 2023-11-21 08:00:47,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1420466.6666666667, ans=0.125 2023-11-21 08:00:56,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1420533.3333333333, ans=0.025 2023-11-21 08:01:12,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213100 2023-11-21 08:01:15,625 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8700, loss[loss=0.08734, simple_loss=0.122, pruned_loss=0.01851, audio_tagging_loss=0.007827, over 15273.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09572, pruned_loss=0.01693, audio_tagging_loss=0.009876, over 3032532.98 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:01:52,036 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.710e+01 8.075e+01 9.054e+01 1.029e+02 1.299e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-21 08:01:53,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1420866.6666666667, ans=0.0 2023-11-21 08:02:12,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.97 vs. limit=22.5 2023-11-21 08:02:18,918 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213150 2023-11-21 08:02:21,325 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8750, loss[loss=0.08446, simple_loss=0.1107, pruned_loss=0.01863, audio_tagging_loss=0.01045, over 15645.00 frames. ], tot_loss[loss=0.07515, simple_loss=0.09656, pruned_loss=0.01706, audio_tagging_loss=0.009809, over 3034762.05 frames. ], batch size: 57, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:03:07,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.71 vs. limit=15.0 2023-11-21 08:03:23,197 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213200 2023-11-21 08:03:26,467 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8800, loss[loss=0.06137, simple_loss=0.08629, pruned_loss=0.0108, audio_tagging_loss=0.007432, over 15528.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.0968, pruned_loss=0.01721, audio_tagging_loss=0.009926, over 3036080.80 frames. ], batch size: 57, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 08:03:37,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1421400.0, ans=0.125 2023-11-21 08:03:42,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1421400.0, ans=0.0 2023-11-21 08:04:01,620 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:04:03,664 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.113e+01 8.751e+01 9.814e+01 1.127e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 08:04:04,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.85 vs. limit=6.0 2023-11-21 08:04:27,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213250 2023-11-21 08:04:29,976 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8850, loss[loss=0.09295, simple_loss=0.1186, pruned_loss=0.02406, audio_tagging_loss=0.009576, over 14600.00 frames. ], tot_loss[loss=0.07649, simple_loss=0.09817, pruned_loss=0.01752, audio_tagging_loss=0.009881, over 3039422.51 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:04:34,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1421666.6666666667, ans=0.1 2023-11-21 08:04:40,796 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.98 vs. limit=15.0 2023-11-21 08:04:43,087 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:04:55,500 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:04:56,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1421800.0, ans=0.0 2023-11-21 08:04:56,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1421800.0, ans=0.125 2023-11-21 08:05:28,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1421933.3333333333, ans=0.125 2023-11-21 08:05:33,396 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213300 2023-11-21 08:05:35,825 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8900, loss[loss=0.08918, simple_loss=0.1174, pruned_loss=0.02275, audio_tagging_loss=0.007727, over 15846.00 frames. ], tot_loss[loss=0.07571, simple_loss=0.09751, pruned_loss=0.01727, audio_tagging_loss=0.009688, over 3043101.03 frames. ], batch size: 54, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:05:39,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1422000.0, ans=0.125 2023-11-21 08:05:46,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.38 vs. limit=10.0 2023-11-21 08:05:48,566 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:05:48,979 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0 2023-11-21 08:05:49,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1422066.6666666667, ans=0.125 2023-11-21 08:06:00,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.33 vs. limit=15.0 2023-11-21 08:06:12,889 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.059e+01 8.546e+01 9.604e+01 1.318e+02, threshold=1.709e+02, percent-clipped=0.0 2023-11-21 08:06:14,850 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.24 vs. limit=22.5 2023-11-21 08:06:19,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.00 vs. limit=10.0 2023-11-21 08:06:37,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1422266.6666666667, ans=0.125 2023-11-21 08:06:37,980 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213350 2023-11-21 08:06:40,312 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 8950, loss[loss=0.08175, simple_loss=0.1035, pruned_loss=0.02257, audio_tagging_loss=0.007437, over 16901.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09748, pruned_loss=0.01718, audio_tagging_loss=0.009628, over 3045780.23 frames. ], batch size: 64, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:06:44,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1422333.3333333333, ans=0.125 2023-11-21 08:06:49,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1422333.3333333333, ans=0.125 2023-11-21 08:07:27,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1422533.3333333333, ans=0.125 2023-11-21 08:07:27,468 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.94 vs. limit=12.0 2023-11-21 08:07:42,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213400 2023-11-21 08:07:42,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1422600.0, ans=0.125 2023-11-21 08:07:43,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1422666.6666666667, ans=0.0 2023-11-21 08:07:45,042 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9000, loss[loss=0.09438, simple_loss=0.1205, pruned_loss=0.02617, audio_tagging_loss=0.00797, over 15179.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.09747, pruned_loss=0.01717, audio_tagging_loss=0.00963, over 3049582.74 frames. ], batch size: 57, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:07:45,042 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 08:08:25,288 INFO [train_asr.py:1253] (3/4) Epoch 18, validation: loss=0.06098, simple_loss=0.05248, pruned_loss=0.005341, audio_tagging_loss=0.02939, over 4681554.00 frames. 2023-11-21 08:08:25,289 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 08:08:26,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1422666.6666666667, ans=0.1 2023-11-21 08:08:49,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1422800.0, ans=0.0 2023-11-21 08:08:57,605 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.31 vs. limit=12.0 2023-11-21 08:08:59,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1422800.0, ans=0.125 2023-11-21 08:09:01,757 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.253e+01 9.144e+01 1.038e+02 1.327e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-21 08:09:08,141 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.39 vs. limit=22.5 2023-11-21 08:09:18,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1422933.3333333333, ans=0.125 2023-11-21 08:09:26,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213450 2023-11-21 08:09:29,091 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9050, loss[loss=0.05874, simple_loss=0.07492, pruned_loss=0.01167, audio_tagging_loss=0.009605, over 13913.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.09764, pruned_loss=0.01731, audio_tagging_loss=0.009542, over 3050267.47 frames. ], batch size: 52, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:09:44,797 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:09:49,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1423066.6666666667, ans=0.05 2023-11-21 08:10:18,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1423200.0, ans=0.2 2023-11-21 08:10:19,022 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.98 vs. limit=15.0 2023-11-21 08:10:31,344 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213500 2023-11-21 08:10:31,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1423266.6666666667, ans=0.0 2023-11-21 08:10:33,719 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9100, loss[loss=0.07063, simple_loss=0.08532, pruned_loss=0.0191, audio_tagging_loss=0.008872, over 15460.00 frames. ], tot_loss[loss=0.07569, simple_loss=0.09798, pruned_loss=0.01722, audio_tagging_loss=0.009481, over 3051845.31 frames. ], batch size: 60, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:10:38,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1423333.3333333333, ans=0.2 2023-11-21 08:10:42,013 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:11:05,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.35 vs. limit=15.0 2023-11-21 08:11:11,211 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.196e+01 8.647e+01 9.263e+01 1.186e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-21 08:11:23,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1423533.3333333333, ans=0.2 2023-11-21 08:11:35,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1423600.0, ans=0.125 2023-11-21 08:11:36,900 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213550 2023-11-21 08:11:40,082 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9150, loss[loss=0.07783, simple_loss=0.1012, pruned_loss=0.01818, audio_tagging_loss=0.009068, over 16530.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09803, pruned_loss=0.01733, audio_tagging_loss=0.00946, over 3047814.97 frames. ], batch size: 62, lr: 3.79e-03, grad_scale: 16.0 2023-11-21 08:11:54,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1423733.3333333333, ans=0.0 2023-11-21 08:11:57,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1423733.3333333333, ans=0.125 2023-11-21 08:11:57,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1423733.3333333333, ans=0.0 2023-11-21 08:11:59,358 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.64 vs. limit=22.5 2023-11-21 08:12:10,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1423800.0, ans=0.2 2023-11-21 08:12:40,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1423933.3333333333, ans=0.125 2023-11-21 08:12:42,558 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213600 2023-11-21 08:12:45,243 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9200, loss[loss=0.08443, simple_loss=0.112, pruned_loss=0.02231, audio_tagging_loss=0.0061, over 15041.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09876, pruned_loss=0.01756, audio_tagging_loss=0.009359, over 3049330.67 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 32.0 2023-11-21 08:12:54,455 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.26 vs. limit=15.0 2023-11-21 08:12:55,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1424000.0, ans=0.1 2023-11-21 08:13:01,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1424066.6666666667, ans=0.2 2023-11-21 08:13:02,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1424066.6666666667, ans=0.0 2023-11-21 08:13:16,415 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.76 vs. limit=6.0 2023-11-21 08:13:23,933 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 7.966e+01 8.700e+01 9.351e+01 1.190e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 08:13:24,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1424200.0, ans=0.125 2023-11-21 08:13:32,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1424200.0, ans=0.1 2023-11-21 08:13:40,749 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.96 vs. limit=15.0 2023-11-21 08:13:47,560 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213650 2023-11-21 08:13:49,886 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9250, loss[loss=0.06738, simple_loss=0.08599, pruned_loss=0.01385, audio_tagging_loss=0.01053, over 15252.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09762, pruned_loss=0.01741, audio_tagging_loss=0.009363, over 3053795.07 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:14:14,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1424400.0, ans=0.0 2023-11-21 08:14:18,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1424466.6666666667, ans=0.125 2023-11-21 08:14:48,216 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:14:50,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1424600.0, ans=0.0 2023-11-21 08:14:52,614 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213700 2023-11-21 08:14:54,995 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9300, loss[loss=0.09015, simple_loss=0.1133, pruned_loss=0.02289, audio_tagging_loss=0.01061, over 15560.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09803, pruned_loss=0.01743, audio_tagging_loss=0.009384, over 3049068.16 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:15:02,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1424666.6666666667, ans=0.125 2023-11-21 08:15:14,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1424733.3333333333, ans=0.0 2023-11-21 08:15:17,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.93 vs. limit=12.0 2023-11-21 08:15:19,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1424800.0, ans=0.125 2023-11-21 08:15:31,884 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.285e+01 7.980e+01 8.564e+01 9.426e+01 1.371e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-21 08:15:35,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1424866.6666666667, ans=0.1 2023-11-21 08:15:39,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1424866.6666666667, ans=0.07 2023-11-21 08:15:57,661 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213750 2023-11-21 08:15:57,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1424933.3333333333, ans=0.1 2023-11-21 08:15:59,932 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9350, loss[loss=0.07288, simple_loss=0.08655, pruned_loss=0.01674, audio_tagging_loss=0.01286, over 16574.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09687, pruned_loss=0.01729, audio_tagging_loss=0.009574, over 3050038.46 frames. ], batch size: 64, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:16:06,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1425000.0, ans=0.125 2023-11-21 08:16:06,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1425000.0, ans=0.0 2023-11-21 08:16:13,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1425066.6666666667, ans=0.2 2023-11-21 08:16:18,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1425066.6666666667, ans=0.0 2023-11-21 08:16:28,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1425133.3333333333, ans=0.2 2023-11-21 08:16:30,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.48 vs. limit=6.0 2023-11-21 08:16:50,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1425200.0, ans=0.04949747468305833 2023-11-21 08:17:02,401 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213800 2023-11-21 08:17:05,130 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9400, loss[loss=0.08191, simple_loss=0.1176, pruned_loss=0.01573, audio_tagging_loss=0.007392, over 15457.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09674, pruned_loss=0.01728, audio_tagging_loss=0.009687, over 3039697.43 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:17:15,158 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0 2023-11-21 08:17:23,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1425400.0, ans=0.125 2023-11-21 08:17:26,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1425400.0, ans=0.1 2023-11-21 08:17:43,414 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 8.186e+01 8.902e+01 9.704e+01 1.344e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-21 08:17:59,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1425600.0, ans=0.125 2023-11-21 08:18:08,493 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:18:08,554 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213850 2023-11-21 08:18:10,872 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9450, loss[loss=0.06598, simple_loss=0.07685, pruned_loss=0.01354, audio_tagging_loss=0.01402, over 14617.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09609, pruned_loss=0.01704, audio_tagging_loss=0.009774, over 3041519.52 frames. ], batch size: 59, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:18:15,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1425666.6666666667, ans=0.125 2023-11-21 08:18:18,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1425666.6666666667, ans=0.0 2023-11-21 08:18:28,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1425733.3333333333, ans=10.0 2023-11-21 08:18:56,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1425866.6666666667, ans=0.125 2023-11-21 08:18:57,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1425866.6666666667, ans=0.0 2023-11-21 08:19:13,862 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213900 2023-11-21 08:19:16,295 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9500, loss[loss=0.09532, simple_loss=0.1382, pruned_loss=0.01956, audio_tagging_loss=0.00668, over 16015.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09705, pruned_loss=0.01736, audio_tagging_loss=0.009793, over 3050931.15 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:19:18,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1426000.0, ans=0.125 2023-11-21 08:19:25,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1426000.0, ans=0.0 2023-11-21 08:19:37,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1426066.6666666667, ans=0.95 2023-11-21 08:19:43,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1426133.3333333333, ans=0.125 2023-11-21 08:19:47,385 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.57 vs. limit=15.0 2023-11-21 08:19:52,348 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.79 vs. limit=15.0 2023-11-21 08:19:52,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1426133.3333333333, ans=0.0 2023-11-21 08:19:53,767 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.280e+01 9.020e+01 9.623e+01 1.395e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-21 08:19:56,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1426200.0, ans=0.2 2023-11-21 08:20:03,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1426200.0, ans=0.2 2023-11-21 08:20:04,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1426200.0, ans=0.125 2023-11-21 08:20:04,891 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.90 vs. limit=10.0 2023-11-21 08:20:17,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 213950 2023-11-21 08:20:20,270 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9550, loss[loss=0.07307, simple_loss=0.09198, pruned_loss=0.01622, audio_tagging_loss=0.01087, over 14301.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09692, pruned_loss=0.01725, audio_tagging_loss=0.009913, over 3044228.36 frames. ], batch size: 54, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:20:21,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1426333.3333333333, ans=0.125 2023-11-21 08:20:28,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1426333.3333333333, ans=10.0 2023-11-21 08:20:30,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1426333.3333333333, ans=0.125 2023-11-21 08:20:31,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1426333.3333333333, ans=0.125 2023-11-21 08:20:42,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1426400.0, ans=0.1 2023-11-21 08:21:06,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1426533.3333333333, ans=0.125 2023-11-21 08:21:06,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1426533.3333333333, ans=0.2 2023-11-21 08:21:22,581 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214000 2023-11-21 08:21:22,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1426600.0, ans=0.125 2023-11-21 08:21:25,343 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9600, loss[loss=0.08112, simple_loss=0.1082, pruned_loss=0.01943, audio_tagging_loss=0.007606, over 15305.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09547, pruned_loss=0.01684, audio_tagging_loss=0.009994, over 3041034.85 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:21:43,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1426733.3333333333, ans=0.125 2023-11-21 08:21:50,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1426800.0, ans=0.1 2023-11-21 08:21:53,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1426800.0, ans=0.125 2023-11-21 08:21:59,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1426800.0, ans=0.125 2023-11-21 08:22:02,374 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.40 vs. limit=15.0 2023-11-21 08:22:02,935 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.086e+01 8.937e+01 9.718e+01 1.374e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-21 08:22:25,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1426933.3333333333, ans=0.125 2023-11-21 08:22:29,422 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214050 2023-11-21 08:22:31,860 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9650, loss[loss=0.05311, simple_loss=0.06611, pruned_loss=0.009136, audio_tagging_loss=0.01092, over 14496.00 frames. ], tot_loss[loss=0.07423, simple_loss=0.09509, pruned_loss=0.01677, audio_tagging_loss=0.009921, over 3040049.73 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:22:52,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1427066.6666666667, ans=0.125 2023-11-21 08:22:53,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1427066.6666666667, ans=0.0 2023-11-21 08:23:04,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1427133.3333333333, ans=0.125 2023-11-21 08:23:19,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1427200.0, ans=0.0 2023-11-21 08:23:28,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.39 vs. limit=22.5 2023-11-21 08:23:34,006 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214100 2023-11-21 08:23:36,415 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9700, loss[loss=0.06807, simple_loss=0.08711, pruned_loss=0.01508, audio_tagging_loss=0.009428, over 16788.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.09566, pruned_loss=0.01683, audio_tagging_loss=0.009771, over 3043720.77 frames. ], batch size: 63, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:23:43,419 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.09 vs. limit=12.0 2023-11-21 08:23:49,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1427400.0, ans=0.125 2023-11-21 08:24:10,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.29 vs. limit=15.0 2023-11-21 08:24:16,179 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.525e+01 8.096e+01 8.550e+01 9.320e+01 1.224e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-21 08:24:16,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1427533.3333333333, ans=0.125 2023-11-21 08:24:18,839 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:24:33,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1427600.0, ans=0.1 2023-11-21 08:24:36,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1427600.0, ans=0.1 2023-11-21 08:24:38,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214150 2023-11-21 08:24:41,420 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9750, loss[loss=0.08712, simple_loss=0.1152, pruned_loss=0.02223, audio_tagging_loss=0.007311, over 15389.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09586, pruned_loss=0.01679, audio_tagging_loss=0.009573, over 3035073.82 frames. ], batch size: 62, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:25:08,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0 2023-11-21 08:25:08,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1427800.0, ans=0.125 2023-11-21 08:25:09,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1427800.0, ans=0.0 2023-11-21 08:25:12,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1427800.0, ans=0.125 2023-11-21 08:25:17,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0 2023-11-21 08:25:31,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1427866.6666666667, ans=0.0 2023-11-21 08:25:35,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1427933.3333333333, ans=0.0 2023-11-21 08:25:39,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1427933.3333333333, ans=0.125 2023-11-21 08:25:45,056 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214200 2023-11-21 08:25:47,816 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9800, loss[loss=0.06101, simple_loss=0.08142, pruned_loss=0.01066, audio_tagging_loss=0.00964, over 13886.00 frames. ], tot_loss[loss=0.0755, simple_loss=0.09784, pruned_loss=0.01714, audio_tagging_loss=0.009437, over 3036768.85 frames. ], batch size: 53, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:25:48,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1428000.0, ans=0.125 2023-11-21 08:26:06,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1428066.6666666667, ans=0.1 2023-11-21 08:26:19,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1428133.3333333333, ans=0.1 2023-11-21 08:26:26,815 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.204e+01 8.669e+01 9.345e+01 1.280e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-21 08:26:44,292 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:26:50,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214250 2023-11-21 08:26:52,999 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9850, loss[loss=0.07496, simple_loss=0.09122, pruned_loss=0.01703, audio_tagging_loss=0.01233, over 14468.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09749, pruned_loss=0.01729, audio_tagging_loss=0.009392, over 3034132.53 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:27:08,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1428400.0, ans=0.125 2023-11-21 08:27:32,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1428533.3333333333, ans=0.125 2023-11-21 08:27:37,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=1428533.3333333333, ans=15.0 2023-11-21 08:27:55,029 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214300 2023-11-21 08:27:57,970 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9900, loss[loss=0.08895, simple_loss=0.1201, pruned_loss=0.02246, audio_tagging_loss=0.00645, over 15648.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09717, pruned_loss=0.01713, audio_tagging_loss=0.00934, over 3037069.89 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:28:00,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1428666.6666666667, ans=0.2 2023-11-21 08:28:08,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1428666.6666666667, ans=0.125 2023-11-21 08:28:11,452 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.31 vs. limit=15.0 2023-11-21 08:28:27,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1428800.0, ans=0.0 2023-11-21 08:28:37,139 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.117e+01 8.820e+01 9.379e+01 1.222e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-21 08:28:38,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1428866.6666666667, ans=0.05 2023-11-21 08:29:01,199 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214350 2023-11-21 08:29:02,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1429000.0, ans=0.125 2023-11-21 08:29:03,562 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 9950, loss[loss=0.05526, simple_loss=0.06985, pruned_loss=0.008907, audio_tagging_loss=0.01143, over 14936.00 frames. ], tot_loss[loss=0.07519, simple_loss=0.09723, pruned_loss=0.01724, audio_tagging_loss=0.009331, over 3036540.21 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:29:14,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.73 vs. limit=12.0 2023-11-21 08:29:16,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1429066.6666666667, ans=10.0 2023-11-21 08:29:27,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1429133.3333333333, ans=0.2 2023-11-21 08:29:36,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.24 vs. limit=22.5 2023-11-21 08:30:06,146 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214400 2023-11-21 08:30:08,812 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10000, loss[loss=0.07737, simple_loss=0.1005, pruned_loss=0.01705, audio_tagging_loss=0.01006, over 15323.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09611, pruned_loss=0.01697, audio_tagging_loss=0.00947, over 3040448.14 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 32.0 2023-11-21 08:30:35,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1429466.6666666667, ans=0.125 2023-11-21 08:30:44,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1429466.6666666667, ans=0.1 2023-11-21 08:30:47,783 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.046e+01 8.852e+01 9.481e+01 1.142e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-21 08:30:54,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1429533.3333333333, ans=0.125 2023-11-21 08:31:01,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.76 vs. limit=15.0 2023-11-21 08:31:10,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214450 2023-11-21 08:31:12,997 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10050, loss[loss=0.07877, simple_loss=0.0971, pruned_loss=0.02061, audio_tagging_loss=0.009615, over 14948.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09653, pruned_loss=0.01705, audio_tagging_loss=0.009519, over 3049359.19 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:31:22,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1429666.6666666667, ans=0.125 2023-11-21 08:31:47,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.89 vs. limit=15.0 2023-11-21 08:31:55,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1429866.6666666667, ans=0.0 2023-11-21 08:32:03,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.13 vs. limit=15.0 2023-11-21 08:32:16,201 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214500 2023-11-21 08:32:19,104 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10100, loss[loss=0.05957, simple_loss=0.06972, pruned_loss=0.01315, audio_tagging_loss=0.01156, over 14732.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.096, pruned_loss=0.01687, audio_tagging_loss=0.009581, over 3060962.78 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:32:26,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1430000.0, ans=0.125 2023-11-21 08:32:45,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.73 vs. limit=15.0 2023-11-21 08:32:58,372 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.203e+01 8.739e+01 9.472e+01 1.226e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 08:33:09,603 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:33:12,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1430266.6666666667, ans=0.1 2023-11-21 08:33:20,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214550 2023-11-21 08:33:22,985 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10150, loss[loss=0.06548, simple_loss=0.08412, pruned_loss=0.01342, audio_tagging_loss=0.009991, over 15674.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09634, pruned_loss=0.01701, audio_tagging_loss=0.009581, over 3063340.36 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:33:51,623 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:33:54,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1430466.6666666667, ans=0.0 2023-11-21 08:34:01,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1430533.3333333333, ans=0.1 2023-11-21 08:34:03,146 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.67 vs. limit=15.0 2023-11-21 08:34:06,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1430533.3333333333, ans=0.0 2023-11-21 08:34:11,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1430533.3333333333, ans=0.125 2023-11-21 08:34:23,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1430600.0, ans=0.125 2023-11-21 08:34:25,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214600 2023-11-21 08:34:28,264 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10200, loss[loss=0.06241, simple_loss=0.07094, pruned_loss=0.01267, audio_tagging_loss=0.01426, over 15120.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09632, pruned_loss=0.01695, audio_tagging_loss=0.009704, over 3056889.72 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:34:33,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1430666.6666666667, ans=0.1 2023-11-21 08:34:46,962 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0 2023-11-21 08:34:51,440 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:34:54,444 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.69 vs. limit=22.5 2023-11-21 08:35:02,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1430800.0, ans=0.2 2023-11-21 08:35:05,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1430800.0, ans=0.125 2023-11-21 08:35:08,559 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.283e+01 8.036e+01 8.614e+01 9.487e+01 2.302e+02, threshold=1.723e+02, percent-clipped=1.0 2023-11-21 08:35:09,101 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.58 vs. limit=22.5 2023-11-21 08:35:27,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1430933.3333333333, ans=0.2 2023-11-21 08:35:31,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214650 2023-11-21 08:35:33,542 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10250, loss[loss=0.06569, simple_loss=0.08222, pruned_loss=0.01381, audio_tagging_loss=0.01077, over 14900.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.0969, pruned_loss=0.01716, audio_tagging_loss=0.009784, over 3054383.46 frames. ], batch size: 59, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:35:38,556 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0 2023-11-21 08:35:54,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1431066.6666666667, ans=0.09899494936611666 2023-11-21 08:36:22,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1431200.0, ans=0.1 2023-11-21 08:36:36,412 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214700 2023-11-21 08:36:38,823 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10300, loss[loss=0.0733, simple_loss=0.09599, pruned_loss=0.0164, audio_tagging_loss=0.008906, over 15312.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09698, pruned_loss=0.01715, audio_tagging_loss=0.009788, over 3057178.09 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:36:46,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1431333.3333333333, ans=0.95 2023-11-21 08:37:19,815 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.659e+01 8.199e+01 8.798e+01 9.773e+01 1.568e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-21 08:37:23,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1431533.3333333333, ans=0.125 2023-11-21 08:37:29,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1431600.0, ans=0.0 2023-11-21 08:37:37,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1431600.0, ans=0.0 2023-11-21 08:37:41,043 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214750 2023-11-21 08:37:43,452 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10350, loss[loss=0.06217, simple_loss=0.08046, pruned_loss=0.01204, audio_tagging_loss=0.009904, over 15346.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09622, pruned_loss=0.01686, audio_tagging_loss=0.009958, over 3054162.48 frames. ], batch size: 60, lr: 3.78e-03, grad_scale: 16.0 2023-11-21 08:37:46,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1431666.6666666667, ans=0.125 2023-11-21 08:37:49,886 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.30 vs. limit=22.5 2023-11-21 08:37:53,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1431666.6666666667, ans=0.0 2023-11-21 08:38:05,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1431733.3333333333, ans=0.0 2023-11-21 08:38:07,145 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:38:09,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1431800.0, ans=0.0 2023-11-21 08:38:10,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1431800.0, ans=0.1 2023-11-21 08:38:11,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.18 vs. limit=15.0 2023-11-21 08:38:20,900 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:38:24,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1431866.6666666667, ans=0.125 2023-11-21 08:38:46,795 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214800 2023-11-21 08:38:49,873 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10400, loss[loss=0.1095, simple_loss=0.1436, pruned_loss=0.03084, audio_tagging_loss=0.006835, over 16480.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09541, pruned_loss=0.01686, audio_tagging_loss=0.01007, over 3050133.42 frames. ], batch size: 60, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 08:38:53,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1432000.0, ans=0.07 2023-11-21 08:39:04,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1432066.6666666667, ans=0.125 2023-11-21 08:39:09,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1432066.6666666667, ans=0.025 2023-11-21 08:39:17,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=12.0 2023-11-21 08:39:31,165 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.245e+01 8.104e+01 8.738e+01 9.653e+01 1.281e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 08:39:34,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1432200.0, ans=0.0 2023-11-21 08:39:47,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1432266.6666666667, ans=0.125 2023-11-21 08:39:50,597 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.19 vs. limit=15.0 2023-11-21 08:39:51,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1432266.6666666667, ans=0.0 2023-11-21 08:39:52,331 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214850 2023-11-21 08:39:53,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1432333.3333333333, ans=0.125 2023-11-21 08:39:54,699 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10450, loss[loss=0.07462, simple_loss=0.09752, pruned_loss=0.01816, audio_tagging_loss=0.007705, over 15436.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09553, pruned_loss=0.01678, audio_tagging_loss=0.009951, over 3048961.68 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:39:54,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1432333.3333333333, ans=0.0 2023-11-21 08:40:03,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1432333.3333333333, ans=0.95 2023-11-21 08:40:05,271 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.44 vs. limit=10.0 2023-11-21 08:40:06,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1432400.0, ans=0.0 2023-11-21 08:40:19,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1432466.6666666667, ans=0.2 2023-11-21 08:40:22,637 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.58 vs. limit=10.0 2023-11-21 08:40:27,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.71 vs. limit=15.0 2023-11-21 08:40:32,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1432466.6666666667, ans=0.125 2023-11-21 08:40:42,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1432533.3333333333, ans=0.125 2023-11-21 08:40:56,623 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214900 2023-11-21 08:40:59,115 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10500, loss[loss=0.07802, simple_loss=0.1059, pruned_loss=0.01655, audio_tagging_loss=0.008507, over 15181.00 frames. ], tot_loss[loss=0.07433, simple_loss=0.09535, pruned_loss=0.01679, audio_tagging_loss=0.009869, over 3048661.48 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:41:13,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1432733.3333333333, ans=0.0 2023-11-21 08:41:14,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1432733.3333333333, ans=10.0 2023-11-21 08:41:32,160 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.82 vs. limit=6.0 2023-11-21 08:41:35,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1432800.0, ans=0.125 2023-11-21 08:41:39,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1432866.6666666667, ans=0.125 2023-11-21 08:41:40,097 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.754e+01 7.879e+01 8.434e+01 9.185e+01 1.259e+02, threshold=1.687e+02, percent-clipped=0.0 2023-11-21 08:41:44,591 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.79 vs. limit=22.5 2023-11-21 08:41:48,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.06 vs. limit=22.5 2023-11-21 08:42:01,005 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 214950 2023-11-21 08:42:01,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1432933.3333333333, ans=0.0 2023-11-21 08:42:03,529 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10550, loss[loss=0.07633, simple_loss=0.09263, pruned_loss=0.01581, audio_tagging_loss=0.0142, over 16062.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09589, pruned_loss=0.0169, audio_tagging_loss=0.009821, over 3052571.34 frames. ], batch size: 62, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:42:05,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1433000.0, ans=0.125 2023-11-21 08:42:22,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1433066.6666666667, ans=0.07 2023-11-21 08:42:40,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1433200.0, ans=0.125 2023-11-21 08:43:06,200 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215000 2023-11-21 08:43:08,949 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10600, loss[loss=0.09375, simple_loss=0.1139, pruned_loss=0.0271, audio_tagging_loss=0.009722, over 14728.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09587, pruned_loss=0.01686, audio_tagging_loss=0.009658, over 3042192.97 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:43:09,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1433333.3333333333, ans=0.1 2023-11-21 08:43:10,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1433333.3333333333, ans=0.125 2023-11-21 08:43:10,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.77 vs. limit=15.0 2023-11-21 08:43:24,902 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:43:27,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1433400.0, ans=0.5 2023-11-21 08:43:29,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1433400.0, ans=0.125 2023-11-21 08:43:41,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1433466.6666666667, ans=0.1 2023-11-21 08:43:42,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1433466.6666666667, ans=0.125 2023-11-21 08:43:46,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1433533.3333333333, ans=0.2 2023-11-21 08:43:51,459 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.274e+01 8.887e+01 9.806e+01 1.375e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-21 08:43:58,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1433533.3333333333, ans=0.1 2023-11-21 08:44:03,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.00 vs. limit=15.0 2023-11-21 08:44:03,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1433600.0, ans=0.0 2023-11-21 08:44:04,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1433600.0, ans=0.125 2023-11-21 08:44:05,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1433600.0, ans=0.07 2023-11-21 08:44:09,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215050 2023-11-21 08:44:12,276 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10650, loss[loss=0.07892, simple_loss=0.1061, pruned_loss=0.01673, audio_tagging_loss=0.009124, over 15608.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09649, pruned_loss=0.01706, audio_tagging_loss=0.009494, over 3044333.38 frames. ], batch size: 60, lr: 3.77e-03, grad_scale: 8.0 2023-11-21 08:44:17,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.99 vs. limit=22.5 2023-11-21 08:44:30,530 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:44:32,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.63 vs. limit=22.5 2023-11-21 08:44:49,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1433800.0, ans=0.0 2023-11-21 08:44:54,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1433866.6666666667, ans=0.125 2023-11-21 08:45:14,992 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215100 2023-11-21 08:45:17,255 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10700, loss[loss=0.07032, simple_loss=0.09671, pruned_loss=0.01324, audio_tagging_loss=0.008732, over 14843.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09555, pruned_loss=0.01683, audio_tagging_loss=0.009563, over 3043913.84 frames. ], batch size: 54, lr: 3.77e-03, grad_scale: 8.0 2023-11-21 08:45:30,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1434066.6666666667, ans=0.125 2023-11-21 08:45:51,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1434133.3333333333, ans=0.0 2023-11-21 08:46:00,181 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.074e+01 8.707e+01 9.700e+01 1.286e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-21 08:46:05,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1434200.0, ans=0.125 2023-11-21 08:46:13,216 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=15.0 2023-11-21 08:46:21,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215150 2023-11-21 08:46:23,787 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10750, loss[loss=0.04275, simple_loss=0.04561, pruned_loss=0.007309, audio_tagging_loss=0.01263, over 15380.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09526, pruned_loss=0.01658, audio_tagging_loss=0.009551, over 3047382.15 frames. ], batch size: 61, lr: 3.77e-03, grad_scale: 8.0 2023-11-21 08:46:32,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1434333.3333333333, ans=0.0 2023-11-21 08:46:36,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1434400.0, ans=0.09899494936611666 2023-11-21 08:46:42,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1434400.0, ans=0.0 2023-11-21 08:47:08,974 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.39 vs. limit=15.0 2023-11-21 08:47:10,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1434533.3333333333, ans=0.125 2023-11-21 08:47:25,608 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215200 2023-11-21 08:47:28,310 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10800, loss[loss=0.06539, simple_loss=0.08389, pruned_loss=0.01041, audio_tagging_loss=0.01303, over 16275.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09569, pruned_loss=0.01689, audio_tagging_loss=0.009624, over 3054174.67 frames. ], batch size: 64, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:47:33,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1434666.6666666667, ans=0.125 2023-11-21 08:47:40,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.69 vs. limit=15.0 2023-11-21 08:47:42,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1434733.3333333333, ans=0.125 2023-11-21 08:48:08,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1.whitening_limit, batch_count=1434866.6666666667, ans=10.0 2023-11-21 08:48:11,099 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.512e+01 8.195e+01 8.985e+01 9.543e+01 1.112e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-21 08:48:13,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1434866.6666666667, ans=0.2 2023-11-21 08:48:25,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.65 vs. limit=12.0 2023-11-21 08:48:29,547 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215250 2023-11-21 08:48:32,526 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10850, loss[loss=0.08806, simple_loss=0.1179, pruned_loss=0.0196, audio_tagging_loss=0.009535, over 15182.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09568, pruned_loss=0.01679, audio_tagging_loss=0.009499, over 3053440.75 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:48:36,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1435000.0, ans=0.0 2023-11-21 08:48:49,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1435066.6666666667, ans=0.0 2023-11-21 08:49:10,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1435200.0, ans=0.125 2023-11-21 08:49:13,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1435200.0, ans=0.0 2023-11-21 08:49:33,035 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:49:35,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215300 2023-11-21 08:49:38,446 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10900, loss[loss=0.06892, simple_loss=0.08618, pruned_loss=0.01515, audio_tagging_loss=0.01068, over 15456.00 frames. ], tot_loss[loss=0.07461, simple_loss=0.09614, pruned_loss=0.01697, audio_tagging_loss=0.009572, over 3046938.47 frames. ], batch size: 58, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:49:43,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1435333.3333333333, ans=0.07 2023-11-21 08:49:44,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1435333.3333333333, ans=0.0 2023-11-21 08:49:46,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1435333.3333333333, ans=0.125 2023-11-21 08:49:58,628 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0 2023-11-21 08:49:59,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.82 vs. limit=15.0 2023-11-21 08:50:00,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1435400.0, ans=0.0 2023-11-21 08:50:01,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.69 vs. limit=15.0 2023-11-21 08:50:03,677 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.03 vs. limit=15.0 2023-11-21 08:50:03,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.43 vs. limit=22.5 2023-11-21 08:50:16,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1435533.3333333333, ans=0.015 2023-11-21 08:50:16,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1435533.3333333333, ans=0.0 2023-11-21 08:50:17,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1435533.3333333333, ans=0.125 2023-11-21 08:50:20,528 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.007e+01 7.936e+01 8.755e+01 9.807e+01 1.231e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 08:50:37,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1435600.0, ans=0.1 2023-11-21 08:50:40,630 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215350 2023-11-21 08:50:42,936 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 10950, loss[loss=0.08449, simple_loss=0.1067, pruned_loss=0.02118, audio_tagging_loss=0.009939, over 14370.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09584, pruned_loss=0.01699, audio_tagging_loss=0.009633, over 3036184.28 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:50:53,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1435666.6666666667, ans=0.125 2023-11-21 08:51:10,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1435800.0, ans=0.2 2023-11-21 08:51:10,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1435800.0, ans=0.0 2023-11-21 08:51:14,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.24 vs. limit=15.0 2023-11-21 08:51:44,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215400 2023-11-21 08:51:47,122 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11000, loss[loss=0.09274, simple_loss=0.1118, pruned_loss=0.0251, audio_tagging_loss=0.01173, over 15583.00 frames. ], tot_loss[loss=0.07468, simple_loss=0.09585, pruned_loss=0.01698, audio_tagging_loss=0.009772, over 3032394.21 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:51:56,356 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 08:52:19,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1436133.3333333333, ans=0.05 2023-11-21 08:52:30,016 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.054e+01 8.831e+01 9.756e+01 2.281e+02, threshold=1.766e+02, percent-clipped=1.0 2023-11-21 08:52:50,273 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215450 2023-11-21 08:52:52,529 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11050, loss[loss=0.06452, simple_loss=0.08238, pruned_loss=0.01011, audio_tagging_loss=0.01322, over 14706.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09539, pruned_loss=0.01679, audio_tagging_loss=0.009835, over 3037129.71 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:53:00,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1436333.3333333333, ans=0.0 2023-11-21 08:53:05,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1436400.0, ans=0.1 2023-11-21 08:53:18,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1436466.6666666667, ans=0.125 2023-11-21 08:53:27,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.74 vs. limit=15.0 2023-11-21 08:53:32,146 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.75 vs. limit=15.0 2023-11-21 08:53:36,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1436533.3333333333, ans=0.1 2023-11-21 08:53:37,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1436533.3333333333, ans=0.125 2023-11-21 08:53:44,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.86 vs. limit=15.0 2023-11-21 08:53:46,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1436600.0, ans=0.125 2023-11-21 08:53:54,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215500 2023-11-21 08:53:56,843 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11100, loss[loss=0.08481, simple_loss=0.1063, pruned_loss=0.01894, audio_tagging_loss=0.01271, over 15010.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09675, pruned_loss=0.01708, audio_tagging_loss=0.009927, over 3045948.59 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:54:08,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1436733.3333333333, ans=0.0 2023-11-21 08:54:20,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1436800.0, ans=0.0 2023-11-21 08:54:39,002 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.330e+01 8.983e+01 9.923e+01 1.426e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-21 08:54:39,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1436866.6666666667, ans=0.0 2023-11-21 08:54:51,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.14 vs. limit=6.0 2023-11-21 08:54:58,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215550 2023-11-21 08:54:58,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.06 vs. limit=15.0 2023-11-21 08:55:00,453 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11150, loss[loss=0.07777, simple_loss=0.1012, pruned_loss=0.02008, audio_tagging_loss=0.007075, over 15325.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.09737, pruned_loss=0.01714, audio_tagging_loss=0.009962, over 3053183.47 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0 2023-11-21 08:55:00,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1437000.0, ans=0.125 2023-11-21 08:56:02,939 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215600 2023-11-21 08:56:05,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1437333.3333333333, ans=0.125 2023-11-21 08:56:06,623 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11200, loss[loss=0.06938, simple_loss=0.09037, pruned_loss=0.0128, audio_tagging_loss=0.0114, over 14929.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.097, pruned_loss=0.01707, audio_tagging_loss=0.01003, over 3045513.43 frames. ], batch size: 54, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 08:56:35,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1437466.6666666667, ans=0.125 2023-11-21 08:56:42,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1437466.6666666667, ans=0.125 2023-11-21 08:56:48,510 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.237e+01 8.761e+01 9.554e+01 1.513e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-21 08:56:49,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1437533.3333333333, ans=0.0 2023-11-21 08:56:55,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1437533.3333333333, ans=0.1 2023-11-21 08:57:05,207 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.55 vs. limit=10.0 2023-11-21 08:57:08,363 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215650 2023-11-21 08:57:10,696 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11250, loss[loss=0.07372, simple_loss=0.1028, pruned_loss=0.01255, audio_tagging_loss=0.009763, over 15241.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.09704, pruned_loss=0.01705, audio_tagging_loss=0.009963, over 3049503.36 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 08:57:25,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1437733.3333333333, ans=0.125 2023-11-21 08:57:26,796 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.15 vs. limit=15.0 2023-11-21 08:57:37,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1437800.0, ans=0.1 2023-11-21 08:57:54,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1437866.6666666667, ans=0.125 2023-11-21 08:57:54,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1437866.6666666667, ans=0.0 2023-11-21 08:57:55,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.51 vs. limit=15.0 2023-11-21 08:58:12,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.18 vs. limit=22.5 2023-11-21 08:58:12,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215700 2023-11-21 08:58:15,346 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11300, loss[loss=0.09, simple_loss=0.124, pruned_loss=0.01901, audio_tagging_loss=0.008986, over 14750.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09661, pruned_loss=0.01685, audio_tagging_loss=0.009731, over 3044156.84 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 08:58:31,492 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 08:58:38,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1438066.6666666667, ans=0.0 2023-11-21 08:58:47,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1438133.3333333333, ans=0.1 2023-11-21 08:58:57,853 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 8.122e+01 8.636e+01 9.244e+01 1.479e+02, threshold=1.727e+02, percent-clipped=0.0 2023-11-21 08:59:03,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1438200.0, ans=0.2 2023-11-21 08:59:09,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1438266.6666666667, ans=0.125 2023-11-21 08:59:17,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215750 2023-11-21 08:59:17,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1438266.6666666667, ans=0.0 2023-11-21 08:59:20,443 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11350, loss[loss=0.08413, simple_loss=0.1201, pruned_loss=0.01589, audio_tagging_loss=0.008161, over 15367.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09547, pruned_loss=0.01667, audio_tagging_loss=0.009672, over 3042832.61 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 08:59:31,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1438400.0, ans=0.1 2023-11-21 08:59:44,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1438466.6666666667, ans=0.125 2023-11-21 08:59:46,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1438466.6666666667, ans=0.125 2023-11-21 08:59:46,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1438466.6666666667, ans=0.09899494936611666 2023-11-21 08:59:56,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1438533.3333333333, ans=0.125 2023-11-21 08:59:59,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1438533.3333333333, ans=0.125 2023-11-21 09:00:14,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1438600.0, ans=0.0 2023-11-21 09:00:22,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215800 2023-11-21 09:00:24,882 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11400, loss[loss=0.06792, simple_loss=0.09048, pruned_loss=0.01461, audio_tagging_loss=0.008071, over 13950.00 frames. ], tot_loss[loss=0.07442, simple_loss=0.09599, pruned_loss=0.01685, audio_tagging_loss=0.00957, over 3033981.78 frames. ], batch size: 54, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 09:00:25,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1438666.6666666667, ans=0.125 2023-11-21 09:01:07,971 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.283e+01 9.027e+01 9.488e+01 1.297e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-21 09:01:24,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1438933.3333333333, ans=0.125 2023-11-21 09:01:26,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215850 2023-11-21 09:01:29,041 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11450, loss[loss=0.06086, simple_loss=0.07677, pruned_loss=0.01186, audio_tagging_loss=0.01062, over 14709.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09599, pruned_loss=0.01675, audio_tagging_loss=0.009529, over 3041209.26 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 09:01:45,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1439066.6666666667, ans=0.0 2023-11-21 09:02:02,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1439133.3333333333, ans=0.1 2023-11-21 09:02:32,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215900 2023-11-21 09:02:34,716 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11500, loss[loss=0.08372, simple_loss=0.1164, pruned_loss=0.01795, audio_tagging_loss=0.00755, over 15128.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09683, pruned_loss=0.01693, audio_tagging_loss=0.009529, over 3046127.27 frames. ], batch size: 54, lr: 3.77e-03, grad_scale: 32.0 2023-11-21 09:02:38,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1439333.3333333333, ans=0.125 2023-11-21 09:02:39,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1439333.3333333333, ans=0.2 2023-11-21 09:02:47,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1439400.0, ans=0.1 2023-11-21 09:02:57,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1439400.0, ans=0.2 2023-11-21 09:03:17,259 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.959e+01 7.996e+01 8.548e+01 9.424e+01 1.295e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-21 09:03:17,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1439533.3333333333, ans=0.125 2023-11-21 09:03:29,363 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 09:03:37,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 215950 2023-11-21 09:03:40,096 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11550, loss[loss=0.06348, simple_loss=0.08523, pruned_loss=0.01156, audio_tagging_loss=0.009302, over 14513.00 frames. ], tot_loss[loss=0.07552, simple_loss=0.09786, pruned_loss=0.0171, audio_tagging_loss=0.009488, over 3049081.35 frames. ], batch size: 59, lr: 3.76e-03, grad_scale: 32.0 2023-11-21 09:03:58,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1439733.3333333333, ans=0.125 2023-11-21 09:03:59,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1439733.3333333333, ans=0.125 2023-11-21 09:04:11,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1439800.0, ans=0.0 2023-11-21 09:04:18,444 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 09:04:18,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1439866.6666666667, ans=0.125 2023-11-21 09:04:27,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1439866.6666666667, ans=0.125 2023-11-21 09:04:27,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.63 vs. limit=10.0 2023-11-21 09:04:36,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1439933.3333333333, ans=15.0 2023-11-21 09:04:36,557 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.26 vs. limit=15.0 2023-11-21 09:04:39,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1439933.3333333333, ans=0.125 2023-11-21 09:04:42,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216000 2023-11-21 09:04:47,540 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11600, loss[loss=0.08664, simple_loss=0.09813, pruned_loss=0.0268, audio_tagging_loss=0.01077, over 15949.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.0982, pruned_loss=0.01725, audio_tagging_loss=0.009543, over 3053045.78 frames. ], batch size: 60, lr: 3.76e-03, grad_scale: 32.0 2023-11-21 09:05:28,997 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.81 vs. limit=22.5 2023-11-21 09:05:29,713 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.268e+01 8.853e+01 9.480e+01 1.180e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-21 09:05:36,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.88 vs. limit=10.0 2023-11-21 09:05:49,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216050 2023-11-21 09:05:51,734 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11650, loss[loss=0.09672, simple_loss=0.1259, pruned_loss=0.02466, audio_tagging_loss=0.009116, over 15037.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09903, pruned_loss=0.01753, audio_tagging_loss=0.009514, over 3050771.97 frames. ], batch size: 55, lr: 3.76e-03, grad_scale: 32.0 2023-11-21 09:05:52,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.44 vs. limit=15.0 2023-11-21 09:06:07,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1440400.0, ans=0.125 2023-11-21 09:06:12,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1440400.0, ans=0.2 2023-11-21 09:06:27,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1440466.6666666667, ans=0.1 2023-11-21 09:06:50,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1440600.0, ans=0.125 2023-11-21 09:06:54,302 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216100 2023-11-21 09:06:56,644 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11700, loss[loss=0.09523, simple_loss=0.124, pruned_loss=0.02443, audio_tagging_loss=0.008795, over 14482.00 frames. ], tot_loss[loss=0.07589, simple_loss=0.09797, pruned_loss=0.01735, audio_tagging_loss=0.009556, over 3046465.07 frames. ], batch size: 53, lr: 3.76e-03, grad_scale: 32.0 2023-11-21 09:07:13,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1440733.3333333333, ans=15.0 2023-11-21 09:07:15,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.38 vs. limit=15.0 2023-11-21 09:07:37,846 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 09:07:38,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1440866.6666666667, ans=15.0 2023-11-21 09:07:38,774 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.159e+01 8.719e+01 9.346e+01 1.199e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-21 09:07:39,141 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=6.812e-02 2023-11-21 09:07:40,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1440866.6666666667, ans=0.125 2023-11-21 09:07:43,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1440866.6666666667, ans=0.0 2023-11-21 09:07:52,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1440933.3333333333, ans=0.125 2023-11-21 09:07:57,282 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216150 2023-11-21 09:07:59,737 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11750, loss[loss=0.09212, simple_loss=0.1211, pruned_loss=0.02369, audio_tagging_loss=0.007906, over 15676.00 frames. ], tot_loss[loss=0.07607, simple_loss=0.09813, pruned_loss=0.01738, audio_tagging_loss=0.009622, over 3037516.97 frames. ], batch size: 58, lr: 3.76e-03, grad_scale: 32.0 2023-11-21 09:08:29,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1441133.3333333333, ans=0.125 2023-11-21 09:08:34,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1441133.3333333333, ans=0.125 2023-11-21 09:08:44,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.90 vs. limit=15.0 2023-11-21 09:08:51,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1441266.6666666667, ans=0.025 2023-11-21 09:08:57,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1441266.6666666667, ans=0.0 2023-11-21 09:09:01,158 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216200 2023-11-21 09:09:04,062 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11800, loss[loss=0.06756, simple_loss=0.08318, pruned_loss=0.01015, audio_tagging_loss=0.01582, over 14577.00 frames. ], tot_loss[loss=0.07624, simple_loss=0.09832, pruned_loss=0.01739, audio_tagging_loss=0.009695, over 3042373.21 frames. ], batch size: 54, lr: 3.76e-03, grad_scale: 16.0 2023-11-21 09:09:13,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1441333.3333333333, ans=0.2 2023-11-21 09:09:13,928 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.25 vs. limit=22.5 2023-11-21 09:09:22,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1441400.0, ans=0.125 2023-11-21 09:09:23,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.79 vs. limit=15.0 2023-11-21 09:09:39,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1441466.6666666667, ans=0.0 2023-11-21 09:09:44,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1441533.3333333333, ans=0.125 2023-11-21 09:09:48,134 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 8.145e+01 8.665e+01 9.541e+01 1.453e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 09:09:51,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1441533.3333333333, ans=0.125 2023-11-21 09:09:55,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1441600.0, ans=0.04949747468305833 2023-11-21 09:10:00,016 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.75 vs. limit=15.0 2023-11-21 09:10:07,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216250 2023-11-21 09:10:10,305 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11850, loss[loss=0.08736, simple_loss=0.1084, pruned_loss=0.01812, audio_tagging_loss=0.01503, over 14335.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09818, pruned_loss=0.01739, audio_tagging_loss=0.009789, over 3043588.21 frames. ], batch size: 56, lr: 3.76e-03, grad_scale: 16.0 2023-11-21 09:10:14,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.99 vs. limit=22.5 2023-11-21 09:10:19,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.79 vs. limit=15.0 2023-11-21 09:10:24,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1441733.3333333333, ans=0.125 2023-11-21 09:10:34,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1441800.0, ans=0.1 2023-11-21 09:10:40,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1441800.0, ans=0.0 2023-11-21 09:10:56,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1441866.6666666667, ans=0.0 2023-11-21 09:11:03,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.17 vs. limit=22.5 2023-11-21 09:11:11,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216300 2023-11-21 09:11:14,258 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11900, loss[loss=0.0862, simple_loss=0.1185, pruned_loss=0.01961, audio_tagging_loss=0.00731, over 15488.00 frames. ], tot_loss[loss=0.07608, simple_loss=0.098, pruned_loss=0.01719, audio_tagging_loss=0.009883, over 3045802.20 frames. ], batch size: 60, lr: 3.76e-03, grad_scale: 16.0 2023-11-21 09:11:42,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.10 vs. limit=12.0 2023-11-21 09:11:46,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1442133.3333333333, ans=0.0 2023-11-21 09:11:58,212 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.062e+01 8.187e+01 8.956e+01 9.688e+01 1.192e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-21 09:12:04,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1442266.6666666667, ans=0.125 2023-11-21 09:12:13,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1442266.6666666667, ans=0.0 2023-11-21 09:12:13,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1442266.6666666667, ans=0.2 2023-11-21 09:12:15,543 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216350 2023-11-21 09:12:18,005 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 11950, loss[loss=0.07978, simple_loss=0.1103, pruned_loss=0.01573, audio_tagging_loss=0.008905, over 14991.00 frames. ], tot_loss[loss=0.07575, simple_loss=0.09749, pruned_loss=0.017, audio_tagging_loss=0.01001, over 3047798.49 frames. ], batch size: 54, lr: 3.76e-03, grad_scale: 16.0 2023-11-21 09:12:19,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=1442333.3333333333, ans=0.05 2023-11-21 09:12:51,191 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.45 vs. limit=10.0 2023-11-21 09:12:54,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1442466.6666666667, ans=0.04949747468305833 2023-11-21 09:12:56,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1442533.3333333333, ans=0.1 2023-11-21 09:13:00,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1442533.3333333333, ans=0.125 2023-11-21 09:13:05,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1442533.3333333333, ans=0.125 2023-11-21 09:13:13,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0 2023-11-21 09:13:17,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216400 2023-11-21 09:13:19,968 INFO [train_asr.py:1221] (3/4) Epoch 18, batch 12000, loss[loss=0.09788, simple_loss=0.1274, pruned_loss=0.0258, audio_tagging_loss=0.0084, over 15350.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09671, pruned_loss=0.01699, audio_tagging_loss=0.01008, over 3045545.10 frames. ], batch size: 55, lr: 3.76e-03, grad_scale: 32.0 2023-11-21 09:13:19,968 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 09:14:01,460 INFO [train_asr.py:1253] (3/4) Epoch 18, validation: loss=0.06063, simple_loss=0.05245, pruned_loss=0.005328, audio_tagging_loss=0.02908, over 4681554.00 frames. 2023-11-21 09:14:01,461 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 09:14:03,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1442666.6666666667, ans=0.1 2023-11-21 09:14:15,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1442733.3333333333, ans=0.125 2023-11-21 09:14:18,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1442733.3333333333, ans=0.0 2023-11-21 09:15:03,524 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 0, loss[loss=0.08775, simple_loss=0.1081, pruned_loss=0.01336, audio_tagging_loss=0.02031, over 14965.00 frames. ], tot_loss[loss=0.08775, simple_loss=0.1081, pruned_loss=0.01336, audio_tagging_loss=0.02031, over 14965.00 frames. ], batch size: 55, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:15:03,525 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 09:15:25,121 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9712, 3.1351, 2.9363, 3.1328, 3.3448, 2.5348, 3.3641, 2.5438], device='cuda:3') 2023-11-21 09:15:39,002 INFO [train_asr.py:1253] (3/4) Epoch 19, validation: loss=0.05975, simple_loss=0.05244, pruned_loss=0.005316, audio_tagging_loss=0.02822, over 4681554.00 frames. 2023-11-21 09:15:39,003 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 09:15:39,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.31 vs. limit=15.0 2023-11-21 09:15:52,326 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.140e+01 8.995e+01 9.841e+01 1.386e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-21 09:16:08,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1442953.3333333333, ans=0.125 2023-11-21 09:16:11,940 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216450 2023-11-21 09:16:23,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1443020.0, ans=0.0 2023-11-21 09:16:27,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1443020.0, ans=0.125 2023-11-21 09:16:37,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1443086.6666666667, ans=0.125 2023-11-21 09:16:43,266 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 50, loss[loss=0.06658, simple_loss=0.07857, pruned_loss=0.009285, audio_tagging_loss=0.018, over 14807.00 frames. ], tot_loss[loss=0.08765, simple_loss=0.1013, pruned_loss=0.0183, audio_tagging_loss=0.0187, over 688464.36 frames. ], batch size: 56, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:16:49,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1443153.3333333333, ans=0.125 2023-11-21 09:17:03,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1443220.0, ans=0.0 2023-11-21 09:17:08,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1443286.6666666667, ans=0.0 2023-11-21 09:17:09,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1443286.6666666667, ans=0.2 2023-11-21 09:17:15,874 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216500 2023-11-21 09:17:17,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1443286.6666666667, ans=0.1 2023-11-21 09:17:34,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1443420.0, ans=0.125 2023-11-21 09:17:46,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1443420.0, ans=0.0 2023-11-21 09:17:49,159 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 100, loss[loss=0.08448, simple_loss=0.1072, pruned_loss=0.01348, audio_tagging_loss=0.01739, over 14875.00 frames. ], tot_loss[loss=0.0848, simple_loss=0.09823, pruned_loss=0.01768, audio_tagging_loss=0.01801, over 1202960.17 frames. ], batch size: 54, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:17:53,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1443486.6666666667, ans=0.0 2023-11-21 09:18:02,448 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.866e+01 9.575e+01 1.050e+02 1.381e+02, threshold=1.915e+02, percent-clipped=0.0 2023-11-21 09:18:13,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1443620.0, ans=0.125 2023-11-21 09:18:16,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1443620.0, ans=0.07 2023-11-21 09:18:19,834 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216550 2023-11-21 09:18:25,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.18 vs. limit=15.0 2023-11-21 09:18:41,529 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.35 vs. limit=22.5 2023-11-21 09:18:46,204 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.76 vs. limit=15.0 2023-11-21 09:18:50,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1443753.3333333333, ans=0.125 2023-11-21 09:18:52,662 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 150, loss[loss=0.0724, simple_loss=0.09116, pruned_loss=0.01465, audio_tagging_loss=0.01218, over 15191.00 frames. ], tot_loss[loss=0.08305, simple_loss=0.09863, pruned_loss=0.01778, audio_tagging_loss=0.01596, over 1609373.05 frames. ], batch size: 56, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:19:13,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1443886.6666666667, ans=0.125 2023-11-21 09:19:19,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1443953.3333333333, ans=0.0 2023-11-21 09:19:20,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1443953.3333333333, ans=0.0 2023-11-21 09:19:24,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1443953.3333333333, ans=0.0 2023-11-21 09:19:25,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216600 2023-11-21 09:19:25,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1443953.3333333333, ans=0.0 2023-11-21 09:19:33,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1444020.0, ans=0.125 2023-11-21 09:19:34,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1444020.0, ans=0.125 2023-11-21 09:19:52,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1444086.6666666667, ans=0.0 2023-11-21 09:19:56,866 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 200, loss[loss=0.09946, simple_loss=0.1299, pruned_loss=0.02399, audio_tagging_loss=0.01055, over 15239.00 frames. ], tot_loss[loss=0.081, simple_loss=0.0989, pruned_loss=0.01751, audio_tagging_loss=0.01404, over 1930340.36 frames. ], batch size: 54, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:20:12,117 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.223e+01 8.596e+01 9.461e+01 1.153e+02, threshold=1.719e+02, percent-clipped=0.0 2023-11-21 09:20:13,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1444220.0, ans=0.125 2023-11-21 09:20:14,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1444220.0, ans=0.125 2023-11-21 09:20:20,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1444220.0, ans=0.025 2023-11-21 09:20:29,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216650 2023-11-21 09:20:43,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1444353.3333333333, ans=0.1 2023-11-21 09:20:50,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1444420.0, ans=0.125 2023-11-21 09:20:54,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1444420.0, ans=0.0 2023-11-21 09:21:02,351 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 250, loss[loss=0.09928, simple_loss=0.1275, pruned_loss=0.02633, audio_tagging_loss=0.009182, over 15648.00 frames. ], tot_loss[loss=0.07961, simple_loss=0.099, pruned_loss=0.01742, audio_tagging_loss=0.01268, over 2184288.98 frames. ], batch size: 56, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:21:10,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.85 vs. limit=15.0 2023-11-21 09:21:26,883 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.66 vs. limit=12.0 2023-11-21 09:21:33,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216700 2023-11-21 09:21:41,577 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=12.0 2023-11-21 09:22:00,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1444753.3333333333, ans=0.125 2023-11-21 09:22:06,601 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 300, loss[loss=0.08605, simple_loss=0.1118, pruned_loss=0.01998, audio_tagging_loss=0.01017, over 15737.00 frames. ], tot_loss[loss=0.07859, simple_loss=0.09882, pruned_loss=0.01745, audio_tagging_loss=0.01174, over 2375802.45 frames. ], batch size: 60, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:22:14,558 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.84 vs. limit=22.5 2023-11-21 09:22:19,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1444886.6666666667, ans=0.0 2023-11-21 09:22:19,963 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.255e+01 8.343e+01 8.887e+01 9.711e+01 1.312e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-21 09:22:29,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1444886.6666666667, ans=0.1 2023-11-21 09:22:38,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216750 2023-11-21 09:22:43,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1445020.0, ans=0.125 2023-11-21 09:22:51,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1445020.0, ans=0.0 2023-11-21 09:23:01,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1445086.6666666667, ans=0.1 2023-11-21 09:23:06,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.05 vs. limit=12.0 2023-11-21 09:23:09,906 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 350, loss[loss=0.09282, simple_loss=0.1271, pruned_loss=0.01921, audio_tagging_loss=0.01005, over 15632.00 frames. ], tot_loss[loss=0.07784, simple_loss=0.09898, pruned_loss=0.01728, audio_tagging_loss=0.01107, over 2527011.77 frames. ], batch size: 58, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:23:36,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1445286.6666666667, ans=0.025 2023-11-21 09:23:43,013 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216800 2023-11-21 09:23:48,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1445353.3333333333, ans=0.1 2023-11-21 09:23:55,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1445353.3333333333, ans=0.125 2023-11-21 09:24:01,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1445420.0, ans=0.125 2023-11-21 09:24:11,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1445420.0, ans=0.0 2023-11-21 09:24:15,661 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 400, loss[loss=0.08636, simple_loss=0.1188, pruned_loss=0.01869, audio_tagging_loss=0.008245, over 17043.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.0994, pruned_loss=0.01722, audio_tagging_loss=0.01067, over 2648938.47 frames. ], batch size: 63, lr: 3.66e-03, grad_scale: 32.0 2023-11-21 09:24:29,288 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.137e+01 8.063e+01 8.732e+01 9.531e+01 1.428e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 09:24:34,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.97 vs. limit=15.0 2023-11-21 09:24:41,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1445620.0, ans=0.1 2023-11-21 09:24:47,457 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216850 2023-11-21 09:24:47,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.11 vs. limit=12.0 2023-11-21 09:25:02,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1445686.6666666667, ans=0.1 2023-11-21 09:25:03,242 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.55 vs. limit=15.0 2023-11-21 09:25:19,755 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 450, loss[loss=0.06333, simple_loss=0.08081, pruned_loss=0.01191, audio_tagging_loss=0.01102, over 15340.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09793, pruned_loss=0.01705, audio_tagging_loss=0.01049, over 2736334.53 frames. ], batch size: 59, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:25:26,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1445820.0, ans=0.125 2023-11-21 09:25:28,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1445820.0, ans=0.2 2023-11-21 09:25:37,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1445886.6666666667, ans=0.125 2023-11-21 09:25:39,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1445886.6666666667, ans=0.2 2023-11-21 09:25:44,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1445953.3333333333, ans=0.0 2023-11-21 09:25:52,613 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=22.16 vs. limit=22.5 2023-11-21 09:25:53,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216900 2023-11-21 09:26:03,160 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0 2023-11-21 09:26:12,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1446086.6666666667, ans=0.0 2023-11-21 09:26:19,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1446086.6666666667, ans=0.5 2023-11-21 09:26:24,303 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 500, loss[loss=0.0856, simple_loss=0.1116, pruned_loss=0.02033, audio_tagging_loss=0.009466, over 14649.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.09808, pruned_loss=0.01689, audio_tagging_loss=0.01028, over 2808092.18 frames. ], batch size: 54, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:26:34,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.76 vs. limit=12.0 2023-11-21 09:26:41,040 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.147e+01 8.789e+01 9.630e+01 1.210e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 09:26:53,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1446286.6666666667, ans=0.0 2023-11-21 09:26:56,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 216950 2023-11-21 09:27:03,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1446353.3333333333, ans=0.125 2023-11-21 09:27:03,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0 2023-11-21 09:27:08,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1446353.3333333333, ans=0.125 2023-11-21 09:27:13,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1446353.3333333333, ans=0.125 2023-11-21 09:27:20,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.75 vs. limit=15.0 2023-11-21 09:27:29,715 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 550, loss[loss=0.0757, simple_loss=0.09049, pruned_loss=0.01928, audio_tagging_loss=0.01118, over 13616.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09691, pruned_loss=0.01661, audio_tagging_loss=0.01013, over 2860191.11 frames. ], batch size: 53, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:27:29,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1446486.6666666667, ans=0.0 2023-11-21 09:27:53,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1446620.0, ans=0.2 2023-11-21 09:28:00,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217000 2023-11-21 09:28:00,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1446620.0, ans=0.05 2023-11-21 09:28:00,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1446620.0, ans=0.125 2023-11-21 09:28:02,870 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.53 vs. limit=10.0 2023-11-21 09:28:31,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1446753.3333333333, ans=0.1 2023-11-21 09:28:33,554 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 600, loss[loss=0.04854, simple_loss=0.05569, pruned_loss=0.007016, audio_tagging_loss=0.01368, over 14243.00 frames. ], tot_loss[loss=0.07527, simple_loss=0.09692, pruned_loss=0.01677, audio_tagging_loss=0.01003, over 2900289.12 frames. ], batch size: 53, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:28:37,667 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 09:28:45,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.64 vs. limit=15.0 2023-11-21 09:28:48,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1446886.6666666667, ans=0.125 2023-11-21 09:28:49,012 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.367e+01 7.902e+01 8.372e+01 9.097e+01 1.242e+02, threshold=1.674e+02, percent-clipped=0.0 2023-11-21 09:28:55,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1446886.6666666667, ans=0.125 2023-11-21 09:28:57,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1446886.6666666667, ans=0.1 2023-11-21 09:29:07,206 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217050 2023-11-21 09:29:09,084 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.88 vs. limit=22.5 2023-11-21 09:29:15,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0 2023-11-21 09:29:25,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1447086.6666666667, ans=0.0 2023-11-21 09:29:31,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1447086.6666666667, ans=0.0 2023-11-21 09:29:33,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1447086.6666666667, ans=0.0 2023-11-21 09:29:38,227 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 650, loss[loss=0.06151, simple_loss=0.07788, pruned_loss=0.01184, audio_tagging_loss=0.01073, over 16028.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.0965, pruned_loss=0.01667, audio_tagging_loss=0.01004, over 2930999.34 frames. ], batch size: 61, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:30:10,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217100 2023-11-21 09:30:16,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.85 vs. limit=15.0 2023-11-21 09:30:30,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1447420.0, ans=0.0 2023-11-21 09:30:30,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1447420.0, ans=0.125 2023-11-21 09:30:44,282 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 700, loss[loss=0.07321, simple_loss=0.0944, pruned_loss=0.01526, audio_tagging_loss=0.01075, over 15502.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09712, pruned_loss=0.0169, audio_tagging_loss=0.00995, over 2957358.30 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:30:49,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1447486.6666666667, ans=0.125 2023-11-21 09:30:58,914 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.199e+01 8.920e+01 9.625e+01 1.204e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-21 09:31:01,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1447553.3333333333, ans=0.125 2023-11-21 09:31:09,492 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.53 vs. limit=15.0 2023-11-21 09:31:14,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217150 2023-11-21 09:31:18,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1447620.0, ans=0.125 2023-11-21 09:31:22,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1447686.6666666667, ans=0.04949747468305833 2023-11-21 09:31:29,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1447686.6666666667, ans=0.07 2023-11-21 09:31:33,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1447686.6666666667, ans=0.125 2023-11-21 09:31:47,782 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 750, loss[loss=0.06088, simple_loss=0.07298, pruned_loss=0.01575, audio_tagging_loss=0.008648, over 14613.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09836, pruned_loss=0.01719, audio_tagging_loss=0.009834, over 2983509.56 frames. ], batch size: 59, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:31:56,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.31 vs. limit=22.5 2023-11-21 09:32:20,134 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217200 2023-11-21 09:32:51,795 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 800, loss[loss=0.06805, simple_loss=0.09045, pruned_loss=0.01374, audio_tagging_loss=0.009091, over 15394.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.09998, pruned_loss=0.01748, audio_tagging_loss=0.009757, over 3004939.80 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:33:08,195 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.219e+01 8.917e+01 9.925e+01 1.296e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-21 09:33:08,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1448220.0, ans=0.0 2023-11-21 09:33:13,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1448220.0, ans=0.125 2023-11-21 09:33:24,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217250 2023-11-21 09:33:34,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1448353.3333333333, ans=0.125 2023-11-21 09:33:40,100 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.92 vs. limit=15.0 2023-11-21 09:33:45,127 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.19 vs. limit=15.0 2023-11-21 09:33:57,404 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 850, loss[loss=0.08815, simple_loss=0.1177, pruned_loss=0.01914, audio_tagging_loss=0.01015, over 15908.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.09967, pruned_loss=0.01739, audio_tagging_loss=0.009832, over 3018725.58 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:34:06,217 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.48 vs. limit=15.0 2023-11-21 09:34:07,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.83 vs. limit=22.5 2023-11-21 09:34:17,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1448553.3333333333, ans=0.0 2023-11-21 09:34:28,599 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217300 2023-11-21 09:34:29,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.30 vs. limit=22.5 2023-11-21 09:34:30,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.89 vs. limit=15.0 2023-11-21 09:34:39,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1448686.6666666667, ans=0.0 2023-11-21 09:34:44,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1448686.6666666667, ans=0.125 2023-11-21 09:34:47,697 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.24 vs. limit=15.0 2023-11-21 09:34:55,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1448753.3333333333, ans=0.0 2023-11-21 09:34:59,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.24 vs. limit=12.0 2023-11-21 09:35:01,260 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 900, loss[loss=0.06015, simple_loss=0.0685, pruned_loss=0.01366, audio_tagging_loss=0.01225, over 16330.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09905, pruned_loss=0.01728, audio_tagging_loss=0.00991, over 3028634.95 frames. ], batch size: 61, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:35:01,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1448820.0, ans=0.125 2023-11-21 09:35:15,737 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.060e+01 7.979e+01 8.801e+01 9.598e+01 1.201e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-21 09:35:33,210 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217350 2023-11-21 09:35:38,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1449020.0, ans=0.125 2023-11-21 09:35:44,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1449020.0, ans=0.125 2023-11-21 09:35:50,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1449086.6666666667, ans=0.0 2023-11-21 09:36:04,297 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 950, loss[loss=0.05846, simple_loss=0.0762, pruned_loss=0.0126, audio_tagging_loss=0.007764, over 15952.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09823, pruned_loss=0.01718, audio_tagging_loss=0.0098, over 3028647.15 frames. ], batch size: 60, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:36:04,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.65 vs. limit=15.0 2023-11-21 09:36:25,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1449220.0, ans=0.125 2023-11-21 09:36:35,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1449286.6666666667, ans=0.0 2023-11-21 09:36:36,777 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217400 2023-11-21 09:36:53,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1449353.3333333333, ans=0.125 2023-11-21 09:36:53,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.57 vs. limit=15.0 2023-11-21 09:37:05,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1449420.0, ans=0.125 2023-11-21 09:37:08,033 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1000, loss[loss=0.0588, simple_loss=0.07644, pruned_loss=0.007609, audio_tagging_loss=0.01298, over 15765.00 frames. ], tot_loss[loss=0.07582, simple_loss=0.09806, pruned_loss=0.01719, audio_tagging_loss=0.009605, over 3029828.17 frames. ], batch size: 62, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:37:25,119 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.132e+01 7.890e+01 8.682e+01 9.562e+01 1.074e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-21 09:37:31,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1449553.3333333333, ans=0.125 2023-11-21 09:37:35,082 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 09:37:36,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.27 vs. limit=15.0 2023-11-21 09:37:39,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217450 2023-11-21 09:38:06,815 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.89 vs. limit=22.5 2023-11-21 09:38:08,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1449753.3333333333, ans=0.125 2023-11-21 09:38:12,120 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1050, loss[loss=0.06545, simple_loss=0.08418, pruned_loss=0.01297, audio_tagging_loss=0.01039, over 15100.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09714, pruned_loss=0.01699, audio_tagging_loss=0.009536, over 3035957.61 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:38:14,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1449820.0, ans=0.0 2023-11-21 09:38:16,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1449820.0, ans=0.125 2023-11-21 09:38:23,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.66 vs. limit=15.0 2023-11-21 09:38:34,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1449886.6666666667, ans=0.1 2023-11-21 09:38:35,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1449953.3333333333, ans=0.125 2023-11-21 09:38:35,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1449953.3333333333, ans=0.2 2023-11-21 09:38:43,229 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217500 2023-11-21 09:38:55,637 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 09:39:03,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.15 vs. limit=15.0 2023-11-21 09:39:05,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1450086.6666666667, ans=0.125 2023-11-21 09:39:14,695 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1100, loss[loss=0.08487, simple_loss=0.1125, pruned_loss=0.02176, audio_tagging_loss=0.006838, over 14613.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09645, pruned_loss=0.01681, audio_tagging_loss=0.009523, over 3030116.82 frames. ], batch size: 53, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:39:18,229 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 09:39:31,322 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.109e+01 8.788e+01 9.339e+01 1.148e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 09:39:37,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten.whitening_limit, batch_count=1450220.0, ans=15.0 2023-11-21 09:39:47,984 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217550 2023-11-21 09:39:48,477 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.47 vs. limit=15.0 2023-11-21 09:40:06,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1450420.0, ans=0.0 2023-11-21 09:40:18,402 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1150, loss[loss=0.0776, simple_loss=0.09427, pruned_loss=0.01876, audio_tagging_loss=0.0117, over 16623.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09711, pruned_loss=0.01693, audio_tagging_loss=0.00946, over 3034424.95 frames. ], batch size: 64, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:40:27,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1450486.6666666667, ans=0.0 2023-11-21 09:40:51,355 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217600 2023-11-21 09:41:23,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1450820.0, ans=0.125 2023-11-21 09:41:24,282 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1200, loss[loss=0.1012, simple_loss=0.1368, pruned_loss=0.02533, audio_tagging_loss=0.0074, over 14736.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09674, pruned_loss=0.01682, audio_tagging_loss=0.009437, over 3040764.27 frames. ], batch size: 53, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:41:26,054 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.26 vs. limit=15.0 2023-11-21 09:41:29,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1450820.0, ans=0.07 2023-11-21 09:41:30,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1450820.0, ans=0.125 2023-11-21 09:41:40,317 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.126e+01 8.918e+01 9.759e+01 1.159e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-21 09:41:55,362 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217650 2023-11-21 09:42:16,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1451086.6666666667, ans=0.125 2023-11-21 09:42:21,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1451086.6666666667, ans=0.0 2023-11-21 09:42:27,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1451153.3333333333, ans=0.125 2023-11-21 09:42:28,257 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1250, loss[loss=0.09801, simple_loss=0.1284, pruned_loss=0.02406, audio_tagging_loss=0.009749, over 15418.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09629, pruned_loss=0.01675, audio_tagging_loss=0.009416, over 3032633.99 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:42:28,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1451153.3333333333, ans=0.2 2023-11-21 09:42:28,802 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.55 vs. limit=6.0 2023-11-21 09:42:33,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1451153.3333333333, ans=0.125 2023-11-21 09:42:34,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1451153.3333333333, ans=0.0 2023-11-21 09:42:57,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.64 vs. limit=22.5 2023-11-21 09:43:00,172 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217700 2023-11-21 09:43:17,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1451420.0, ans=0.125 2023-11-21 09:43:19,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.81 vs. limit=15.0 2023-11-21 09:43:20,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1451420.0, ans=0.125 2023-11-21 09:43:20,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.84 vs. limit=15.0 2023-11-21 09:43:21,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1451420.0, ans=0.0 2023-11-21 09:43:23,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1451420.0, ans=0.0 2023-11-21 09:43:31,371 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1300, loss[loss=0.08078, simple_loss=0.1064, pruned_loss=0.02116, audio_tagging_loss=0.006411, over 14380.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.0953, pruned_loss=0.01653, audio_tagging_loss=0.009467, over 3028046.84 frames. ], batch size: 52, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:43:48,495 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.540e+01 8.140e+01 8.673e+01 9.454e+01 1.388e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 09:43:54,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1451553.3333333333, ans=0.0 2023-11-21 09:43:58,453 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.38 vs. limit=6.0 2023-11-21 09:44:01,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1451620.0, ans=0.125 2023-11-21 09:44:03,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217750 2023-11-21 09:44:13,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1451686.6666666667, ans=0.0 2023-11-21 09:44:13,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1451686.6666666667, ans=0.95 2023-11-21 09:44:29,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.43 vs. limit=6.0 2023-11-21 09:44:35,291 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1350, loss[loss=0.07245, simple_loss=0.08898, pruned_loss=0.01676, audio_tagging_loss=0.01119, over 14615.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09451, pruned_loss=0.01648, audio_tagging_loss=0.009577, over 3028099.62 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:44:39,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.28 vs. limit=15.0 2023-11-21 09:45:06,576 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217800 2023-11-21 09:45:21,503 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 09:45:23,357 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.84 vs. limit=15.0 2023-11-21 09:45:38,990 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1400, loss[loss=0.0882, simple_loss=0.111, pruned_loss=0.02312, audio_tagging_loss=0.009572, over 14785.00 frames. ], tot_loss[loss=0.07391, simple_loss=0.09532, pruned_loss=0.01672, audio_tagging_loss=0.009533, over 3035453.28 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:45:54,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1452220.0, ans=0.125 2023-11-21 09:45:56,178 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.285e+01 9.066e+01 9.966e+01 1.256e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-21 09:46:10,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217850 2023-11-21 09:46:24,084 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.12 vs. limit=15.0 2023-11-21 09:46:26,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1452353.3333333333, ans=0.0 2023-11-21 09:46:29,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1452420.0, ans=0.0 2023-11-21 09:46:41,625 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1450, loss[loss=0.07539, simple_loss=0.09344, pruned_loss=0.01881, audio_tagging_loss=0.009854, over 15064.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09562, pruned_loss=0.01678, audio_tagging_loss=0.009667, over 3038329.94 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:46:47,754 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=23.19 vs. limit=22.5 2023-11-21 09:46:52,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1452486.6666666667, ans=0.0 2023-11-21 09:46:58,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1452553.3333333333, ans=0.125 2023-11-21 09:47:00,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=12.0 2023-11-21 09:47:01,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.90 vs. limit=10.0 2023-11-21 09:47:02,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1452553.3333333333, ans=0.1 2023-11-21 09:47:03,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1452553.3333333333, ans=0.125 2023-11-21 09:47:04,028 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 09:47:14,048 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217900 2023-11-21 09:47:36,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1452753.3333333333, ans=0.2 2023-11-21 09:47:46,034 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1500, loss[loss=0.06985, simple_loss=0.08963, pruned_loss=0.01455, audio_tagging_loss=0.01048, over 14730.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09581, pruned_loss=0.01686, audio_tagging_loss=0.009676, over 3027902.34 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:47:51,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1452820.0, ans=0.125 2023-11-21 09:47:56,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1452820.0, ans=0.95 2023-11-21 09:48:03,705 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.762e+01 8.061e+01 8.622e+01 9.502e+01 1.412e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-21 09:48:09,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.55 vs. limit=15.0 2023-11-21 09:48:17,343 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 217950 2023-11-21 09:48:25,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.57 vs. limit=15.0 2023-11-21 09:48:36,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=1453086.6666666667, ans=0.025 2023-11-21 09:48:49,039 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1550, loss[loss=0.07378, simple_loss=0.09864, pruned_loss=0.01251, audio_tagging_loss=0.01195, over 15555.00 frames. ], tot_loss[loss=0.07513, simple_loss=0.09681, pruned_loss=0.01697, audio_tagging_loss=0.00975, over 3029803.24 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 16.0 2023-11-21 09:49:21,207 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=15.0 2023-11-21 09:49:21,850 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218000 2023-11-21 09:49:22,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1453286.6666666667, ans=0.125 2023-11-21 09:49:29,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1453353.3333333333, ans=0.0 2023-11-21 09:49:36,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1453353.3333333333, ans=0.125 2023-11-21 09:49:53,195 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1600, loss[loss=0.0751, simple_loss=0.08983, pruned_loss=0.01978, audio_tagging_loss=0.0104, over 16103.00 frames. ], tot_loss[loss=0.07561, simple_loss=0.09731, pruned_loss=0.01714, audio_tagging_loss=0.00981, over 3034401.06 frames. ], batch size: 61, lr: 3.65e-03, grad_scale: 32.0 2023-11-21 09:49:54,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1453486.6666666667, ans=0.125 2023-11-21 09:49:59,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1453486.6666666667, ans=0.125 2023-11-21 09:50:11,475 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.287e+01 9.004e+01 9.532e+01 2.614e+02, threshold=1.801e+02, percent-clipped=1.0 2023-11-21 09:50:25,386 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218050 2023-11-21 09:50:33,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1453686.6666666667, ans=0.125 2023-11-21 09:50:40,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1453686.6666666667, ans=0.125 2023-11-21 09:50:42,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1453686.6666666667, ans=0.125 2023-11-21 09:50:57,614 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1650, loss[loss=0.07221, simple_loss=0.08864, pruned_loss=0.01826, audio_tagging_loss=0.009638, over 14732.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09747, pruned_loss=0.0171, audio_tagging_loss=0.00981, over 3043097.84 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 32.0 2023-11-21 09:50:59,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1453820.0, ans=0.125 2023-11-21 09:51:03,232 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.07 vs. limit=15.0 2023-11-21 09:51:06,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1453820.0, ans=0.07 2023-11-21 09:51:17,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=15.30 vs. limit=15.0 2023-11-21 09:51:18,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1453886.6666666667, ans=0.125 2023-11-21 09:51:20,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0 2023-11-21 09:51:29,043 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218100 2023-11-21 09:51:31,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1453953.3333333333, ans=0.2 2023-11-21 09:51:38,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.86 vs. limit=10.0 2023-11-21 09:51:42,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1454020.0, ans=0.125 2023-11-21 09:52:01,488 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1700, loss[loss=0.07826, simple_loss=0.1017, pruned_loss=0.01758, audio_tagging_loss=0.009854, over 15152.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09713, pruned_loss=0.01699, audio_tagging_loss=0.009795, over 3038069.59 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 32.0 2023-11-21 09:52:16,519 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.72 vs. limit=15.0 2023-11-21 09:52:19,397 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.104e+01 8.594e+01 9.360e+01 1.130e+02, threshold=1.719e+02, percent-clipped=0.0 2023-11-21 09:52:20,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1454220.0, ans=0.2 2023-11-21 09:52:29,275 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.12 vs. limit=15.0 2023-11-21 09:52:33,611 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218150 2023-11-21 09:52:51,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1454420.0, ans=0.1 2023-11-21 09:53:01,310 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=13.43 vs. limit=15.0 2023-11-21 09:53:05,409 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1750, loss[loss=0.07528, simple_loss=0.09565, pruned_loss=0.0145, audio_tagging_loss=0.01296, over 15307.00 frames. ], tot_loss[loss=0.07569, simple_loss=0.09752, pruned_loss=0.01715, audio_tagging_loss=0.00978, over 3044345.45 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 32.0 2023-11-21 09:53:18,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1454553.3333333333, ans=0.125 2023-11-21 09:53:21,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1454553.3333333333, ans=0.125 2023-11-21 09:53:24,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1454553.3333333333, ans=0.125 2023-11-21 09:53:37,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218200 2023-11-21 09:53:41,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1454620.0, ans=0.125 2023-11-21 09:53:59,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1454753.3333333333, ans=0.0 2023-11-21 09:54:00,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1454753.3333333333, ans=0.0 2023-11-21 09:54:06,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1454753.3333333333, ans=0.1 2023-11-21 09:54:09,250 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1800, loss[loss=0.06166, simple_loss=0.0859, pruned_loss=0.01108, audio_tagging_loss=0.007632, over 16765.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.09684, pruned_loss=0.01699, audio_tagging_loss=0.009656, over 3048515.65 frames. ], batch size: 66, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 09:54:15,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1454820.0, ans=0.1 2023-11-21 09:54:19,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1454820.0, ans=0.2 2023-11-21 09:54:23,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1454886.6666666667, ans=0.0 2023-11-21 09:54:28,213 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 7.963e+01 8.409e+01 9.319e+01 2.355e+02, threshold=1.682e+02, percent-clipped=0.0 2023-11-21 09:54:39,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1454953.3333333333, ans=0.07 2023-11-21 09:54:41,232 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218250 2023-11-21 09:54:47,720 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.35 vs. limit=15.0 2023-11-21 09:55:08,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.50 vs. limit=15.0 2023-11-21 09:55:13,332 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1850, loss[loss=0.0785, simple_loss=0.1022, pruned_loss=0.01837, audio_tagging_loss=0.009029, over 14987.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09667, pruned_loss=0.01695, audio_tagging_loss=0.009656, over 3041187.94 frames. ], batch size: 54, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 09:55:29,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1455220.0, ans=0.125 2023-11-21 09:55:32,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1455220.0, ans=0.0 2023-11-21 09:55:45,238 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218300 2023-11-21 09:55:52,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1455353.3333333333, ans=0.125 2023-11-21 09:55:52,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1455353.3333333333, ans=0.0 2023-11-21 09:56:01,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1455353.3333333333, ans=0.125 2023-11-21 09:56:13,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.15 vs. limit=22.5 2023-11-21 09:56:16,178 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1900, loss[loss=0.0633, simple_loss=0.08847, pruned_loss=0.01029, audio_tagging_loss=0.008777, over 16077.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.0974, pruned_loss=0.01692, audio_tagging_loss=0.009605, over 3050485.92 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 09:56:27,117 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.55 vs. limit=15.0 2023-11-21 09:56:28,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1455553.3333333333, ans=0.1 2023-11-21 09:56:28,472 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.73 vs. limit=22.5 2023-11-21 09:56:35,908 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.400e+01 8.910e+01 9.667e+01 1.226e+02, threshold=1.782e+02, percent-clipped=1.0 2023-11-21 09:56:48,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218350 2023-11-21 09:56:48,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1455620.0, ans=0.1 2023-11-21 09:56:56,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1455686.6666666667, ans=0.125 2023-11-21 09:57:06,899 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.78 vs. limit=15.0 2023-11-21 09:57:08,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1455753.3333333333, ans=0.125 2023-11-21 09:57:20,927 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 1950, loss[loss=0.04244, simple_loss=0.05013, pruned_loss=0.005305, audio_tagging_loss=0.01208, over 14773.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09719, pruned_loss=0.01686, audio_tagging_loss=0.009606, over 3050144.38 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 09:57:33,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1455886.6666666667, ans=0.05 2023-11-21 09:57:48,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1455953.3333333333, ans=0.1 2023-11-21 09:57:51,119 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 09:57:52,315 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218400 2023-11-21 09:58:07,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1456020.0, ans=0.0 2023-11-21 09:58:10,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1456020.0, ans=0.0 2023-11-21 09:58:21,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1456086.6666666667, ans=0.125 2023-11-21 09:58:25,230 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2000, loss[loss=0.08271, simple_loss=0.1055, pruned_loss=0.0163, audio_tagging_loss=0.01364, over 14687.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09694, pruned_loss=0.01691, audio_tagging_loss=0.009711, over 3050497.99 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 32.0 2023-11-21 09:58:43,538 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.117e+01 8.612e+01 9.115e+01 1.173e+02, threshold=1.722e+02, percent-clipped=0.0 2023-11-21 09:58:44,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.59 vs. limit=15.0 2023-11-21 09:58:56,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1456286.6666666667, ans=0.125 2023-11-21 09:58:57,066 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218450 2023-11-21 09:59:13,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1456353.3333333333, ans=0.0 2023-11-21 09:59:15,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1456420.0, ans=10.0 2023-11-21 09:59:15,623 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.59 vs. limit=15.0 2023-11-21 09:59:16,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1456420.0, ans=0.2 2023-11-21 09:59:28,353 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2050, loss[loss=0.08304, simple_loss=0.1074, pruned_loss=0.02201, audio_tagging_loss=0.007335, over 15425.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09729, pruned_loss=0.01717, audio_tagging_loss=0.009669, over 3047833.56 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 32.0 2023-11-21 09:59:28,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1456486.6666666667, ans=0.125 2023-11-21 09:59:35,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1456486.6666666667, ans=0.125 2023-11-21 09:59:47,058 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 09:59:50,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1456553.3333333333, ans=0.0 2023-11-21 09:59:50,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1456553.3333333333, ans=0.04949747468305833 2023-11-21 09:59:57,758 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.70 vs. limit=22.5 2023-11-21 10:00:00,943 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218500 2023-11-21 10:00:03,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1456620.0, ans=0.0 2023-11-21 10:00:04,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1456620.0, ans=0.0 2023-11-21 10:00:09,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.50 vs. limit=15.0 2023-11-21 10:00:20,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1456753.3333333333, ans=0.125 2023-11-21 10:00:28,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1456753.3333333333, ans=0.0 2023-11-21 10:00:31,893 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2100, loss[loss=0.07757, simple_loss=0.1054, pruned_loss=0.01563, audio_tagging_loss=0.009248, over 15398.00 frames. ], tot_loss[loss=0.07556, simple_loss=0.09732, pruned_loss=0.01725, audio_tagging_loss=0.00965, over 3046605.36 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:00:52,762 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.577e+01 8.022e+01 8.596e+01 9.289e+01 1.125e+02, threshold=1.719e+02, percent-clipped=0.0 2023-11-21 10:01:03,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1456953.3333333333, ans=0.125 2023-11-21 10:01:04,082 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218550 2023-11-21 10:01:36,271 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2150, loss[loss=0.06965, simple_loss=0.08602, pruned_loss=0.01767, audio_tagging_loss=0.008967, over 14129.00 frames. ], tot_loss[loss=0.0755, simple_loss=0.09755, pruned_loss=0.01714, audio_tagging_loss=0.009582, over 3051936.81 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:01:40,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1457153.3333333333, ans=0.0 2023-11-21 10:01:42,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1457153.3333333333, ans=0.125 2023-11-21 10:01:48,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1457220.0, ans=0.125 2023-11-21 10:01:53,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1457220.0, ans=0.0 2023-11-21 10:02:03,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1457286.6666666667, ans=0.125 2023-11-21 10:02:07,002 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218600 2023-11-21 10:02:14,327 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:02:39,306 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2200, loss[loss=0.05149, simple_loss=0.06437, pruned_loss=0.01116, audio_tagging_loss=0.00814, over 14614.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09799, pruned_loss=0.01698, audio_tagging_loss=0.009555, over 3055363.15 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:02:41,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1457486.6666666667, ans=0.0 2023-11-21 10:02:44,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1457486.6666666667, ans=0.125 2023-11-21 10:02:55,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.92 vs. limit=15.0 2023-11-21 10:02:59,799 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.151e+01 8.877e+01 9.686e+01 1.220e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-21 10:03:12,489 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218650 2023-11-21 10:03:33,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1457753.3333333333, ans=0.1 2023-11-21 10:03:36,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1457753.3333333333, ans=0.125 2023-11-21 10:03:43,228 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2250, loss[loss=0.07173, simple_loss=0.104, pruned_loss=0.01207, audio_tagging_loss=0.007671, over 15996.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09731, pruned_loss=0.01676, audio_tagging_loss=0.009572, over 3056580.60 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:03:47,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.72 vs. limit=15.0 2023-11-21 10:04:00,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1457886.6666666667, ans=0.1 2023-11-21 10:04:05,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1457886.6666666667, ans=0.0 2023-11-21 10:04:16,216 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218700 2023-11-21 10:04:16,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.02 vs. limit=22.5 2023-11-21 10:04:18,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1457953.3333333333, ans=0.125 2023-11-21 10:04:40,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1458086.6666666667, ans=0.0 2023-11-21 10:04:42,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1458086.6666666667, ans=0.125 2023-11-21 10:04:42,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1458086.6666666667, ans=0.1 2023-11-21 10:04:48,596 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.20 vs. limit=15.0 2023-11-21 10:04:49,113 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2300, loss[loss=0.0676, simple_loss=0.09204, pruned_loss=0.0131, audio_tagging_loss=0.008484, over 15467.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09666, pruned_loss=0.0166, audio_tagging_loss=0.009707, over 3055626.42 frames. ], batch size: 61, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:05:10,245 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.131e+01 8.990e+01 9.981e+01 1.492e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-21 10:05:20,299 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218750 2023-11-21 10:05:21,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1458286.6666666667, ans=0.125 2023-11-21 10:05:45,478 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:05:52,964 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2350, loss[loss=0.08427, simple_loss=0.1025, pruned_loss=0.02252, audio_tagging_loss=0.01052, over 14852.00 frames. ], tot_loss[loss=0.07528, simple_loss=0.09721, pruned_loss=0.01691, audio_tagging_loss=0.009764, over 3048807.63 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 8.0 2023-11-21 10:05:58,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.17 vs. limit=15.0 2023-11-21 10:06:22,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1458620.0, ans=0.1 2023-11-21 10:06:26,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218800 2023-11-21 10:06:29,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.18 vs. limit=15.0 2023-11-21 10:06:39,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0 2023-11-21 10:06:45,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1458753.3333333333, ans=0.125 2023-11-21 10:06:54,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1458753.3333333333, ans=0.2 2023-11-21 10:06:57,870 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2400, loss[loss=0.05954, simple_loss=0.07429, pruned_loss=0.01146, audio_tagging_loss=0.01094, over 16008.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09688, pruned_loss=0.0168, audio_tagging_loss=0.009862, over 3051263.00 frames. ], batch size: 60, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:07:01,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1458820.0, ans=0.125 2023-11-21 10:07:15,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1458886.6666666667, ans=0.2 2023-11-21 10:07:20,777 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.958e+01 7.990e+01 8.660e+01 9.489e+01 1.183e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 10:07:30,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218850 2023-11-21 10:07:53,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1459086.6666666667, ans=0.125 2023-11-21 10:07:53,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1459086.6666666667, ans=10.0 2023-11-21 10:08:02,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1459153.3333333333, ans=0.2 2023-11-21 10:08:03,728 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2450, loss[loss=0.06587, simple_loss=0.08258, pruned_loss=0.01424, audio_tagging_loss=0.01034, over 15329.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09727, pruned_loss=0.01685, audio_tagging_loss=0.00993, over 3053261.54 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:08:35,241 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218900 2023-11-21 10:08:48,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1459353.3333333333, ans=0.0 2023-11-21 10:09:08,355 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2500, loss[loss=0.08043, simple_loss=0.1042, pruned_loss=0.01934, audio_tagging_loss=0.008997, over 15124.00 frames. ], tot_loss[loss=0.07564, simple_loss=0.09769, pruned_loss=0.01696, audio_tagging_loss=0.009831, over 3054880.55 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:09:10,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.97 vs. limit=10.0 2023-11-21 10:09:29,204 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.48 vs. limit=15.0 2023-11-21 10:09:29,576 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 8.078e+01 8.779e+01 9.710e+01 1.406e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-21 10:09:41,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 218950 2023-11-21 10:09:46,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1459686.6666666667, ans=0.125 2023-11-21 10:09:49,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1459686.6666666667, ans=0.1 2023-11-21 10:09:56,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1459686.6666666667, ans=0.0 2023-11-21 10:10:06,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1459753.3333333333, ans=0.035 2023-11-21 10:10:12,686 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2550, loss[loss=0.0664, simple_loss=0.08756, pruned_loss=0.01584, audio_tagging_loss=0.006779, over 14791.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09763, pruned_loss=0.01709, audio_tagging_loss=0.009814, over 3062539.21 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:10:26,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1459886.6666666667, ans=0.05 2023-11-21 10:10:38,424 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.00 vs. limit=10.0 2023-11-21 10:10:45,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219000 2023-11-21 10:10:54,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1460020.0, ans=0.125 2023-11-21 10:11:09,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1460086.6666666667, ans=0.125 2023-11-21 10:11:17,686 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2600, loss[loss=0.07629, simple_loss=0.1039, pruned_loss=0.01654, audio_tagging_loss=0.007788, over 15339.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09641, pruned_loss=0.01674, audio_tagging_loss=0.009725, over 3056972.53 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:11:39,749 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.043e+01 8.700e+01 9.465e+01 1.335e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 10:11:41,403 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:11:45,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.02 vs. limit=10.0 2023-11-21 10:11:46,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1460286.6666666667, ans=0.125 2023-11-21 10:11:47,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1460286.6666666667, ans=0.1 2023-11-21 10:11:49,698 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219050 2023-11-21 10:11:53,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1460286.6666666667, ans=0.2 2023-11-21 10:12:07,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1460353.3333333333, ans=0.2 2023-11-21 10:12:22,961 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2650, loss[loss=0.06541, simple_loss=0.08881, pruned_loss=0.01139, audio_tagging_loss=0.009609, over 15372.00 frames. ], tot_loss[loss=0.07503, simple_loss=0.09686, pruned_loss=0.01698, audio_tagging_loss=0.009618, over 3053648.83 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:12:27,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.75 vs. limit=6.0 2023-11-21 10:12:55,106 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219100 2023-11-21 10:13:04,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1460686.6666666667, ans=0.0 2023-11-21 10:13:14,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1460753.3333333333, ans=0.0 2023-11-21 10:13:18,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1460753.3333333333, ans=0.1 2023-11-21 10:13:19,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1460753.3333333333, ans=0.0 2023-11-21 10:13:22,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1460753.3333333333, ans=0.0 2023-11-21 10:13:26,498 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2700, loss[loss=0.07962, simple_loss=0.1075, pruned_loss=0.0153, audio_tagging_loss=0.01057, over 14686.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09672, pruned_loss=0.017, audio_tagging_loss=0.009547, over 3049965.28 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:13:26,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1460820.0, ans=0.95 2023-11-21 10:13:34,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1460820.0, ans=0.125 2023-11-21 10:13:48,925 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.184e+01 8.729e+01 9.389e+01 1.078e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 10:13:49,859 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.57 vs. limit=10.0 2023-11-21 10:13:58,751 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219150 2023-11-21 10:14:03,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.91 vs. limit=15.0 2023-11-21 10:14:20,767 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0 2023-11-21 10:14:26,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1461086.6666666667, ans=0.1 2023-11-21 10:14:31,246 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2750, loss[loss=0.09227, simple_loss=0.1142, pruned_loss=0.02637, audio_tagging_loss=0.008823, over 15539.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09607, pruned_loss=0.01682, audio_tagging_loss=0.009542, over 3053094.07 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 16.0 2023-11-21 10:15:02,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219200 2023-11-21 10:15:06,445 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.71 vs. limit=15.0 2023-11-21 10:15:24,965 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.83 vs. limit=22.5 2023-11-21 10:15:26,984 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:15:35,663 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2800, loss[loss=0.0766, simple_loss=0.09938, pruned_loss=0.01917, audio_tagging_loss=0.007746, over 15436.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09642, pruned_loss=0.01679, audio_tagging_loss=0.009537, over 3045276.61 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 32.0 2023-11-21 10:15:46,493 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.33 vs. limit=22.5 2023-11-21 10:15:56,834 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.792e+01 8.334e+01 9.153e+01 1.021e+02 1.703e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-21 10:15:57,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1461553.3333333333, ans=0.125 2023-11-21 10:16:02,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1461620.0, ans=0.2 2023-11-21 10:16:08,004 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219250 2023-11-21 10:16:10,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.90 vs. limit=15.0 2023-11-21 10:16:22,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1461686.6666666667, ans=0.0 2023-11-21 10:16:32,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1461753.3333333333, ans=0.125 2023-11-21 10:16:35,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.13 vs. limit=15.0 2023-11-21 10:16:39,979 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2850, loss[loss=0.08556, simple_loss=0.1093, pruned_loss=0.02095, audio_tagging_loss=0.009941, over 15817.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09648, pruned_loss=0.0168, audio_tagging_loss=0.009597, over 3038783.61 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:16:46,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1461820.0, ans=0.125 2023-11-21 10:17:12,678 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219300 2023-11-21 10:17:12,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1461953.3333333333, ans=0.125 2023-11-21 10:17:45,353 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2900, loss[loss=0.08633, simple_loss=0.1026, pruned_loss=0.02533, audio_tagging_loss=0.009705, over 14146.00 frames. ], tot_loss[loss=0.07429, simple_loss=0.09604, pruned_loss=0.01676, audio_tagging_loss=0.009511, over 3039603.91 frames. ], batch size: 54, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:17:49,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1462153.3333333333, ans=0.025 2023-11-21 10:18:03,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1462220.0, ans=0.5 2023-11-21 10:18:08,043 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.530e+01 8.282e+01 8.695e+01 9.389e+01 1.119e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 10:18:17,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219350 2023-11-21 10:18:19,813 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:18:35,696 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:18:49,481 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 2950, loss[loss=0.07318, simple_loss=0.09959, pruned_loss=0.01523, audio_tagging_loss=0.008156, over 14602.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09431, pruned_loss=0.01651, audio_tagging_loss=0.009667, over 3039527.60 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:18:54,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1462486.6666666667, ans=0.0 2023-11-21 10:19:21,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219400 2023-11-21 10:19:52,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1462820.0, ans=0.1 2023-11-21 10:19:53,234 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3000, loss[loss=0.07168, simple_loss=0.09624, pruned_loss=0.01228, audio_tagging_loss=0.01129, over 16792.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09472, pruned_loss=0.01652, audio_tagging_loss=0.009712, over 3037659.49 frames. ], batch size: 61, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:19:53,235 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 10:20:09,777 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.5770, 1.4411, 3.3079, 3.3877, 3.1826, 3.3028, 3.2590, 3.0166], device='cuda:3') 2023-11-21 10:20:14,818 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.0883, 3.6137, 3.9727, 3.7561], device='cuda:3') 2023-11-21 10:20:28,311 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2237, 4.1572, 4.4030, 4.4289], device='cuda:3') 2023-11-21 10:20:32,672 INFO [train_asr.py:1253] (3/4) Epoch 19, validation: loss=0.05961, simple_loss=0.05235, pruned_loss=0.005214, audio_tagging_loss=0.02822, over 4681554.00 frames. 2023-11-21 10:20:32,673 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 10:20:37,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1462820.0, ans=0.0 2023-11-21 10:20:55,275 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.194e+01 8.910e+01 9.878e+01 1.451e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-21 10:21:03,322 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.39 vs. limit=15.0 2023-11-21 10:21:03,891 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219450 2023-11-21 10:21:05,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1462953.3333333333, ans=0.07 2023-11-21 10:21:36,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1463153.3333333333, ans=0.125 2023-11-21 10:21:37,032 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3050, loss[loss=0.07874, simple_loss=0.09246, pruned_loss=0.02008, audio_tagging_loss=0.01243, over 13951.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09519, pruned_loss=0.01663, audio_tagging_loss=0.009843, over 3037685.08 frames. ], batch size: 52, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:22:04,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1463286.6666666667, ans=0.2 2023-11-21 10:22:09,583 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219500 2023-11-21 10:22:15,109 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:22:27,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1463420.0, ans=0.0 2023-11-21 10:22:38,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1463420.0, ans=0.125 2023-11-21 10:22:40,801 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3100, loss[loss=0.06566, simple_loss=0.07551, pruned_loss=0.01684, audio_tagging_loss=0.01107, over 14681.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.09501, pruned_loss=0.01661, audio_tagging_loss=0.009847, over 3041371.31 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:22:44,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1463486.6666666667, ans=0.2 2023-11-21 10:22:49,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1463486.6666666667, ans=0.1 2023-11-21 10:22:50,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.65 vs. limit=6.0 2023-11-21 10:23:05,419 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.202e+01 8.849e+01 9.632e+01 1.395e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-21 10:23:13,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.47 vs. limit=22.5 2023-11-21 10:23:14,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219550 2023-11-21 10:23:31,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1463753.3333333333, ans=0.1 2023-11-21 10:23:32,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1463753.3333333333, ans=0.2 2023-11-21 10:23:45,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1463820.0, ans=0.2 2023-11-21 10:23:46,191 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3150, loss[loss=0.08063, simple_loss=0.1088, pruned_loss=0.01928, audio_tagging_loss=0.006942, over 15606.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09599, pruned_loss=0.01696, audio_tagging_loss=0.00988, over 3041432.65 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:23:51,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1463820.0, ans=0.125 2023-11-21 10:24:17,455 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219600 2023-11-21 10:24:50,981 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3200, loss[loss=0.07121, simple_loss=0.0869, pruned_loss=0.01994, audio_tagging_loss=0.007824, over 14149.00 frames. ], tot_loss[loss=0.07461, simple_loss=0.09567, pruned_loss=0.01684, audio_tagging_loss=0.009934, over 3041102.17 frames. ], batch size: 52, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:25:08,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1464220.0, ans=0.0 2023-11-21 10:25:13,498 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.202e+01 8.879e+01 9.509e+01 1.631e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-21 10:25:15,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1464286.6666666667, ans=0.1 2023-11-21 10:25:23,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219650 2023-11-21 10:25:33,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1464353.3333333333, ans=0.1 2023-11-21 10:25:36,143 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.94 vs. limit=12.0 2023-11-21 10:25:55,577 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3250, loss[loss=0.05096, simple_loss=0.06718, pruned_loss=0.007788, audio_tagging_loss=0.009581, over 15157.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09527, pruned_loss=0.01674, audio_tagging_loss=0.009984, over 3039511.52 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:26:08,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1464553.3333333333, ans=0.0 2023-11-21 10:26:09,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1464553.3333333333, ans=0.0 2023-11-21 10:26:10,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1464553.3333333333, ans=0.1 2023-11-21 10:26:25,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1464620.0, ans=0.125 2023-11-21 10:26:28,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219700 2023-11-21 10:26:31,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1464620.0, ans=0.125 2023-11-21 10:26:39,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=1464686.6666666667, ans=0.02 2023-11-21 10:26:53,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1464753.3333333333, ans=0.125 2023-11-21 10:26:58,968 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3300, loss[loss=0.08376, simple_loss=0.1119, pruned_loss=0.02006, audio_tagging_loss=0.007727, over 16142.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09585, pruned_loss=0.01695, audio_tagging_loss=0.01008, over 3045932.67 frames. ], batch size: 60, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:27:04,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1464820.0, ans=0.2 2023-11-21 10:27:10,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1464820.0, ans=0.2 2023-11-21 10:27:19,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1464886.6666666667, ans=0.0 2023-11-21 10:27:24,141 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.128e+01 8.736e+01 9.388e+01 1.641e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 10:27:32,971 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219750 2023-11-21 10:28:06,494 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3350, loss[loss=0.07968, simple_loss=0.0969, pruned_loss=0.02091, audio_tagging_loss=0.01032, over 14878.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.0968, pruned_loss=0.01715, audio_tagging_loss=0.009855, over 3050313.45 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:28:08,618 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.31 vs. limit=10.0 2023-11-21 10:28:25,661 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.71 vs. limit=6.0 2023-11-21 10:28:29,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1465220.0, ans=0.2 2023-11-21 10:28:32,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.04 vs. limit=15.0 2023-11-21 10:28:37,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219800 2023-11-21 10:28:40,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1465286.6666666667, ans=0.2 2023-11-21 10:28:59,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.62 vs. limit=10.0 2023-11-21 10:29:03,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1465420.0, ans=0.125 2023-11-21 10:29:11,215 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3400, loss[loss=0.0586, simple_loss=0.0765, pruned_loss=0.009863, audio_tagging_loss=0.01049, over 15246.00 frames. ], tot_loss[loss=0.07552, simple_loss=0.09755, pruned_loss=0.01715, audio_tagging_loss=0.009592, over 3049748.86 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:29:35,133 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.502e+01 9.051e+01 9.796e+01 1.205e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-21 10:29:45,033 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219850 2023-11-21 10:29:47,046 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=14.74 vs. limit=15.0 2023-11-21 10:29:55,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1465686.6666666667, ans=0.2 2023-11-21 10:30:13,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1465753.3333333333, ans=0.125 2023-11-21 10:30:15,840 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3450, loss[loss=0.0722, simple_loss=0.09049, pruned_loss=0.01623, audio_tagging_loss=0.01072, over 14897.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.09787, pruned_loss=0.0172, audio_tagging_loss=0.009471, over 3048640.92 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:30:16,667 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.74 vs. limit=15.0 2023-11-21 10:30:17,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1465820.0, ans=0.0 2023-11-21 10:30:32,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1465886.6666666667, ans=0.0 2023-11-21 10:30:39,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1465886.6666666667, ans=0.015 2023-11-21 10:30:49,328 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219900 2023-11-21 10:30:58,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1466020.0, ans=0.0 2023-11-21 10:31:05,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1466020.0, ans=0.2 2023-11-21 10:31:20,033 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=10.24 vs. limit=10.0 2023-11-21 10:31:22,085 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3500, loss[loss=0.07555, simple_loss=0.09337, pruned_loss=0.01784, audio_tagging_loss=0.01102, over 16145.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09878, pruned_loss=0.01742, audio_tagging_loss=0.009362, over 3053655.70 frames. ], batch size: 62, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:31:32,381 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.85 vs. limit=15.0 2023-11-21 10:31:44,237 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.587e+01 8.045e+01 8.820e+01 9.717e+01 1.244e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-21 10:31:52,914 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 219950 2023-11-21 10:31:54,003 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:32:09,997 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:32:16,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1466420.0, ans=0.125 2023-11-21 10:32:26,279 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3550, loss[loss=0.07643, simple_loss=0.09917, pruned_loss=0.01777, audio_tagging_loss=0.009075, over 15166.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09819, pruned_loss=0.01726, audio_tagging_loss=0.009331, over 3058454.45 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:32:31,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1466486.6666666667, ans=0.1 2023-11-21 10:32:43,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1466553.3333333333, ans=0.125 2023-11-21 10:32:51,628 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.81 vs. limit=15.0 2023-11-21 10:32:56,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1466620.0, ans=0.5 2023-11-21 10:32:58,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1466620.0, ans=0.1 2023-11-21 10:32:58,994 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220000 2023-11-21 10:33:05,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1466620.0, ans=0.1 2023-11-21 10:33:15,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.88 vs. limit=15.0 2023-11-21 10:33:34,057 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3600, loss[loss=0.06864, simple_loss=0.08311, pruned_loss=0.01706, audio_tagging_loss=0.01003, over 14615.00 frames. ], tot_loss[loss=0.07569, simple_loss=0.09788, pruned_loss=0.01736, audio_tagging_loss=0.00939, over 3053359.37 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 32.0 2023-11-21 10:33:54,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1466886.6666666667, ans=0.0 2023-11-21 10:34:00,213 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.124e+01 7.919e+01 8.702e+01 9.494e+01 1.216e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 10:34:00,814 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.11 vs. limit=15.0 2023-11-21 10:34:04,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1466953.3333333333, ans=0.125 2023-11-21 10:34:04,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1466953.3333333333, ans=0.2 2023-11-21 10:34:07,750 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220050 2023-11-21 10:34:11,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1466953.3333333333, ans=0.125 2023-11-21 10:34:39,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1467153.3333333333, ans=0.0 2023-11-21 10:34:40,915 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3650, loss[loss=0.08727, simple_loss=0.1202, pruned_loss=0.02139, audio_tagging_loss=0.005799, over 15334.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09797, pruned_loss=0.01726, audio_tagging_loss=0.009323, over 3044931.37 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:34:48,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1467153.3333333333, ans=0.5 2023-11-21 10:34:53,271 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.91 vs. limit=22.5 2023-11-21 10:34:54,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1467220.0, ans=0.125 2023-11-21 10:35:05,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1467286.6666666667, ans=0.04949747468305833 2023-11-21 10:35:09,274 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.43 vs. limit=15.0 2023-11-21 10:35:12,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220100 2023-11-21 10:35:29,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1467353.3333333333, ans=0.125 2023-11-21 10:35:30,666 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.50 vs. limit=6.0 2023-11-21 10:35:44,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.83 vs. limit=15.0 2023-11-21 10:35:44,974 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3700, loss[loss=0.06301, simple_loss=0.08166, pruned_loss=0.01396, audio_tagging_loss=0.00822, over 15853.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09873, pruned_loss=0.01738, audio_tagging_loss=0.009238, over 3048577.47 frames. ], batch size: 61, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:35:45,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1467486.6666666667, ans=0.0 2023-11-21 10:35:48,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1467486.6666666667, ans=0.0 2023-11-21 10:35:54,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1467486.6666666667, ans=0.1 2023-11-21 10:35:59,005 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0 2023-11-21 10:36:08,756 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.151e+01 8.696e+01 9.414e+01 1.695e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 10:36:12,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1467620.0, ans=0.0 2023-11-21 10:36:17,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220150 2023-11-21 10:36:28,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1467686.6666666667, ans=0.5 2023-11-21 10:36:37,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1467753.3333333333, ans=0.0 2023-11-21 10:36:49,677 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3750, loss[loss=0.1011, simple_loss=0.1392, pruned_loss=0.02393, audio_tagging_loss=0.007543, over 15493.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.09861, pruned_loss=0.01745, audio_tagging_loss=0.009355, over 3058296.70 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:36:51,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1467820.0, ans=0.2 2023-11-21 10:37:14,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1467886.6666666667, ans=0.125 2023-11-21 10:37:23,601 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220200 2023-11-21 10:37:35,363 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:37:45,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1468086.6666666667, ans=0.125 2023-11-21 10:37:50,273 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.37 vs. limit=10.0 2023-11-21 10:37:57,226 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3800, loss[loss=0.05028, simple_loss=0.06519, pruned_loss=0.00838, audio_tagging_loss=0.009307, over 15018.00 frames. ], tot_loss[loss=0.07608, simple_loss=0.09835, pruned_loss=0.0174, audio_tagging_loss=0.009512, over 3067311.41 frames. ], batch size: 59, lr: 3.63e-03, grad_scale: 8.0 2023-11-21 10:38:05,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.21 vs. limit=10.0 2023-11-21 10:38:19,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1468220.0, ans=0.125 2023-11-21 10:38:22,689 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.173e+01 8.825e+01 9.624e+01 1.863e+02, threshold=1.765e+02, percent-clipped=1.0 2023-11-21 10:38:24,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1468286.6666666667, ans=0.0 2023-11-21 10:38:25,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1468286.6666666667, ans=0.0 2023-11-21 10:38:28,977 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220250 2023-11-21 10:38:31,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1468286.6666666667, ans=0.125 2023-11-21 10:38:33,349 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.20 vs. limit=22.5 2023-11-21 10:39:01,413 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3850, loss[loss=0.07928, simple_loss=0.1013, pruned_loss=0.0189, audio_tagging_loss=0.009725, over 14754.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09722, pruned_loss=0.01717, audio_tagging_loss=0.00965, over 3061689.90 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 8.0 2023-11-21 10:39:34,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220300 2023-11-21 10:39:35,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1468620.0, ans=0.125 2023-11-21 10:39:45,265 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.56 vs. limit=22.5 2023-11-21 10:39:48,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1468686.6666666667, ans=0.125 2023-11-21 10:39:57,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1468753.3333333333, ans=0.0 2023-11-21 10:40:04,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=1468753.3333333333, ans=15.0 2023-11-21 10:40:05,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1468820.0, ans=0.2 2023-11-21 10:40:06,815 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3900, loss[loss=0.07026, simple_loss=0.0976, pruned_loss=0.01165, audio_tagging_loss=0.00981, over 16007.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09581, pruned_loss=0.01671, audio_tagging_loss=0.009687, over 3059008.65 frames. ], batch size: 60, lr: 3.63e-03, grad_scale: 8.0 2023-11-21 10:40:12,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1468820.0, ans=0.125 2023-11-21 10:40:28,189 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.65 vs. limit=22.5 2023-11-21 10:40:29,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1468886.6666666667, ans=0.125 2023-11-21 10:40:34,093 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.413e+01 7.996e+01 8.695e+01 9.569e+01 1.427e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 10:40:40,565 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220350 2023-11-21 10:40:44,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.38 vs. limit=15.0 2023-11-21 10:40:52,975 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:40:56,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1469020.0, ans=0.1 2023-11-21 10:41:11,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1469086.6666666667, ans=0.125 2023-11-21 10:41:13,862 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 3950, loss[loss=0.08256, simple_loss=0.1044, pruned_loss=0.0212, audio_tagging_loss=0.009177, over 14976.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09643, pruned_loss=0.01684, audio_tagging_loss=0.009754, over 3057752.54 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 8.0 2023-11-21 10:41:14,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1469153.3333333333, ans=0.2 2023-11-21 10:41:18,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1469153.3333333333, ans=0.125 2023-11-21 10:41:40,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.69 vs. limit=15.0 2023-11-21 10:41:46,254 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220400 2023-11-21 10:41:54,558 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.43 vs. limit=15.0 2023-11-21 10:42:19,397 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4000, loss[loss=0.06175, simple_loss=0.08023, pruned_loss=0.01017, audio_tagging_loss=0.01147, over 14973.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09706, pruned_loss=0.01707, audio_tagging_loss=0.009781, over 3057144.37 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 16.0 2023-11-21 10:42:19,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1469486.6666666667, ans=0.125 2023-11-21 10:42:25,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1469486.6666666667, ans=0.125 2023-11-21 10:42:44,791 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.206e+01 8.898e+01 9.673e+01 1.616e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-21 10:42:52,357 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220450 2023-11-21 10:42:59,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1469686.6666666667, ans=0.125 2023-11-21 10:43:03,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1469686.6666666667, ans=0.0 2023-11-21 10:43:06,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1469686.6666666667, ans=0.125 2023-11-21 10:43:23,438 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.97 vs. limit=12.0 2023-11-21 10:43:24,154 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4050, loss[loss=0.07974, simple_loss=0.1081, pruned_loss=0.01769, audio_tagging_loss=0.007986, over 14880.00 frames. ], tot_loss[loss=0.07531, simple_loss=0.09688, pruned_loss=0.01699, audio_tagging_loss=0.009876, over 3056450.06 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:43:26,797 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:43:26,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1469820.0, ans=0.0 2023-11-21 10:43:41,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1469886.6666666667, ans=0.125 2023-11-21 10:43:56,952 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220500 2023-11-21 10:44:03,389 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:44:19,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1470086.6666666667, ans=0.0 2023-11-21 10:44:29,552 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4100, loss[loss=0.08573, simple_loss=0.1112, pruned_loss=0.01887, audio_tagging_loss=0.01129, over 15411.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09788, pruned_loss=0.01711, audio_tagging_loss=0.009817, over 3058150.06 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:44:32,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1470153.3333333333, ans=0.1 2023-11-21 10:44:45,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1470220.0, ans=0.125 2023-11-21 10:44:53,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.97 vs. limit=15.0 2023-11-21 10:44:55,167 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.123e+01 8.556e+01 9.611e+01 1.349e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-21 10:45:01,803 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220550 2023-11-21 10:45:12,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1470353.3333333333, ans=0.125 2023-11-21 10:45:35,355 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4150, loss[loss=0.06912, simple_loss=0.09131, pruned_loss=0.01521, audio_tagging_loss=0.008255, over 15430.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09764, pruned_loss=0.01715, audio_tagging_loss=0.009688, over 3051508.11 frames. ], batch size: 60, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:45:46,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1470553.3333333333, ans=0.2 2023-11-21 10:45:46,993 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.69 vs. limit=15.0 2023-11-21 10:45:48,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.34 vs. limit=12.0 2023-11-21 10:46:01,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1470620.0, ans=0.1 2023-11-21 10:46:04,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1470620.0, ans=0.0 2023-11-21 10:46:07,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220600 2023-11-21 10:46:09,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1470620.0, ans=0.0 2023-11-21 10:46:12,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1470620.0, ans=0.1 2023-11-21 10:46:21,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1470686.6666666667, ans=0.125 2023-11-21 10:46:22,588 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:46:22,852 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:46:39,654 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4200, loss[loss=0.05938, simple_loss=0.07962, pruned_loss=0.01381, audio_tagging_loss=0.005761, over 14856.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09618, pruned_loss=0.01674, audio_tagging_loss=0.009537, over 3053141.86 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:46:40,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.54 vs. limit=15.0 2023-11-21 10:46:56,886 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.11 vs. limit=15.0 2023-11-21 10:47:06,455 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.111e+01 8.714e+01 9.250e+01 1.135e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 10:47:10,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1470953.3333333333, ans=0.0 2023-11-21 10:47:12,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220650 2023-11-21 10:47:25,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1471020.0, ans=0.125 2023-11-21 10:47:39,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1471086.6666666667, ans=0.2 2023-11-21 10:47:43,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1471153.3333333333, ans=0.125 2023-11-21 10:47:44,736 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4250, loss[loss=0.08177, simple_loss=0.1069, pruned_loss=0.01748, audio_tagging_loss=0.01083, over 14625.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09711, pruned_loss=0.01701, audio_tagging_loss=0.009472, over 3049218.54 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:47:49,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1471153.3333333333, ans=0.0 2023-11-21 10:48:17,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220700 2023-11-21 10:48:18,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1471286.6666666667, ans=0.125 2023-11-21 10:48:24,500 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.03 vs. limit=15.0 2023-11-21 10:48:48,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1471420.0, ans=0.125 2023-11-21 10:48:50,637 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4300, loss[loss=0.07956, simple_loss=0.1042, pruned_loss=0.01812, audio_tagging_loss=0.009321, over 15961.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09686, pruned_loss=0.01697, audio_tagging_loss=0.009444, over 3057561.72 frames. ], batch size: 60, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:49:14,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1471620.0, ans=0.2 2023-11-21 10:49:15,450 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.061e+01 8.689e+01 9.437e+01 1.140e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-21 10:49:22,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220750 2023-11-21 10:49:22,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1471620.0, ans=0.0 2023-11-21 10:49:46,777 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.49 vs. limit=22.5 2023-11-21 10:49:50,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.29 vs. limit=15.0 2023-11-21 10:49:54,716 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4350, loss[loss=0.07683, simple_loss=0.09379, pruned_loss=0.01885, audio_tagging_loss=0.01108, over 14872.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09775, pruned_loss=0.01716, audio_tagging_loss=0.0094, over 3049901.60 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:50:20,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1471953.3333333333, ans=0.125 2023-11-21 10:50:21,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1471953.3333333333, ans=0.125 2023-11-21 10:50:27,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1471953.3333333333, ans=0.0 2023-11-21 10:50:28,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220800 2023-11-21 10:50:41,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1472020.0, ans=0.125 2023-11-21 10:51:00,750 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4400, loss[loss=0.07445, simple_loss=0.09225, pruned_loss=0.01761, audio_tagging_loss=0.01072, over 15471.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09785, pruned_loss=0.01719, audio_tagging_loss=0.009326, over 3045918.87 frames. ], batch size: 59, lr: 3.62e-03, grad_scale: 32.0 2023-11-21 10:51:09,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1472153.3333333333, ans=0.125 2023-11-21 10:51:21,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1472220.0, ans=0.125 2023-11-21 10:51:27,107 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.481e+01 8.151e+01 8.586e+01 9.427e+01 1.147e+02, threshold=1.717e+02, percent-clipped=0.0 2023-11-21 10:51:33,378 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220850 2023-11-21 10:51:39,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1472353.3333333333, ans=0.125 2023-11-21 10:51:45,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1472353.3333333333, ans=0.125 2023-11-21 10:51:51,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1472420.0, ans=0.0 2023-11-21 10:52:06,468 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4450, loss[loss=0.08257, simple_loss=0.1102, pruned_loss=0.01781, audio_tagging_loss=0.009663, over 15121.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09807, pruned_loss=0.01714, audio_tagging_loss=0.009256, over 3044392.89 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 32.0 2023-11-21 10:52:38,096 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220900 2023-11-21 10:52:56,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1472686.6666666667, ans=0.125 2023-11-21 10:53:02,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1472753.3333333333, ans=15.0 2023-11-21 10:53:11,304 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4500, loss[loss=0.09318, simple_loss=0.1267, pruned_loss=0.01846, audio_tagging_loss=0.01137, over 15021.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.09857, pruned_loss=0.01721, audio_tagging_loss=0.0093, over 3046261.45 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 32.0 2023-11-21 10:53:21,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1472820.0, ans=0.1 2023-11-21 10:53:33,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1472886.6666666667, ans=0.125 2023-11-21 10:53:38,067 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.114e+01 8.781e+01 9.396e+01 1.182e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-21 10:53:45,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 220950 2023-11-21 10:54:00,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1473020.0, ans=0.125 2023-11-21 10:54:06,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1473086.6666666667, ans=0.05 2023-11-21 10:54:10,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1473086.6666666667, ans=0.0 2023-11-21 10:54:16,247 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4550, loss[loss=0.06994, simple_loss=0.09917, pruned_loss=0.01064, audio_tagging_loss=0.009717, over 15305.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.0972, pruned_loss=0.01691, audio_tagging_loss=0.009326, over 3045897.53 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 32.0 2023-11-21 10:54:21,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1473153.3333333333, ans=0.125 2023-11-21 10:54:43,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1473286.6666666667, ans=0.125 2023-11-21 10:54:49,104 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221000 2023-11-21 10:54:51,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1473286.6666666667, ans=0.125 2023-11-21 10:54:54,574 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 10:55:01,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1473353.3333333333, ans=0.0 2023-11-21 10:55:02,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1473353.3333333333, ans=0.125 2023-11-21 10:55:05,203 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 10:55:08,536 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.43 vs. limit=8.0 2023-11-21 10:55:09,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1473420.0, ans=0.0 2023-11-21 10:55:11,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1473420.0, ans=0.125 2023-11-21 10:55:11,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1473420.0, ans=0.0 2023-11-21 10:55:22,531 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4600, loss[loss=0.08997, simple_loss=0.124, pruned_loss=0.01811, audio_tagging_loss=0.00987, over 15440.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09693, pruned_loss=0.0168, audio_tagging_loss=0.009391, over 3039938.64 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:55:32,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1473486.6666666667, ans=0.1 2023-11-21 10:55:48,764 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.454e+01 8.163e+01 8.661e+01 9.433e+01 1.192e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 10:55:53,763 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221050 2023-11-21 10:56:05,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1473686.6666666667, ans=0.1 2023-11-21 10:56:27,019 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4650, loss[loss=0.07112, simple_loss=0.0869, pruned_loss=0.01776, audio_tagging_loss=0.009916, over 15096.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09645, pruned_loss=0.01679, audio_tagging_loss=0.009553, over 3037949.24 frames. ], batch size: 59, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:56:32,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.09 vs. limit=15.0 2023-11-21 10:56:40,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1473886.6666666667, ans=0.0 2023-11-21 10:56:57,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1473953.3333333333, ans=0.2 2023-11-21 10:56:59,987 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221100 2023-11-21 10:57:26,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1474086.6666666667, ans=0.125 2023-11-21 10:57:31,379 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4700, loss[loss=0.08467, simple_loss=0.1063, pruned_loss=0.01948, audio_tagging_loss=0.01206, over 15049.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09682, pruned_loss=0.01676, audio_tagging_loss=0.009712, over 3042376.60 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:57:59,446 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.726e+01 8.412e+01 8.839e+01 9.940e+01 1.268e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 10:58:04,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221150 2023-11-21 10:58:26,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.33 vs. limit=15.0 2023-11-21 10:58:36,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1474486.6666666667, ans=0.0 2023-11-21 10:58:36,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1474486.6666666667, ans=0.2 2023-11-21 10:58:37,199 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4750, loss[loss=0.06812, simple_loss=0.07992, pruned_loss=0.01673, audio_tagging_loss=0.01143, over 16379.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09629, pruned_loss=0.01676, audio_tagging_loss=0.009825, over 3039905.44 frames. ], batch size: 64, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 10:58:38,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1474486.6666666667, ans=0.2 2023-11-21 10:58:43,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.03 vs. limit=15.0 2023-11-21 10:58:52,511 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.31 vs. limit=22.5 2023-11-21 10:59:09,403 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221200 2023-11-21 10:59:33,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1474753.3333333333, ans=0.125 2023-11-21 10:59:43,344 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4800, loss[loss=0.06694, simple_loss=0.07879, pruned_loss=0.01669, audio_tagging_loss=0.01086, over 15583.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09628, pruned_loss=0.01676, audio_tagging_loss=0.00991, over 3050917.58 frames. ], batch size: 60, lr: 3.62e-03, grad_scale: 32.0 2023-11-21 10:59:51,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1474820.0, ans=0.0 2023-11-21 10:59:55,260 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.46 vs. limit=15.0 2023-11-21 10:59:56,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.76 vs. limit=22.5 2023-11-21 10:59:58,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1474886.6666666667, ans=0.2 2023-11-21 10:59:59,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1474886.6666666667, ans=0.0 2023-11-21 11:00:07,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1474953.3333333333, ans=0.0 2023-11-21 11:00:10,189 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.162e+01 8.841e+01 9.779e+01 1.301e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 11:00:15,881 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221250 2023-11-21 11:00:29,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1475020.0, ans=0.2 2023-11-21 11:00:38,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1475086.6666666667, ans=0.125 2023-11-21 11:00:38,583 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.40 vs. limit=15.0 2023-11-21 11:00:47,759 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4850, loss[loss=0.09165, simple_loss=0.1162, pruned_loss=0.02281, audio_tagging_loss=0.01075, over 14064.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09619, pruned_loss=0.01663, audio_tagging_loss=0.0101, over 3049206.61 frames. ], batch size: 53, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 11:00:51,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1475153.3333333333, ans=0.2 2023-11-21 11:01:01,160 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.59 vs. limit=15.0 2023-11-21 11:01:12,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=22.5 2023-11-21 11:01:21,613 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221300 2023-11-21 11:01:38,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1475420.0, ans=0.125 2023-11-21 11:01:52,958 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4900, loss[loss=0.08285, simple_loss=0.1095, pruned_loss=0.01892, audio_tagging_loss=0.009185, over 14477.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09614, pruned_loss=0.01667, audio_tagging_loss=0.01002, over 3045284.44 frames. ], batch size: 53, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 11:02:09,311 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.95 vs. limit=22.5 2023-11-21 11:02:10,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1475553.3333333333, ans=0.1 2023-11-21 11:02:20,760 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.034e+01 8.624e+01 9.399e+01 1.492e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-21 11:02:24,596 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221350 2023-11-21 11:02:43,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1475753.3333333333, ans=0.2 2023-11-21 11:02:57,364 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 4950, loss[loss=0.06517, simple_loss=0.08342, pruned_loss=0.01326, audio_tagging_loss=0.01021, over 15130.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09631, pruned_loss=0.01666, audio_tagging_loss=0.009742, over 3041703.02 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 11:03:06,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1475820.0, ans=0.0 2023-11-21 11:03:12,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1475886.6666666667, ans=0.0 2023-11-21 11:03:15,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1475886.6666666667, ans=0.125 2023-11-21 11:03:17,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1475886.6666666667, ans=0.0 2023-11-21 11:03:27,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1475953.3333333333, ans=0.5 2023-11-21 11:03:29,671 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221400 2023-11-21 11:03:38,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1476020.0, ans=0.125 2023-11-21 11:03:46,253 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.33 vs. limit=10.0 2023-11-21 11:03:54,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1476086.6666666667, ans=0.0 2023-11-21 11:03:56,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1476086.6666666667, ans=0.1 2023-11-21 11:04:02,075 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5000, loss[loss=0.0714, simple_loss=0.1005, pruned_loss=0.01581, audio_tagging_loss=0.005359, over 14263.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09648, pruned_loss=0.01676, audio_tagging_loss=0.009649, over 3037240.19 frames. ], batch size: 53, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 11:04:08,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1476153.3333333333, ans=0.0 2023-11-21 11:04:31,458 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.334e+01 8.366e+01 9.036e+01 1.004e+02 1.239e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-21 11:04:35,412 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221450 2023-11-21 11:05:03,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1476420.0, ans=0.0 2023-11-21 11:05:07,096 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5050, loss[loss=0.07668, simple_loss=0.1003, pruned_loss=0.01701, audio_tagging_loss=0.009498, over 15120.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09531, pruned_loss=0.01647, audio_tagging_loss=0.009575, over 3045515.80 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 11:05:39,946 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221500 2023-11-21 11:05:55,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1476686.6666666667, ans=0.125 2023-11-21 11:06:09,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1476753.3333333333, ans=0.0 2023-11-21 11:06:12,965 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5100, loss[loss=0.06801, simple_loss=0.08883, pruned_loss=0.01711, audio_tagging_loss=0.006486, over 14412.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09571, pruned_loss=0.01669, audio_tagging_loss=0.009542, over 3046148.42 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 11:06:28,855 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.65 vs. limit=15.0 2023-11-21 11:06:32,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1476886.6666666667, ans=0.125 2023-11-21 11:06:36,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1476886.6666666667, ans=0.0 2023-11-21 11:06:40,829 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.039e+01 8.516e+01 9.144e+01 1.339e+02, threshold=1.703e+02, percent-clipped=0.0 2023-11-21 11:06:45,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221550 2023-11-21 11:07:00,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1477020.0, ans=0.1 2023-11-21 11:07:18,251 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5150, loss[loss=0.06867, simple_loss=0.08649, pruned_loss=0.01652, audio_tagging_loss=0.008906, over 15651.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09672, pruned_loss=0.0169, audio_tagging_loss=0.009599, over 3048738.97 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 16.0 2023-11-21 11:07:19,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1477153.3333333333, ans=0.125 2023-11-21 11:07:30,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1477220.0, ans=0.1 2023-11-21 11:07:51,158 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221600 2023-11-21 11:08:03,205 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=15.0 2023-11-21 11:08:10,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1477420.0, ans=0.125 2023-11-21 11:08:23,356 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5200, loss[loss=0.08638, simple_loss=0.114, pruned_loss=0.02066, audio_tagging_loss=0.008725, over 14882.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.09724, pruned_loss=0.01704, audio_tagging_loss=0.009635, over 3050070.70 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 32.0 2023-11-21 11:08:49,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1477620.0, ans=0.125 2023-11-21 11:08:51,497 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.310e+01 8.141e+01 8.886e+01 9.437e+01 1.227e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-21 11:08:51,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1477620.0, ans=0.125 2023-11-21 11:08:52,301 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.28 vs. limit=15.0 2023-11-21 11:08:53,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1477620.0, ans=0.0 2023-11-21 11:08:55,950 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221650 2023-11-21 11:08:59,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1477620.0, ans=0.0 2023-11-21 11:09:01,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1477686.6666666667, ans=0.0 2023-11-21 11:09:12,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1477686.6666666667, ans=0.2 2023-11-21 11:09:25,763 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.11 vs. limit=10.0 2023-11-21 11:09:28,809 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5250, loss[loss=0.1024, simple_loss=0.1324, pruned_loss=0.0289, audio_tagging_loss=0.00731, over 15686.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09852, pruned_loss=0.0172, audio_tagging_loss=0.009458, over 3054036.27 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 32.0 2023-11-21 11:09:46,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1477886.6666666667, ans=0.125 2023-11-21 11:09:59,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1477953.3333333333, ans=0.125 2023-11-21 11:10:00,076 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221700 2023-11-21 11:10:24,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1478086.6666666667, ans=0.1 2023-11-21 11:10:32,517 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5300, loss[loss=0.07913, simple_loss=0.1128, pruned_loss=0.01554, audio_tagging_loss=0.007172, over 14642.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09868, pruned_loss=0.01717, audio_tagging_loss=0.009342, over 3046754.55 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:10:51,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1478220.0, ans=0.0 2023-11-21 11:11:01,607 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.119e+01 8.657e+01 9.101e+01 1.165e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 11:11:06,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221750 2023-11-21 11:11:08,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1478286.6666666667, ans=0.1 2023-11-21 11:11:36,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.99 vs. limit=15.0 2023-11-21 11:11:36,915 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5350, loss[loss=0.08803, simple_loss=0.1132, pruned_loss=0.0231, audio_tagging_loss=0.00834, over 15600.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09855, pruned_loss=0.01717, audio_tagging_loss=0.009418, over 3041264.58 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:11:48,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1478486.6666666667, ans=0.125 2023-11-21 11:12:09,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221800 2023-11-21 11:12:17,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1478686.6666666667, ans=0.1 2023-11-21 11:12:17,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1478686.6666666667, ans=0.05 2023-11-21 11:12:35,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1478753.3333333333, ans=10.0 2023-11-21 11:12:37,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1478753.3333333333, ans=0.125 2023-11-21 11:12:42,766 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5400, loss[loss=0.06794, simple_loss=0.08478, pruned_loss=0.01582, audio_tagging_loss=0.009731, over 15331.00 frames. ], tot_loss[loss=0.0755, simple_loss=0.09802, pruned_loss=0.01702, audio_tagging_loss=0.009476, over 3048380.85 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:12:45,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1478820.0, ans=0.0 2023-11-21 11:12:47,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1478820.0, ans=0.125 2023-11-21 11:13:11,526 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.155e+01 8.706e+01 9.355e+01 1.764e+02, threshold=1.741e+02, percent-clipped=1.0 2023-11-21 11:13:14,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221850 2023-11-21 11:13:29,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1479020.0, ans=0.1 2023-11-21 11:13:33,552 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:13:42,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1479086.6666666667, ans=0.125 2023-11-21 11:13:42,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1479086.6666666667, ans=0.125 2023-11-21 11:13:43,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1479086.6666666667, ans=0.0 2023-11-21 11:13:46,686 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5450, loss[loss=0.07626, simple_loss=0.1036, pruned_loss=0.01407, audio_tagging_loss=0.01042, over 15381.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09895, pruned_loss=0.01723, audio_tagging_loss=0.009451, over 3044037.06 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:13:59,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1479220.0, ans=0.0 2023-11-21 11:14:04,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1479220.0, ans=0.0 2023-11-21 11:14:09,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1479220.0, ans=0.125 2023-11-21 11:14:10,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1479220.0, ans=0.2 2023-11-21 11:14:11,574 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.47 vs. limit=15.0 2023-11-21 11:14:15,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1479286.6666666667, ans=0.125 2023-11-21 11:14:19,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221900 2023-11-21 11:14:50,957 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5500, loss[loss=0.104, simple_loss=0.139, pruned_loss=0.0265, audio_tagging_loss=0.007976, over 15004.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.0987, pruned_loss=0.01721, audio_tagging_loss=0.009568, over 3047710.08 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:15:06,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1479553.3333333333, ans=0.1 2023-11-21 11:15:11,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1479553.3333333333, ans=0.015 2023-11-21 11:15:21,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.123e+01 9.057e+01 9.951e+01 1.227e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-21 11:15:23,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 221950 2023-11-21 11:15:38,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1479686.6666666667, ans=0.0 2023-11-21 11:15:46,650 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=15.0 2023-11-21 11:15:49,677 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:15:56,244 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5550, loss[loss=0.1013, simple_loss=0.1312, pruned_loss=0.02484, audio_tagging_loss=0.01082, over 15172.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09826, pruned_loss=0.01713, audio_tagging_loss=0.009777, over 3052050.11 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:16:04,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1479820.0, ans=0.1 2023-11-21 11:16:21,894 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.81 vs. limit=12.0 2023-11-21 11:16:27,021 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222000 2023-11-21 11:16:54,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1480086.6666666667, ans=0.0 2023-11-21 11:17:00,034 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5600, loss[loss=0.07944, simple_loss=0.104, pruned_loss=0.01645, audio_tagging_loss=0.011, over 15643.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.097, pruned_loss=0.01662, audio_tagging_loss=0.009902, over 3048475.66 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:17:02,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1480153.3333333333, ans=0.2 2023-11-21 11:17:11,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1480220.0, ans=0.05 2023-11-21 11:17:12,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1480220.0, ans=0.2 2023-11-21 11:17:16,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1480220.0, ans=0.1 2023-11-21 11:17:20,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1480220.0, ans=10.0 2023-11-21 11:17:30,534 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.739e+01 7.938e+01 8.660e+01 9.316e+01 1.150e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 11:17:31,878 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222050 2023-11-21 11:17:32,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1480286.6666666667, ans=0.04949747468305833 2023-11-21 11:17:44,755 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 11:18:02,793 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5650, loss[loss=0.08772, simple_loss=0.1132, pruned_loss=0.02217, audio_tagging_loss=0.008957, over 15106.00 frames. ], tot_loss[loss=0.07468, simple_loss=0.09634, pruned_loss=0.01661, audio_tagging_loss=0.0099, over 3044624.80 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:18:35,867 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222100 2023-11-21 11:18:38,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1480620.0, ans=0.2 2023-11-21 11:18:55,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1480753.3333333333, ans=0.125 2023-11-21 11:19:07,305 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5700, loss[loss=0.06855, simple_loss=0.0856, pruned_loss=0.01502, audio_tagging_loss=0.01073, over 14901.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09589, pruned_loss=0.01662, audio_tagging_loss=0.009887, over 3047655.53 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:19:13,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=13.68 vs. limit=22.5 2023-11-21 11:19:21,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1480886.6666666667, ans=0.0 2023-11-21 11:19:34,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1480953.3333333333, ans=0.0 2023-11-21 11:19:37,375 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.400e+01 9.034e+01 9.970e+01 1.319e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-21 11:19:38,764 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222150 2023-11-21 11:19:38,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1480953.3333333333, ans=0.0 2023-11-21 11:19:43,900 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:19:50,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1481020.0, ans=0.1 2023-11-21 11:20:11,592 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5750, loss[loss=0.07728, simple_loss=0.0996, pruned_loss=0.01921, audio_tagging_loss=0.008264, over 15531.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09592, pruned_loss=0.01676, audio_tagging_loss=0.009726, over 3046271.91 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:20:21,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1481153.3333333333, ans=0.2 2023-11-21 11:20:43,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222200 2023-11-21 11:21:04,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1481420.0, ans=0.125 2023-11-21 11:21:13,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1481486.6666666667, ans=0.125 2023-11-21 11:21:14,846 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5800, loss[loss=0.07404, simple_loss=0.09289, pruned_loss=0.01465, audio_tagging_loss=0.01295, over 13830.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09593, pruned_loss=0.01678, audio_tagging_loss=0.009644, over 3042771.59 frames. ], batch size: 53, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:21:23,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1481486.6666666667, ans=0.0 2023-11-21 11:21:37,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1481553.3333333333, ans=0.0 2023-11-21 11:21:44,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1481620.0, ans=0.0 2023-11-21 11:21:46,273 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.053e+01 8.570e+01 9.375e+01 1.179e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 11:21:47,711 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222250 2023-11-21 11:21:57,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1481686.6666666667, ans=0.1 2023-11-21 11:22:18,582 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5850, loss[loss=0.06913, simple_loss=0.09902, pruned_loss=0.01067, audio_tagging_loss=0.008948, over 15349.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09642, pruned_loss=0.01669, audio_tagging_loss=0.009568, over 3044905.97 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:22:18,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1481820.0, ans=0.0 2023-11-21 11:22:21,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1481820.0, ans=0.125 2023-11-21 11:22:24,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1481820.0, ans=0.2 2023-11-21 11:22:50,186 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222300 2023-11-21 11:22:59,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1482020.0, ans=0.1 2023-11-21 11:23:01,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1482020.0, ans=0.0 2023-11-21 11:23:18,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1482086.6666666667, ans=0.125 2023-11-21 11:23:22,447 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5900, loss[loss=0.06584, simple_loss=0.08659, pruned_loss=0.0121, audio_tagging_loss=0.01045, over 15236.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.096, pruned_loss=0.0167, audio_tagging_loss=0.009484, over 3048277.84 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:23:28,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1482153.3333333333, ans=0.125 2023-11-21 11:23:39,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.96 vs. limit=6.0 2023-11-21 11:23:40,125 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.51 vs. limit=15.0 2023-11-21 11:23:51,651 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.184e+01 8.891e+01 9.441e+01 1.396e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-21 11:23:52,964 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222350 2023-11-21 11:23:58,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1482353.3333333333, ans=0.0 2023-11-21 11:24:00,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1482353.3333333333, ans=0.125 2023-11-21 11:24:20,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1482420.0, ans=0.125 2023-11-21 11:24:21,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1482420.0, ans=0.0 2023-11-21 11:24:24,464 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 5950, loss[loss=0.06425, simple_loss=0.09021, pruned_loss=0.01298, audio_tagging_loss=0.006164, over 15463.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.09709, pruned_loss=0.01696, audio_tagging_loss=0.009445, over 3057632.10 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:24:42,626 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.98 vs. limit=15.0 2023-11-21 11:24:46,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1482553.3333333333, ans=0.125 2023-11-21 11:24:47,864 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.46 vs. limit=12.0 2023-11-21 11:24:56,998 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222400 2023-11-21 11:25:10,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1482686.6666666667, ans=0.125 2023-11-21 11:25:19,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1482753.3333333333, ans=0.2 2023-11-21 11:25:28,514 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6000, loss[loss=0.07334, simple_loss=0.1017, pruned_loss=0.01631, audio_tagging_loss=0.006171, over 13682.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09676, pruned_loss=0.01686, audio_tagging_loss=0.009495, over 3046432.72 frames. ], batch size: 52, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:25:28,515 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 11:26:10,010 INFO [train_asr.py:1253] (3/4) Epoch 19, validation: loss=0.05983, simple_loss=0.05236, pruned_loss=0.005293, audio_tagging_loss=0.02835, over 4681554.00 frames. 2023-11-21 11:26:10,011 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 11:26:23,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1482886.6666666667, ans=0.125 2023-11-21 11:26:28,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1482886.6666666667, ans=0.0 2023-11-21 11:26:35,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1482953.3333333333, ans=0.0 2023-11-21 11:26:40,197 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.474e+01 7.915e+01 8.568e+01 9.449e+01 1.142e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 11:26:41,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222450 2023-11-21 11:26:49,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1483020.0, ans=0.125 2023-11-21 11:26:51,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1483020.0, ans=0.125 2023-11-21 11:26:55,911 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 11:27:12,935 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6050, loss[loss=0.07987, simple_loss=0.1037, pruned_loss=0.0193, audio_tagging_loss=0.008693, over 14514.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09699, pruned_loss=0.017, audio_tagging_loss=0.009379, over 3040955.55 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:27:26,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1483220.0, ans=0.2 2023-11-21 11:27:43,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1483286.6666666667, ans=0.125 2023-11-21 11:27:45,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222500 2023-11-21 11:27:57,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1483353.3333333333, ans=0.125 2023-11-21 11:28:13,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.35 vs. limit=10.0 2023-11-21 11:28:16,885 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6100, loss[loss=0.05709, simple_loss=0.06812, pruned_loss=0.01287, audio_tagging_loss=0.01016, over 15088.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09753, pruned_loss=0.01716, audio_tagging_loss=0.00936, over 3045179.85 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:28:47,590 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.060e+01 8.165e+01 8.686e+01 9.448e+01 1.325e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 11:28:48,952 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222550 2023-11-21 11:29:17,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1483753.3333333333, ans=0.125 2023-11-21 11:29:21,025 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6150, loss[loss=0.08269, simple_loss=0.09543, pruned_loss=0.02153, audio_tagging_loss=0.01344, over 15113.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09636, pruned_loss=0.01702, audio_tagging_loss=0.009508, over 3044077.39 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:29:25,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1483820.0, ans=0.0 2023-11-21 11:29:37,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1483886.6666666667, ans=0.2 2023-11-21 11:29:47,553 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.47 vs. limit=15.0 2023-11-21 11:29:52,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222600 2023-11-21 11:29:59,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1484020.0, ans=0.125 2023-11-21 11:30:04,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1484020.0, ans=0.1 2023-11-21 11:30:12,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1484086.6666666667, ans=0.025 2023-11-21 11:30:18,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1484086.6666666667, ans=0.125 2023-11-21 11:30:25,223 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6200, loss[loss=0.05315, simple_loss=0.06934, pruned_loss=0.007301, audio_tagging_loss=0.01118, over 14841.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.09708, pruned_loss=0.01701, audio_tagging_loss=0.009525, over 3046775.86 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:30:38,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1484220.0, ans=0.2 2023-11-21 11:30:55,990 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.122e+01 8.795e+01 9.552e+01 1.328e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 11:30:57,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222650 2023-11-21 11:30:58,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1484286.6666666667, ans=0.125 2023-11-21 11:31:03,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.52 vs. limit=12.0 2023-11-21 11:31:05,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1484353.3333333333, ans=0.1 2023-11-21 11:31:09,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2.whitening_limit, batch_count=1484353.3333333333, ans=15.0 2023-11-21 11:31:11,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1484353.3333333333, ans=0.125 2023-11-21 11:31:28,751 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6250, loss[loss=0.06166, simple_loss=0.08659, pruned_loss=0.009315, audio_tagging_loss=0.009053, over 15054.00 frames. ], tot_loss[loss=0.0749, simple_loss=0.09668, pruned_loss=0.01693, audio_tagging_loss=0.009636, over 3050468.08 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:31:34,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1484486.6666666667, ans=0.5 2023-11-21 11:31:44,930 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:32:01,092 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222700 2023-11-21 11:32:33,429 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6300, loss[loss=0.06526, simple_loss=0.08153, pruned_loss=0.01601, audio_tagging_loss=0.008482, over 15663.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09645, pruned_loss=0.01696, audio_tagging_loss=0.009686, over 3046267.04 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:32:42,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0 2023-11-21 11:32:47,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1484886.6666666667, ans=0.2 2023-11-21 11:33:01,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1484953.3333333333, ans=0.2 2023-11-21 11:33:03,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.093e+01 8.788e+01 9.540e+01 1.292e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 11:33:05,073 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222750 2023-11-21 11:33:20,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.16 vs. limit=12.0 2023-11-21 11:33:22,026 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.12 vs. limit=15.0 2023-11-21 11:33:37,371 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6350, loss[loss=0.09385, simple_loss=0.1126, pruned_loss=0.02745, audio_tagging_loss=0.01012, over 15302.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09526, pruned_loss=0.01675, audio_tagging_loss=0.00989, over 3043390.77 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:34:03,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1485286.6666666667, ans=0.0 2023-11-21 11:34:05,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1485286.6666666667, ans=0.125 2023-11-21 11:34:09,906 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222800 2023-11-21 11:34:14,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1485286.6666666667, ans=0.1 2023-11-21 11:34:23,470 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.66 vs. limit=15.0 2023-11-21 11:34:29,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1485420.0, ans=15.0 2023-11-21 11:34:34,579 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.57 vs. limit=15.0 2023-11-21 11:34:41,972 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6400, loss[loss=0.07692, simple_loss=0.09972, pruned_loss=0.01764, audio_tagging_loss=0.009427, over 15810.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.0954, pruned_loss=0.01675, audio_tagging_loss=0.009956, over 3046084.35 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 32.0 2023-11-21 11:34:45,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1485486.6666666667, ans=0.5 2023-11-21 11:35:05,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.44 vs. limit=22.5 2023-11-21 11:35:12,462 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.327e+01 9.131e+01 1.015e+02 1.534e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-21 11:35:13,769 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222850 2023-11-21 11:35:29,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1485686.6666666667, ans=0.0 2023-11-21 11:35:40,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1485753.3333333333, ans=0.1 2023-11-21 11:35:46,473 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6450, loss[loss=0.08928, simple_loss=0.1181, pruned_loss=0.02035, audio_tagging_loss=0.009865, over 14393.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09493, pruned_loss=0.01667, audio_tagging_loss=0.01004, over 3042239.39 frames. ], batch size: 53, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:35:50,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1485820.0, ans=0.0 2023-11-21 11:35:54,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1485820.0, ans=0.1 2023-11-21 11:36:18,317 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222900 2023-11-21 11:36:27,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1486020.0, ans=0.0 2023-11-21 11:36:44,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1486086.6666666667, ans=0.2 2023-11-21 11:36:49,992 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6500, loss[loss=0.06954, simple_loss=0.08507, pruned_loss=0.01605, audio_tagging_loss=0.01095, over 15496.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09489, pruned_loss=0.01666, audio_tagging_loss=0.01002, over 3038655.42 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 16.0 2023-11-21 11:37:03,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1486220.0, ans=0.2 2023-11-21 11:37:21,501 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:37:23,023 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.393e+01 8.000e+01 8.623e+01 9.429e+01 1.650e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-21 11:37:23,166 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 222950 2023-11-21 11:37:35,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1486353.3333333333, ans=0.125 2023-11-21 11:37:40,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1486420.0, ans=0.125 2023-11-21 11:37:41,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1486420.0, ans=0.125 2023-11-21 11:37:54,463 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6550, loss[loss=0.07743, simple_loss=0.09889, pruned_loss=0.01873, audio_tagging_loss=0.009252, over 14403.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09516, pruned_loss=0.01669, audio_tagging_loss=0.009844, over 3036573.04 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:38:22,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1486620.0, ans=0.125 2023-11-21 11:38:23,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1486620.0, ans=0.125 2023-11-21 11:38:24,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1486620.0, ans=0.0 2023-11-21 11:38:27,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223000 2023-11-21 11:38:28,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1486620.0, ans=0.1 2023-11-21 11:38:30,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1486620.0, ans=0.0 2023-11-21 11:39:00,736 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6600, loss[loss=0.07501, simple_loss=0.1005, pruned_loss=0.01545, audio_tagging_loss=0.009336, over 14707.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09553, pruned_loss=0.01666, audio_tagging_loss=0.009723, over 3037918.29 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:39:05,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1486820.0, ans=0.2 2023-11-21 11:39:08,496 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:39:12,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1486886.6666666667, ans=0.0 2023-11-21 11:39:22,390 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.58 vs. limit=12.0 2023-11-21 11:39:32,148 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.360e+01 8.932e+01 9.626e+01 1.286e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-21 11:39:32,303 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223050 2023-11-21 11:40:04,837 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6650, loss[loss=0.07259, simple_loss=0.1024, pruned_loss=0.01446, audio_tagging_loss=0.006911, over 15935.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09584, pruned_loss=0.01673, audio_tagging_loss=0.009693, over 3039903.35 frames. ], batch size: 61, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:40:10,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.58 vs. limit=15.0 2023-11-21 11:40:14,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1487153.3333333333, ans=0.07 2023-11-21 11:40:16,377 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.70 vs. limit=22.5 2023-11-21 11:40:17,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1487220.0, ans=0.0 2023-11-21 11:40:17,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1487220.0, ans=0.125 2023-11-21 11:40:19,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1487220.0, ans=0.125 2023-11-21 11:40:37,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223100 2023-11-21 11:40:54,020 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.12 vs. limit=22.5 2023-11-21 11:40:57,592 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.10 vs. limit=22.5 2023-11-21 11:41:09,291 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6700, loss[loss=0.08135, simple_loss=0.1061, pruned_loss=0.02013, audio_tagging_loss=0.008164, over 15587.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.09509, pruned_loss=0.01655, audio_tagging_loss=0.009694, over 3039715.53 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:41:18,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.17 vs. limit=15.0 2023-11-21 11:41:42,374 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.137e+01 8.757e+01 9.597e+01 1.374e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 11:41:42,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223150 2023-11-21 11:41:53,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1487686.6666666667, ans=0.0 2023-11-21 11:42:08,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1487753.3333333333, ans=0.125 2023-11-21 11:42:14,855 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6750, loss[loss=0.08294, simple_loss=0.1054, pruned_loss=0.01937, audio_tagging_loss=0.01087, over 15747.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09559, pruned_loss=0.01673, audio_tagging_loss=0.009602, over 3033466.98 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:42:16,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1487820.0, ans=0.1 2023-11-21 11:42:16,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1487820.0, ans=0.1 2023-11-21 11:42:20,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1487820.0, ans=0.0 2023-11-21 11:42:44,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.90 vs. limit=10.0 2023-11-21 11:42:46,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223200 2023-11-21 11:42:56,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1488020.0, ans=0.125 2023-11-21 11:43:04,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1488020.0, ans=0.04949747468305833 2023-11-21 11:43:14,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1488086.6666666667, ans=0.1 2023-11-21 11:43:14,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1488086.6666666667, ans=0.1 2023-11-21 11:43:18,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.73 vs. limit=22.5 2023-11-21 11:43:20,323 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6800, loss[loss=0.08041, simple_loss=0.1122, pruned_loss=0.01856, audio_tagging_loss=0.005743, over 16167.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09605, pruned_loss=0.01677, audio_tagging_loss=0.009467, over 3042688.88 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:43:31,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1488220.0, ans=0.2 2023-11-21 11:43:32,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1488220.0, ans=0.125 2023-11-21 11:43:37,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1488220.0, ans=0.2 2023-11-21 11:43:37,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1488220.0, ans=0.0 2023-11-21 11:43:51,762 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 8.121e+01 8.832e+01 9.588e+01 1.152e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-21 11:43:51,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223250 2023-11-21 11:43:56,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1488286.6666666667, ans=0.0 2023-11-21 11:44:08,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1488353.3333333333, ans=0.04949747468305833 2023-11-21 11:44:13,961 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:44:23,444 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6850, loss[loss=0.08498, simple_loss=0.1188, pruned_loss=0.01711, audio_tagging_loss=0.008456, over 15921.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09582, pruned_loss=0.01665, audio_tagging_loss=0.00946, over 3044927.90 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:44:27,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.86 vs. limit=22.5 2023-11-21 11:44:29,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1488486.6666666667, ans=0.2 2023-11-21 11:44:30,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1488486.6666666667, ans=0.2 2023-11-21 11:44:40,923 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.72 vs. limit=15.0 2023-11-21 11:44:44,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1488553.3333333333, ans=0.125 2023-11-21 11:44:56,697 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223300 2023-11-21 11:45:13,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1488753.3333333333, ans=0.125 2023-11-21 11:45:20,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1488753.3333333333, ans=0.0 2023-11-21 11:45:29,253 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6900, loss[loss=0.0929, simple_loss=0.1179, pruned_loss=0.02592, audio_tagging_loss=0.008038, over 15836.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09571, pruned_loss=0.01667, audio_tagging_loss=0.009561, over 3038421.29 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:45:45,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1488886.6666666667, ans=0.125 2023-11-21 11:45:49,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1488886.6666666667, ans=0.05 2023-11-21 11:45:53,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1488953.3333333333, ans=0.0 2023-11-21 11:46:00,718 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.015e+01 8.658e+01 9.630e+01 1.322e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 11:46:00,861 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223350 2023-11-21 11:46:00,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1488953.3333333333, ans=0.125 2023-11-21 11:46:01,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1488953.3333333333, ans=0.0 2023-11-21 11:46:16,212 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.90 vs. limit=22.5 2023-11-21 11:46:19,841 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 11:46:30,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1489086.6666666667, ans=0.125 2023-11-21 11:46:31,054 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.55 vs. limit=15.0 2023-11-21 11:46:34,098 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 6950, loss[loss=0.06359, simple_loss=0.0775, pruned_loss=0.01498, audio_tagging_loss=0.009865, over 16155.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09696, pruned_loss=0.0169, audio_tagging_loss=0.009547, over 3046142.19 frames. ], batch size: 62, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:46:38,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1489153.3333333333, ans=0.0 2023-11-21 11:46:45,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1489220.0, ans=0.1 2023-11-21 11:46:56,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1489220.0, ans=0.125 2023-11-21 11:46:59,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1489286.6666666667, ans=0.0 2023-11-21 11:47:06,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223400 2023-11-21 11:47:06,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1489286.6666666667, ans=0.0 2023-11-21 11:47:10,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1489286.6666666667, ans=0.0 2023-11-21 11:47:15,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1489353.3333333333, ans=0.0 2023-11-21 11:47:23,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.88 vs. limit=8.0 2023-11-21 11:47:38,469 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7000, loss[loss=0.08296, simple_loss=0.09986, pruned_loss=0.02183, audio_tagging_loss=0.0112, over 14900.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09597, pruned_loss=0.01672, audio_tagging_loss=0.009636, over 3040762.26 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:47:42,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.00 vs. limit=22.5 2023-11-21 11:47:47,009 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5 2023-11-21 11:47:51,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1489553.3333333333, ans=0.1 2023-11-21 11:47:52,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1489553.3333333333, ans=0.125 2023-11-21 11:48:10,558 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 7.954e+01 8.531e+01 9.123e+01 1.113e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-21 11:48:10,700 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223450 2023-11-21 11:48:21,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1489686.6666666667, ans=0.125 2023-11-21 11:48:26,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1489686.6666666667, ans=0.0 2023-11-21 11:48:30,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1489753.3333333333, ans=0.125 2023-11-21 11:48:38,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1489753.3333333333, ans=0.5 2023-11-21 11:48:39,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1489753.3333333333, ans=0.0 2023-11-21 11:48:42,249 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7050, loss[loss=0.06483, simple_loss=0.07389, pruned_loss=0.01408, audio_tagging_loss=0.01381, over 17002.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09688, pruned_loss=0.01679, audio_tagging_loss=0.009605, over 3044889.72 frames. ], batch size: 67, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:48:58,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1489886.6666666667, ans=0.125 2023-11-21 11:49:00,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1489886.6666666667, ans=0.0 2023-11-21 11:49:00,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1489886.6666666667, ans=0.125 2023-11-21 11:49:09,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1489953.3333333333, ans=0.125 2023-11-21 11:49:13,854 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223500 2023-11-21 11:49:19,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.10 vs. limit=10.0 2023-11-21 11:49:32,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1490086.6666666667, ans=0.0 2023-11-21 11:49:46,556 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7100, loss[loss=0.05396, simple_loss=0.06268, pruned_loss=0.009659, audio_tagging_loss=0.01296, over 15778.00 frames. ], tot_loss[loss=0.07517, simple_loss=0.09739, pruned_loss=0.01684, audio_tagging_loss=0.009638, over 3041888.06 frames. ], batch size: 60, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:49:53,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1490153.3333333333, ans=0.2 2023-11-21 11:50:09,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1490286.6666666667, ans=0.125 2023-11-21 11:50:09,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1490286.6666666667, ans=0.1 2023-11-21 11:50:11,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1490286.6666666667, ans=0.125 2023-11-21 11:50:15,651 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0 2023-11-21 11:50:16,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1490286.6666666667, ans=0.2 2023-11-21 11:50:17,510 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223550 2023-11-21 11:50:19,156 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.140e+01 8.741e+01 9.619e+01 1.480e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 11:50:31,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1490353.3333333333, ans=0.125 2023-11-21 11:50:38,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1490420.0, ans=0.2 2023-11-21 11:50:49,615 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7150, loss[loss=0.05538, simple_loss=0.06129, pruned_loss=0.01245, audio_tagging_loss=0.01228, over 13688.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09775, pruned_loss=0.01691, audio_tagging_loss=0.009616, over 3035401.32 frames. ], batch size: 54, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:50:52,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1490486.6666666667, ans=0.125 2023-11-21 11:51:22,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223600 2023-11-21 11:51:22,742 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.518e-03 2023-11-21 11:51:25,789 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.23 vs. limit=15.0 2023-11-21 11:51:40,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1490753.3333333333, ans=0.0 2023-11-21 11:51:49,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1490753.3333333333, ans=0.0 2023-11-21 11:51:53,998 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7200, loss[loss=0.08421, simple_loss=0.1113, pruned_loss=0.01993, audio_tagging_loss=0.008663, over 15665.00 frames. ], tot_loss[loss=0.07508, simple_loss=0.09706, pruned_loss=0.0168, audio_tagging_loss=0.009738, over 3035328.59 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:52:19,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1490953.3333333333, ans=0.125 2023-11-21 11:52:20,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1490953.3333333333, ans=0.0 2023-11-21 11:52:26,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223650 2023-11-21 11:52:27,430 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.414e+01 8.953e+01 9.739e+01 1.219e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-21 11:52:28,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.81 vs. limit=15.0 2023-11-21 11:52:28,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.88 vs. limit=15.0 2023-11-21 11:52:58,684 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7250, loss[loss=0.05416, simple_loss=0.06579, pruned_loss=0.01145, audio_tagging_loss=0.009815, over 16351.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09747, pruned_loss=0.01699, audio_tagging_loss=0.009854, over 3042170.93 frames. ], batch size: 62, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:53:09,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1491153.3333333333, ans=0.125 2023-11-21 11:53:09,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1491153.3333333333, ans=0.125 2023-11-21 11:53:30,035 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223700 2023-11-21 11:53:41,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1491353.3333333333, ans=0.1 2023-11-21 11:54:02,166 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7300, loss[loss=0.06201, simple_loss=0.08005, pruned_loss=0.01228, audio_tagging_loss=0.009696, over 14185.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.09616, pruned_loss=0.01665, audio_tagging_loss=0.00979, over 3042929.67 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:54:18,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1491553.3333333333, ans=0.125 2023-11-21 11:54:23,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1491553.3333333333, ans=0.1 2023-11-21 11:54:33,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1491620.0, ans=0.0 2023-11-21 11:54:34,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223750 2023-11-21 11:54:35,393 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.195e+01 8.693e+01 9.496e+01 1.146e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 11:54:46,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1491686.6666666667, ans=0.04949747468305833 2023-11-21 11:54:46,579 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.91 vs. limit=15.0 2023-11-21 11:55:05,921 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7350, loss[loss=0.08377, simple_loss=0.09758, pruned_loss=0.02459, audio_tagging_loss=0.01039, over 14286.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09713, pruned_loss=0.01701, audio_tagging_loss=0.009647, over 3045145.99 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 11:55:18,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1491886.6666666667, ans=10.0 2023-11-21 11:55:38,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223800 2023-11-21 11:56:02,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1492086.6666666667, ans=0.125 2023-11-21 11:56:11,494 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7400, loss[loss=0.08062, simple_loss=0.1105, pruned_loss=0.01908, audio_tagging_loss=0.006272, over 15619.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09745, pruned_loss=0.01688, audio_tagging_loss=0.009484, over 3045811.83 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:56:16,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1492153.3333333333, ans=0.125 2023-11-21 11:56:36,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1492286.6666666667, ans=0.125 2023-11-21 11:56:43,075 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223850 2023-11-21 11:56:44,764 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0 2023-11-21 11:56:45,444 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 7.994e+01 8.788e+01 9.602e+01 2.162e+02, threshold=1.758e+02, percent-clipped=1.0 2023-11-21 11:56:59,453 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.54 vs. limit=22.5 2023-11-21 11:57:15,932 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7450, loss[loss=0.09791, simple_loss=0.1285, pruned_loss=0.02757, audio_tagging_loss=0.006096, over 15905.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09697, pruned_loss=0.01674, audio_tagging_loss=0.00943, over 3042633.83 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:57:22,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1492486.6666666667, ans=0.125 2023-11-21 11:57:31,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1492553.3333333333, ans=0.125 2023-11-21 11:57:48,296 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223900 2023-11-21 11:57:50,420 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.61 vs. limit=10.0 2023-11-21 11:57:52,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1492620.0, ans=0.0 2023-11-21 11:58:13,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1492753.3333333333, ans=0.0 2023-11-21 11:58:18,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1492820.0, ans=0.1 2023-11-21 11:58:19,361 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7500, loss[loss=0.07984, simple_loss=0.1034, pruned_loss=0.02083, audio_tagging_loss=0.007293, over 15226.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09737, pruned_loss=0.0168, audio_tagging_loss=0.009384, over 3050344.96 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:58:21,738 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.83 vs. limit=15.0 2023-11-21 11:58:23,853 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 11:58:24,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0 2023-11-21 11:58:30,864 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.35 vs. limit=15.0 2023-11-21 11:58:32,197 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.73 vs. limit=12.0 2023-11-21 11:58:35,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1492886.6666666667, ans=0.1 2023-11-21 11:58:37,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1492886.6666666667, ans=0.05 2023-11-21 11:58:42,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1492886.6666666667, ans=0.125 2023-11-21 11:58:46,368 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=15.0 2023-11-21 11:58:49,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0 2023-11-21 11:58:51,666 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 223950 2023-11-21 11:58:54,580 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.233e+01 8.004e+01 8.893e+01 9.495e+01 1.256e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-21 11:59:16,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1493086.6666666667, ans=0.125 2023-11-21 11:59:24,203 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7550, loss[loss=0.0756, simple_loss=0.09064, pruned_loss=0.01975, audio_tagging_loss=0.01053, over 15439.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09661, pruned_loss=0.01681, audio_tagging_loss=0.009434, over 3053369.90 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 16.0 2023-11-21 11:59:44,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1493220.0, ans=0.125 2023-11-21 11:59:46,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.88 vs. limit=15.0 2023-11-21 11:59:55,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224000 2023-11-21 12:00:02,299 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:00:15,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1493353.3333333333, ans=0.0 2023-11-21 12:00:29,971 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7600, loss[loss=0.0648, simple_loss=0.08507, pruned_loss=0.01408, audio_tagging_loss=0.008182, over 14379.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09651, pruned_loss=0.01673, audio_tagging_loss=0.009388, over 3048798.70 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 12:00:37,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.85 vs. limit=15.0 2023-11-21 12:00:39,275 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:00:47,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1493553.3333333333, ans=0.2 2023-11-21 12:01:02,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224050 2023-11-21 12:01:05,233 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.103e+01 8.665e+01 9.260e+01 1.158e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 12:01:27,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1493753.3333333333, ans=0.125 2023-11-21 12:01:28,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1493753.3333333333, ans=0.125 2023-11-21 12:01:28,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1493753.3333333333, ans=0.125 2023-11-21 12:01:31,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1493753.3333333333, ans=0.0 2023-11-21 12:01:34,047 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7650, loss[loss=0.08251, simple_loss=0.1098, pruned_loss=0.01879, audio_tagging_loss=0.008819, over 15442.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09564, pruned_loss=0.01667, audio_tagging_loss=0.009369, over 3043922.52 frames. ], batch size: 60, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 12:01:45,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1493820.0, ans=0.09899494936611666 2023-11-21 12:02:06,330 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224100 2023-11-21 12:02:10,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1493953.3333333333, ans=0.0 2023-11-21 12:02:13,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1494020.0, ans=0.125 2023-11-21 12:02:23,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.72 vs. limit=15.0 2023-11-21 12:02:35,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.91 vs. limit=15.0 2023-11-21 12:02:38,310 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7700, loss[loss=0.05553, simple_loss=0.06828, pruned_loss=0.008044, audio_tagging_loss=0.01335, over 15496.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09531, pruned_loss=0.01654, audio_tagging_loss=0.009423, over 3044723.85 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 32.0 2023-11-21 12:03:09,808 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224150 2023-11-21 12:03:12,163 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.308e+01 8.854e+01 9.589e+01 1.258e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-21 12:03:28,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1494420.0, ans=0.1 2023-11-21 12:03:31,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1494420.0, ans=0.1 2023-11-21 12:03:41,680 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7750, loss[loss=0.08165, simple_loss=0.1117, pruned_loss=0.01634, audio_tagging_loss=0.009438, over 16294.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09499, pruned_loss=0.01651, audio_tagging_loss=0.009595, over 3043986.79 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:03:44,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1494486.6666666667, ans=0.125 2023-11-21 12:04:02,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1494553.3333333333, ans=0.125 2023-11-21 12:04:14,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224200 2023-11-21 12:04:14,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1494620.0, ans=0.2 2023-11-21 12:04:43,017 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:04:45,901 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7800, loss[loss=0.06972, simple_loss=0.08324, pruned_loss=0.01504, audio_tagging_loss=0.01306, over 15014.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09606, pruned_loss=0.0167, audio_tagging_loss=0.009596, over 3047950.92 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:04:55,411 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.73 vs. limit=15.0 2023-11-21 12:04:58,623 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.37 vs. limit=15.0 2023-11-21 12:05:06,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1494886.6666666667, ans=0.5 2023-11-21 12:05:14,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1494953.3333333333, ans=0.05 2023-11-21 12:05:18,290 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224250 2023-11-21 12:05:18,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1494953.3333333333, ans=0.125 2023-11-21 12:05:21,778 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.303e+01 8.766e+01 9.637e+01 1.238e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-21 12:05:25,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1495020.0, ans=0.125 2023-11-21 12:05:32,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1495020.0, ans=0.125 2023-11-21 12:05:33,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1495020.0, ans=0.0 2023-11-21 12:05:45,851 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.25 vs. limit=15.0 2023-11-21 12:05:50,458 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7850, loss[loss=0.08485, simple_loss=0.1115, pruned_loss=0.02111, audio_tagging_loss=0.007997, over 14008.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09578, pruned_loss=0.01671, audio_tagging_loss=0.009605, over 3049044.87 frames. ], batch size: 53, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:06:05,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1495220.0, ans=0.125 2023-11-21 12:06:08,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1495220.0, ans=0.1 2023-11-21 12:06:16,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1495286.6666666667, ans=0.2 2023-11-21 12:06:21,426 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224300 2023-11-21 12:06:44,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1495420.0, ans=0.1 2023-11-21 12:06:44,714 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.83 vs. limit=15.0 2023-11-21 12:06:51,976 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.70 vs. limit=15.0 2023-11-21 12:06:53,994 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7900, loss[loss=0.06972, simple_loss=0.09118, pruned_loss=0.01183, audio_tagging_loss=0.0123, over 15010.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09612, pruned_loss=0.01674, audio_tagging_loss=0.009686, over 3042480.17 frames. ], batch size: 54, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:07:01,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1495486.6666666667, ans=0.0 2023-11-21 12:07:03,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.60 vs. limit=10.0 2023-11-21 12:07:26,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224350 2023-11-21 12:07:29,890 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.237e+01 8.708e+01 9.758e+01 1.228e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-21 12:07:56,987 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 7950, loss[loss=0.07625, simple_loss=0.0904, pruned_loss=0.01882, audio_tagging_loss=0.01223, over 15029.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09532, pruned_loss=0.01667, audio_tagging_loss=0.009797, over 3047168.27 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:08:11,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1495886.6666666667, ans=0.125 2023-11-21 12:08:14,102 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:08:29,278 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224400 2023-11-21 12:08:34,851 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.91 vs. limit=15.0 2023-11-21 12:08:35,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1496020.0, ans=0.05 2023-11-21 12:08:44,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1496020.0, ans=0.2 2023-11-21 12:08:55,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1496086.6666666667, ans=0.125 2023-11-21 12:09:01,121 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8000, loss[loss=0.0755, simple_loss=0.1018, pruned_loss=0.01598, audio_tagging_loss=0.008643, over 14900.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09482, pruned_loss=0.01658, audio_tagging_loss=0.009867, over 3038675.11 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 32.0 2023-11-21 12:09:03,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1496153.3333333333, ans=0.125 2023-11-21 12:09:22,864 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.65 vs. limit=15.0 2023-11-21 12:09:23,990 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.46 vs. limit=15.0 2023-11-21 12:09:32,089 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224450 2023-11-21 12:09:35,517 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.187e+01 8.210e+01 8.871e+01 9.654e+01 1.306e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-21 12:09:44,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.81 vs. limit=10.0 2023-11-21 12:09:49,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1496353.3333333333, ans=0.125 2023-11-21 12:09:49,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1496353.3333333333, ans=0.0 2023-11-21 12:10:03,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1496486.6666666667, ans=0.0 2023-11-21 12:10:04,409 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8050, loss[loss=0.06795, simple_loss=0.09293, pruned_loss=0.00995, audio_tagging_loss=0.01154, over 17179.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09522, pruned_loss=0.01662, audio_tagging_loss=0.009915, over 3042818.09 frames. ], batch size: 63, lr: 3.59e-03, grad_scale: 32.0 2023-11-21 12:10:06,233 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.71 vs. limit=15.0 2023-11-21 12:10:33,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1496620.0, ans=0.0 2023-11-21 12:10:35,906 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224500 2023-11-21 12:10:43,152 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.07 vs. limit=15.0 2023-11-21 12:11:00,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1496753.3333333333, ans=0.125 2023-11-21 12:11:07,047 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8100, loss[loss=0.05489, simple_loss=0.06507, pruned_loss=0.008462, audio_tagging_loss=0.0139, over 14819.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09456, pruned_loss=0.01648, audio_tagging_loss=0.009958, over 3038465.78 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 32.0 2023-11-21 12:11:39,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224550 2023-11-21 12:11:39,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1496953.3333333333, ans=0.0 2023-11-21 12:11:43,197 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.553e+01 8.024e+01 8.529e+01 9.304e+01 1.158e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-21 12:12:04,251 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:12:10,667 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8150, loss[loss=0.06086, simple_loss=0.07592, pruned_loss=0.01498, audio_tagging_loss=0.007925, over 14295.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09444, pruned_loss=0.01638, audio_tagging_loss=0.009778, over 3040439.34 frames. ], batch size: 53, lr: 3.59e-03, grad_scale: 32.0 2023-11-21 12:12:27,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1497220.0, ans=0.0 2023-11-21 12:12:42,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224600 2023-11-21 12:13:14,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1497486.6666666667, ans=0.125 2023-11-21 12:13:15,664 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8200, loss[loss=0.07254, simple_loss=0.09588, pruned_loss=0.01708, audio_tagging_loss=0.007515, over 14650.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.09524, pruned_loss=0.01664, audio_tagging_loss=0.009671, over 3041831.93 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:13:16,939 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:13:28,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1497553.3333333333, ans=0.125 2023-11-21 12:13:31,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1497553.3333333333, ans=0.0 2023-11-21 12:13:41,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1497620.0, ans=0.125 2023-11-21 12:13:46,783 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224650 2023-11-21 12:13:52,057 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.039e+01 8.805e+01 9.521e+01 1.121e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-21 12:13:59,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1497686.6666666667, ans=0.0 2023-11-21 12:14:11,055 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.40 vs. limit=22.5 2023-11-21 12:14:13,218 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.14 vs. limit=15.0 2023-11-21 12:14:18,896 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8250, loss[loss=0.06895, simple_loss=0.08069, pruned_loss=0.01737, audio_tagging_loss=0.01122, over 14692.00 frames. ], tot_loss[loss=0.07433, simple_loss=0.09578, pruned_loss=0.0168, audio_tagging_loss=0.009633, over 3047431.71 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:14:46,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1497953.3333333333, ans=0.0 2023-11-21 12:14:49,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=1497953.3333333333, ans=15.0 2023-11-21 12:14:51,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224700 2023-11-21 12:14:58,978 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.05 vs. limit=10.0 2023-11-21 12:15:06,290 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.49 vs. limit=15.0 2023-11-21 12:15:22,104 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8300, loss[loss=0.08295, simple_loss=0.1128, pruned_loss=0.01846, audio_tagging_loss=0.008107, over 15197.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09518, pruned_loss=0.01643, audio_tagging_loss=0.009751, over 3049951.68 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 8.0 2023-11-21 12:15:27,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1498153.3333333333, ans=0.0 2023-11-21 12:15:54,057 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224750 2023-11-21 12:16:00,005 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.024e+01 7.847e+01 8.529e+01 9.488e+01 1.210e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-21 12:16:01,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1498353.3333333333, ans=0.5 2023-11-21 12:16:14,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.54 vs. limit=15.0 2023-11-21 12:16:27,138 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8350, loss[loss=0.07916, simple_loss=0.09602, pruned_loss=0.01875, audio_tagging_loss=0.0124, over 14054.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.0948, pruned_loss=0.01634, audio_tagging_loss=0.009634, over 3048917.72 frames. ], batch size: 53, lr: 3.59e-03, grad_scale: 8.0 2023-11-21 12:16:49,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1498553.3333333333, ans=0.125 2023-11-21 12:16:54,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.63 vs. limit=15.0 2023-11-21 12:16:58,283 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224800 2023-11-21 12:17:07,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1498686.6666666667, ans=0.0 2023-11-21 12:17:28,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1498753.3333333333, ans=0.2 2023-11-21 12:17:28,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1498753.3333333333, ans=0.0 2023-11-21 12:17:30,430 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8400, loss[loss=0.09443, simple_loss=0.1269, pruned_loss=0.0228, audio_tagging_loss=0.008202, over 14670.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09449, pruned_loss=0.01631, audio_tagging_loss=0.009657, over 3043697.28 frames. ], batch size: 54, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:17:36,257 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.18 vs. limit=15.0 2023-11-21 12:17:39,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.74 vs. limit=10.0 2023-11-21 12:17:57,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1498953.3333333333, ans=0.125 2023-11-21 12:18:03,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224850 2023-11-21 12:18:09,638 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.252e+01 9.012e+01 9.570e+01 1.298e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-21 12:18:12,718 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.20 vs. limit=10.0 2023-11-21 12:18:24,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1499086.6666666667, ans=0.2 2023-11-21 12:18:30,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1499086.6666666667, ans=0.125 2023-11-21 12:18:34,193 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8450, loss[loss=0.06671, simple_loss=0.0848, pruned_loss=0.01428, audio_tagging_loss=0.01003, over 14073.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09444, pruned_loss=0.01627, audio_tagging_loss=0.009723, over 3042540.59 frames. ], batch size: 54, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:18:43,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1499153.3333333333, ans=0.125 2023-11-21 12:18:45,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1499153.3333333333, ans=0.125 2023-11-21 12:18:47,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1499220.0, ans=0.1 2023-11-21 12:19:01,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1499286.6666666667, ans=15.0 2023-11-21 12:19:06,956 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224900 2023-11-21 12:19:19,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.37 vs. limit=15.0 2023-11-21 12:19:22,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1499353.3333333333, ans=0.125 2023-11-21 12:19:34,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1499420.0, ans=0.0 2023-11-21 12:19:39,587 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8500, loss[loss=0.07226, simple_loss=0.08349, pruned_loss=0.01633, audio_tagging_loss=0.01419, over 14190.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09471, pruned_loss=0.01636, audio_tagging_loss=0.009609, over 3035751.48 frames. ], batch size: 53, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:19:58,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1499553.3333333333, ans=0.2 2023-11-21 12:20:01,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1499553.3333333333, ans=0.125 2023-11-21 12:20:10,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 224950 2023-11-21 12:20:16,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1499686.6666666667, ans=0.125 2023-11-21 12:20:17,531 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.782e+01 8.392e+01 8.931e+01 9.765e+01 1.184e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-21 12:20:29,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1499686.6666666667, ans=0.1 2023-11-21 12:20:44,419 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8550, loss[loss=0.0616, simple_loss=0.0734, pruned_loss=0.01427, audio_tagging_loss=0.01063, over 16504.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09568, pruned_loss=0.01667, audio_tagging_loss=0.009587, over 3040187.39 frames. ], batch size: 62, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:20:47,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1499820.0, ans=0.125 2023-11-21 12:20:50,228 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.31 vs. limit=15.0 2023-11-21 12:20:56,101 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=12.0 2023-11-21 12:21:16,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225000 2023-11-21 12:21:22,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1500020.0, ans=0.125 2023-11-21 12:21:23,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1500020.0, ans=0.0 2023-11-21 12:21:48,074 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8600, loss[loss=0.07603, simple_loss=0.09475, pruned_loss=0.01836, audio_tagging_loss=0.01029, over 14358.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09597, pruned_loss=0.01671, audio_tagging_loss=0.009644, over 3035315.18 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:22:10,016 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.86 vs. limit=12.0 2023-11-21 12:22:21,270 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225050 2023-11-21 12:22:21,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1500286.6666666667, ans=0.0 2023-11-21 12:22:21,846 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.43 vs. limit=15.0 2023-11-21 12:22:27,342 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.106e+01 8.713e+01 9.527e+01 1.348e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 12:22:31,415 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:22:37,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1500353.3333333333, ans=0.125 2023-11-21 12:22:39,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1500420.0, ans=0.2 2023-11-21 12:22:42,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1500420.0, ans=0.95 2023-11-21 12:22:52,974 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8650, loss[loss=0.08235, simple_loss=0.1062, pruned_loss=0.01755, audio_tagging_loss=0.01168, over 15498.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.09616, pruned_loss=0.01677, audio_tagging_loss=0.009667, over 3041046.38 frames. ], batch size: 60, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:23:02,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1500486.6666666667, ans=0.025 2023-11-21 12:23:03,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1500486.6666666667, ans=0.0 2023-11-21 12:23:21,778 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.36 vs. limit=15.0 2023-11-21 12:23:25,135 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225100 2023-11-21 12:23:28,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1500620.0, ans=0.0 2023-11-21 12:23:39,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1500686.6666666667, ans=0.0 2023-11-21 12:23:54,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1500753.3333333333, ans=0.1 2023-11-21 12:23:57,073 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8700, loss[loss=0.068, simple_loss=0.08244, pruned_loss=0.01397, audio_tagging_loss=0.0128, over 15401.00 frames. ], tot_loss[loss=0.07501, simple_loss=0.0966, pruned_loss=0.01692, audio_tagging_loss=0.00979, over 3047683.54 frames. ], batch size: 61, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:23:57,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1500820.0, ans=0.2 2023-11-21 12:24:00,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1500820.0, ans=0.125 2023-11-21 12:24:09,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1500886.6666666667, ans=0.125 2023-11-21 12:24:23,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.91 vs. limit=15.0 2023-11-21 12:24:24,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1500953.3333333333, ans=0.1 2023-11-21 12:24:29,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225150 2023-11-21 12:24:35,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.32 vs. limit=15.0 2023-11-21 12:24:35,999 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.442e+01 8.072e+01 8.843e+01 9.223e+01 1.204e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 12:24:37,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1501020.0, ans=0.0 2023-11-21 12:24:39,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1501020.0, ans=0.125 2023-11-21 12:24:51,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1501086.6666666667, ans=0.125 2023-11-21 12:24:59,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1501153.3333333333, ans=0.125 2023-11-21 12:25:00,548 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8750, loss[loss=0.0889, simple_loss=0.1172, pruned_loss=0.02265, audio_tagging_loss=0.007669, over 15023.00 frames. ], tot_loss[loss=0.07517, simple_loss=0.09701, pruned_loss=0.01684, audio_tagging_loss=0.009819, over 3053494.21 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:25:07,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1501153.3333333333, ans=0.1 2023-11-21 12:25:32,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225200 2023-11-21 12:25:34,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1501286.6666666667, ans=22.5 2023-11-21 12:25:36,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1501286.6666666667, ans=10.0 2023-11-21 12:25:55,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1501420.0, ans=0.125 2023-11-21 12:26:05,340 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8800, loss[loss=0.08037, simple_loss=0.11, pruned_loss=0.01693, audio_tagging_loss=0.008431, over 15132.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09839, pruned_loss=0.0173, audio_tagging_loss=0.009876, over 3060114.28 frames. ], batch size: 54, lr: 3.59e-03, grad_scale: 32.0 2023-11-21 12:26:29,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1501620.0, ans=0.2 2023-11-21 12:26:37,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225250 2023-11-21 12:26:43,228 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.062e+01 8.940e+01 9.452e+01 1.337e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-21 12:27:09,830 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8850, loss[loss=0.06884, simple_loss=0.08527, pruned_loss=0.01645, audio_tagging_loss=0.009757, over 14640.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.0977, pruned_loss=0.01716, audio_tagging_loss=0.009917, over 3059349.03 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 32.0 2023-11-21 12:27:15,311 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.12 vs. limit=15.0 2023-11-21 12:27:22,952 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:27:42,712 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225300 2023-11-21 12:27:58,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1502020.0, ans=0.2 2023-11-21 12:28:15,026 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8900, loss[loss=0.07753, simple_loss=0.09898, pruned_loss=0.01908, audio_tagging_loss=0.008957, over 16299.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09725, pruned_loss=0.017, audio_tagging_loss=0.009716, over 3059468.91 frames. ], batch size: 59, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:28:19,186 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:28:30,237 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:28:33,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1502220.0, ans=0.0 2023-11-21 12:28:47,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225350 2023-11-21 12:28:54,894 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.032e+01 8.160e+01 8.777e+01 9.579e+01 1.339e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 12:29:01,321 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:29:03,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1502353.3333333333, ans=0.07 2023-11-21 12:29:20,435 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 8950, loss[loss=0.08246, simple_loss=0.1166, pruned_loss=0.01545, audio_tagging_loss=0.008722, over 15404.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09847, pruned_loss=0.01726, audio_tagging_loss=0.009561, over 3058827.05 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:29:37,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1502553.3333333333, ans=0.125 2023-11-21 12:29:39,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1502553.3333333333, ans=0.125 2023-11-21 12:29:42,589 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.87 vs. limit=15.0 2023-11-21 12:29:43,865 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.04 vs. limit=15.0 2023-11-21 12:29:51,925 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225400 2023-11-21 12:30:04,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1502686.6666666667, ans=0.125 2023-11-21 12:30:05,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1502686.6666666667, ans=0.0 2023-11-21 12:30:25,208 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9000, loss[loss=0.06983, simple_loss=0.08661, pruned_loss=0.01804, audio_tagging_loss=0.008494, over 14790.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09839, pruned_loss=0.01736, audio_tagging_loss=0.009471, over 3057273.84 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0 2023-11-21 12:30:25,209 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 12:31:06,235 INFO [train_asr.py:1253] (3/4) Epoch 19, validation: loss=0.06044, simple_loss=0.05233, pruned_loss=0.005297, audio_tagging_loss=0.02898, over 4681554.00 frames. 2023-11-21 12:31:06,236 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 12:31:09,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1502820.0, ans=0.0 2023-11-21 12:31:25,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1502886.6666666667, ans=0.125 2023-11-21 12:31:26,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1502886.6666666667, ans=0.1 2023-11-21 12:31:38,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225450 2023-11-21 12:31:45,865 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.087e+01 8.929e+01 9.690e+01 1.260e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-21 12:32:09,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1503086.6666666667, ans=0.09899494936611666 2023-11-21 12:32:11,478 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9050, loss[loss=0.076, simple_loss=0.102, pruned_loss=0.01771, audio_tagging_loss=0.007298, over 14761.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09832, pruned_loss=0.01738, audio_tagging_loss=0.009379, over 3057784.70 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 16.0 2023-11-21 12:32:27,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1503220.0, ans=0.1 2023-11-21 12:32:42,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225500 2023-11-21 12:33:12,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1503420.0, ans=6.0 2023-11-21 12:33:15,759 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9100, loss[loss=0.0747, simple_loss=0.09682, pruned_loss=0.01515, audio_tagging_loss=0.01115, over 15220.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09637, pruned_loss=0.01699, audio_tagging_loss=0.009365, over 3054039.85 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 16.0 2023-11-21 12:33:28,589 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.48 vs. limit=15.0 2023-11-21 12:33:30,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1503553.3333333333, ans=0.0 2023-11-21 12:33:41,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1503620.0, ans=0.125 2023-11-21 12:33:48,287 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225550 2023-11-21 12:33:51,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1503620.0, ans=0.125 2023-11-21 12:33:56,025 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.533e+01 8.001e+01 8.590e+01 9.455e+01 1.329e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-21 12:33:56,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1503686.6666666667, ans=0.0 2023-11-21 12:34:19,563 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9150, loss[loss=0.0795, simple_loss=0.1033, pruned_loss=0.01801, audio_tagging_loss=0.009857, over 16024.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09687, pruned_loss=0.01705, audio_tagging_loss=0.009327, over 3051847.35 frames. ], batch size: 60, lr: 3.58e-03, grad_scale: 16.0 2023-11-21 12:34:52,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225600 2023-11-21 12:34:54,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1503953.3333333333, ans=0.2 2023-11-21 12:34:58,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1504020.0, ans=0.125 2023-11-21 12:35:01,577 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.45 vs. limit=15.0 2023-11-21 12:35:04,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1504020.0, ans=0.1 2023-11-21 12:35:22,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1504086.6666666667, ans=0.2 2023-11-21 12:35:24,763 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9200, loss[loss=0.05514, simple_loss=0.07531, pruned_loss=0.007678, audio_tagging_loss=0.009807, over 14961.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09576, pruned_loss=0.01675, audio_tagging_loss=0.009368, over 3059296.39 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:35:30,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1504153.3333333333, ans=0.5 2023-11-21 12:35:50,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1504286.6666666667, ans=0.05 2023-11-21 12:35:53,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1504286.6666666667, ans=0.1 2023-11-21 12:35:56,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225650 2023-11-21 12:35:56,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1504286.6666666667, ans=0.0 2023-11-21 12:35:56,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1504286.6666666667, ans=0.2 2023-11-21 12:35:57,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=22.5 2023-11-21 12:35:58,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1504286.6666666667, ans=0.0 2023-11-21 12:36:03,556 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.007e+01 8.520e+01 9.210e+01 1.380e+02, threshold=1.704e+02, percent-clipped=0.0 2023-11-21 12:36:14,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.56 vs. limit=15.0 2023-11-21 12:36:29,478 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9250, loss[loss=0.06142, simple_loss=0.076, pruned_loss=0.01208, audio_tagging_loss=0.01135, over 14861.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09567, pruned_loss=0.01676, audio_tagging_loss=0.009399, over 3058659.49 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:36:35,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1504486.6666666667, ans=0.125 2023-11-21 12:36:38,564 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.49 vs. limit=15.0 2023-11-21 12:37:01,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225700 2023-11-21 12:37:03,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1504620.0, ans=0.0 2023-11-21 12:37:29,596 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:37:33,053 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9300, loss[loss=0.06768, simple_loss=0.08756, pruned_loss=0.01544, audio_tagging_loss=0.008464, over 16070.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09619, pruned_loss=0.01669, audio_tagging_loss=0.009368, over 3054453.39 frames. ], batch size: 61, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:37:33,674 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.38 vs. limit=15.0 2023-11-21 12:37:38,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1504820.0, ans=0.125 2023-11-21 12:37:53,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1504886.6666666667, ans=0.125 2023-11-21 12:38:02,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1504953.3333333333, ans=0.125 2023-11-21 12:38:05,495 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225750 2023-11-21 12:38:12,759 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 7.914e+01 8.650e+01 9.354e+01 1.480e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-21 12:38:15,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1505020.0, ans=0.125 2023-11-21 12:38:18,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1505020.0, ans=0.1 2023-11-21 12:38:37,026 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9350, loss[loss=0.07283, simple_loss=0.08531, pruned_loss=0.01744, audio_tagging_loss=0.01274, over 15245.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09639, pruned_loss=0.01658, audio_tagging_loss=0.009389, over 3047387.67 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:38:38,974 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.54 vs. limit=15.0 2023-11-21 12:38:47,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1505153.3333333333, ans=0.0 2023-11-21 12:39:01,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1505286.6666666667, ans=0.0 2023-11-21 12:39:04,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1505286.6666666667, ans=0.0 2023-11-21 12:39:09,026 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225800 2023-11-21 12:39:17,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1505353.3333333333, ans=0.0 2023-11-21 12:39:41,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1505486.6666666667, ans=0.125 2023-11-21 12:39:42,119 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9400, loss[loss=0.082, simple_loss=0.1057, pruned_loss=0.01842, audio_tagging_loss=0.01074, over 16378.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09553, pruned_loss=0.01636, audio_tagging_loss=0.009493, over 3042983.41 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:39:58,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1505553.3333333333, ans=0.1 2023-11-21 12:40:03,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.0 2023-11-21 12:40:12,727 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225850 2023-11-21 12:40:16,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1505620.0, ans=0.2 2023-11-21 12:40:22,995 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.417e+01 9.121e+01 9.724e+01 1.188e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-21 12:40:32,275 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.55 vs. limit=15.0 2023-11-21 12:40:40,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1505753.3333333333, ans=22.5 2023-11-21 12:40:42,770 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:40:45,262 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9450, loss[loss=0.05561, simple_loss=0.07013, pruned_loss=0.01168, audio_tagging_loss=0.008869, over 15109.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09664, pruned_loss=0.01673, audio_tagging_loss=0.009546, over 3045764.08 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 16.0 2023-11-21 12:40:45,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1505820.0, ans=0.0 2023-11-21 12:40:58,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=22.5 2023-11-21 12:41:00,326 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:41:03,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1505886.6666666667, ans=0.125 2023-11-21 12:41:17,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225900 2023-11-21 12:41:20,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1505953.3333333333, ans=0.2 2023-11-21 12:41:23,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.94 vs. limit=15.0 2023-11-21 12:41:48,210 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9500, loss[loss=0.07354, simple_loss=0.09356, pruned_loss=0.01541, audio_tagging_loss=0.01135, over 15029.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09603, pruned_loss=0.01661, audio_tagging_loss=0.009583, over 3041392.54 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 16.0 2023-11-21 12:41:51,870 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.91 vs. limit=15.0 2023-11-21 12:42:15,319 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.82 vs. limit=15.0 2023-11-21 12:42:20,934 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 225950 2023-11-21 12:42:28,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0 2023-11-21 12:42:29,403 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.362e+01 9.191e+01 9.965e+01 1.276e+02, threshold=1.838e+02, percent-clipped=0.0 2023-11-21 12:42:43,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1506420.0, ans=0.0 2023-11-21 12:42:52,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1506486.6666666667, ans=0.125 2023-11-21 12:42:53,262 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9550, loss[loss=0.05375, simple_loss=0.06717, pruned_loss=0.008307, audio_tagging_loss=0.01186, over 13779.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.0967, pruned_loss=0.0166, audio_tagging_loss=0.009677, over 3043936.42 frames. ], batch size: 53, lr: 3.58e-03, grad_scale: 16.0 2023-11-21 12:42:57,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1506486.6666666667, ans=0.0 2023-11-21 12:43:18,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1506620.0, ans=0.2 2023-11-21 12:43:24,277 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226000 2023-11-21 12:43:24,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1506620.0, ans=0.1 2023-11-21 12:43:44,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1506753.3333333333, ans=0.125 2023-11-21 12:43:57,185 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9600, loss[loss=0.07154, simple_loss=0.1042, pruned_loss=0.0123, audio_tagging_loss=0.007125, over 15604.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09711, pruned_loss=0.01662, audio_tagging_loss=0.00971, over 3044757.61 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:44:12,728 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.08 vs. limit=15.0 2023-11-21 12:44:24,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1506953.3333333333, ans=0.125 2023-11-21 12:44:29,265 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226050 2023-11-21 12:44:37,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1507020.0, ans=0.125 2023-11-21 12:44:38,334 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 7.983e+01 8.743e+01 9.613e+01 1.291e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-21 12:44:57,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1507086.6666666667, ans=0.125 2023-11-21 12:44:59,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1507153.3333333333, ans=15.0 2023-11-21 12:45:00,260 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9650, loss[loss=0.08053, simple_loss=0.1064, pruned_loss=0.01927, audio_tagging_loss=0.008064, over 16232.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09691, pruned_loss=0.01665, audio_tagging_loss=0.009601, over 3044607.79 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:45:04,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1507153.3333333333, ans=0.0 2023-11-21 12:45:33,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226100 2023-11-21 12:46:00,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=12.0 2023-11-21 12:46:05,038 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9700, loss[loss=0.06678, simple_loss=0.08583, pruned_loss=0.01588, audio_tagging_loss=0.00798, over 15024.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09675, pruned_loss=0.01656, audio_tagging_loss=0.00952, over 3045906.63 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:46:12,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1507486.6666666667, ans=0.125 2023-11-21 12:46:16,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1507486.6666666667, ans=0.0 2023-11-21 12:46:22,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1507553.3333333333, ans=0.125 2023-11-21 12:46:24,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1507553.3333333333, ans=0.0 2023-11-21 12:46:26,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1507553.3333333333, ans=0.0 2023-11-21 12:46:36,898 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226150 2023-11-21 12:46:37,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.65 vs. limit=6.0 2023-11-21 12:46:38,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.45 vs. limit=22.5 2023-11-21 12:46:45,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1507686.6666666667, ans=0.0 2023-11-21 12:46:46,071 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 7.992e+01 8.607e+01 9.446e+01 1.380e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-21 12:46:54,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.16 vs. limit=15.0 2023-11-21 12:47:08,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1507820.0, ans=0.125 2023-11-21 12:47:09,289 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9750, loss[loss=0.07924, simple_loss=0.09198, pruned_loss=0.01991, audio_tagging_loss=0.01334, over 15237.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09643, pruned_loss=0.01646, audio_tagging_loss=0.009488, over 3053416.78 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:47:27,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.07 vs. limit=22.5 2023-11-21 12:47:41,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226200 2023-11-21 12:48:06,616 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.74 vs. limit=15.0 2023-11-21 12:48:13,258 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9800, loss[loss=0.06774, simple_loss=0.09186, pruned_loss=0.01131, audio_tagging_loss=0.01049, over 15514.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09641, pruned_loss=0.01639, audio_tagging_loss=0.009414, over 3060446.46 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:48:13,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1508153.3333333333, ans=0.0 2023-11-21 12:48:17,140 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:48:25,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1508220.0, ans=0.125 2023-11-21 12:48:45,546 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226250 2023-11-21 12:48:45,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1508286.6666666667, ans=0.125 2023-11-21 12:48:53,952 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.494e+01 7.972e+01 8.418e+01 9.313e+01 1.119e+02, threshold=1.684e+02, percent-clipped=0.0 2023-11-21 12:48:55,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1508353.3333333333, ans=0.1 2023-11-21 12:49:09,518 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:49:10,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.62 vs. limit=15.0 2023-11-21 12:49:17,410 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9850, loss[loss=0.07713, simple_loss=0.1029, pruned_loss=0.01784, audio_tagging_loss=0.007811, over 16874.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.09771, pruned_loss=0.01664, audio_tagging_loss=0.009325, over 3063407.22 frames. ], batch size: 64, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:49:23,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1508486.6666666667, ans=0.0 2023-11-21 12:49:29,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1508553.3333333333, ans=0.0 2023-11-21 12:49:43,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1508620.0, ans=0.125 2023-11-21 12:49:47,977 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.87 vs. limit=15.0 2023-11-21 12:49:48,678 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226300 2023-11-21 12:49:48,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1508620.0, ans=0.125 2023-11-21 12:50:12,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1508753.3333333333, ans=0.0 2023-11-21 12:50:15,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1508753.3333333333, ans=0.125 2023-11-21 12:50:18,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1508753.3333333333, ans=0.125 2023-11-21 12:50:20,967 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9900, loss[loss=0.09937, simple_loss=0.1366, pruned_loss=0.02567, audio_tagging_loss=0.005403, over 15592.00 frames. ], tot_loss[loss=0.0749, simple_loss=0.09759, pruned_loss=0.01669, audio_tagging_loss=0.009417, over 3065342.96 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:50:43,128 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.70 vs. limit=15.0 2023-11-21 12:50:53,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226350 2023-11-21 12:50:55,844 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:51:02,738 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.161e+01 8.727e+01 9.211e+01 1.112e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-21 12:51:14,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1509086.6666666667, ans=0.125 2023-11-21 12:51:24,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1509153.3333333333, ans=0.125 2023-11-21 12:51:25,641 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 9950, loss[loss=0.07584, simple_loss=0.1023, pruned_loss=0.01842, audio_tagging_loss=0.006264, over 15458.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09694, pruned_loss=0.0165, audio_tagging_loss=0.009397, over 3059566.65 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:51:44,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1509220.0, ans=0.125 2023-11-21 12:51:57,939 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226400 2023-11-21 12:51:59,554 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.23 vs. limit=22.5 2023-11-21 12:52:00,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1509286.6666666667, ans=0.125 2023-11-21 12:52:01,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.81 vs. limit=15.0 2023-11-21 12:52:02,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1509286.6666666667, ans=0.125 2023-11-21 12:52:17,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1509420.0, ans=0.125 2023-11-21 12:52:18,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1509420.0, ans=0.2 2023-11-21 12:52:29,828 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10000, loss[loss=0.06146, simple_loss=0.08203, pruned_loss=0.01205, audio_tagging_loss=0.008392, over 14664.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09644, pruned_loss=0.01645, audio_tagging_loss=0.009427, over 3053802.51 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:52:40,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1509486.6666666667, ans=0.0 2023-11-21 12:52:51,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1509553.3333333333, ans=0.0 2023-11-21 12:53:01,805 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226450 2023-11-21 12:53:10,190 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 7.892e+01 8.555e+01 9.256e+01 1.223e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-21 12:53:33,294 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10050, loss[loss=0.0749, simple_loss=0.09563, pruned_loss=0.01703, audio_tagging_loss=0.01006, over 14933.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.0959, pruned_loss=0.01651, audio_tagging_loss=0.009597, over 3056439.18 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:53:34,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1509820.0, ans=0.125 2023-11-21 12:53:37,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1509820.0, ans=0.0 2023-11-21 12:53:38,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.16 vs. limit=22.5 2023-11-21 12:54:05,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226500 2023-11-21 12:54:36,863 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10100, loss[loss=0.07127, simple_loss=0.09695, pruned_loss=0.01304, audio_tagging_loss=0.00976, over 15446.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09621, pruned_loss=0.01667, audio_tagging_loss=0.009673, over 3058526.27 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:54:37,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1510153.3333333333, ans=0.1 2023-11-21 12:54:38,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1510153.3333333333, ans=0.0 2023-11-21 12:55:08,760 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226550 2023-11-21 12:55:08,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1510286.6666666667, ans=0.2 2023-11-21 12:55:12,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1510286.6666666667, ans=0.1 2023-11-21 12:55:12,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1510286.6666666667, ans=0.2 2023-11-21 12:55:17,039 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.133e+01 8.640e+01 9.422e+01 1.310e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-21 12:55:27,656 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:55:40,527 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10150, loss[loss=0.09029, simple_loss=0.1139, pruned_loss=0.02246, audio_tagging_loss=0.01088, over 14865.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09683, pruned_loss=0.01684, audio_tagging_loss=0.009723, over 3056132.14 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:56:07,538 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=6.861e-02 2023-11-21 12:56:09,680 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:56:12,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226600 2023-11-21 12:56:16,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.88 vs. limit=15.0 2023-11-21 12:56:20,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1510686.6666666667, ans=0.125 2023-11-21 12:56:21,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1510686.6666666667, ans=0.125 2023-11-21 12:56:24,283 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:56:25,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1510686.6666666667, ans=0.125 2023-11-21 12:56:40,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1510753.3333333333, ans=0.125 2023-11-21 12:56:44,930 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10200, loss[loss=0.07646, simple_loss=0.1016, pruned_loss=0.01807, audio_tagging_loss=0.007607, over 14680.00 frames. ], tot_loss[loss=0.07497, simple_loss=0.097, pruned_loss=0.01682, audio_tagging_loss=0.009651, over 3059360.78 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:56:57,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1510886.6666666667, ans=0.025 2023-11-21 12:57:07,497 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 12:57:16,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226650 2023-11-21 12:57:26,005 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.532e+01 7.915e+01 8.561e+01 9.293e+01 1.189e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-21 12:57:34,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1511086.6666666667, ans=0.125 2023-11-21 12:57:48,033 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10250, loss[loss=0.07691, simple_loss=0.09257, pruned_loss=0.02342, audio_tagging_loss=0.007201, over 14845.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09726, pruned_loss=0.01691, audio_tagging_loss=0.009698, over 3055712.82 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 32.0 2023-11-21 12:58:01,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1511220.0, ans=0.125 2023-11-21 12:58:12,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1511220.0, ans=0.2 2023-11-21 12:58:13,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1511286.6666666667, ans=0.0 2023-11-21 12:58:14,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1511286.6666666667, ans=0.0 2023-11-21 12:58:17,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1511286.6666666667, ans=0.125 2023-11-21 12:58:20,766 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226700 2023-11-21 12:58:44,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1511420.0, ans=0.1 2023-11-21 12:58:45,088 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.70 vs. limit=22.5 2023-11-21 12:58:46,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1511420.0, ans=0.1 2023-11-21 12:58:51,831 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10300, loss[loss=0.06997, simple_loss=0.0865, pruned_loss=0.01254, audio_tagging_loss=0.01419, over 15855.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09593, pruned_loss=0.0165, audio_tagging_loss=0.009853, over 3055280.56 frames. ], batch size: 60, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 12:58:55,077 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 12:58:59,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.57 vs. limit=15.0 2023-11-21 12:59:02,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1511486.6666666667, ans=0.125 2023-11-21 12:59:24,066 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226750 2023-11-21 12:59:32,483 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.325e+01 8.857e+01 9.782e+01 1.240e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-21 12:59:38,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1511686.6666666667, ans=0.125 2023-11-21 12:59:39,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1511686.6666666667, ans=0.0 2023-11-21 12:59:56,452 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10350, loss[loss=0.06938, simple_loss=0.08527, pruned_loss=0.01598, audio_tagging_loss=0.01076, over 16280.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09603, pruned_loss=0.01657, audio_tagging_loss=0.00996, over 3057165.45 frames. ], batch size: 61, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:00:17,938 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.69 vs. limit=15.0 2023-11-21 13:00:21,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1511953.3333333333, ans=0.125 2023-11-21 13:00:27,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226800 2023-11-21 13:00:30,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1511953.3333333333, ans=0.0 2023-11-21 13:00:31,597 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.52 vs. limit=15.0 2023-11-21 13:00:51,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0 2023-11-21 13:00:58,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.00 vs. limit=15.0 2023-11-21 13:01:00,002 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10400, loss[loss=0.09057, simple_loss=0.1294, pruned_loss=0.02004, audio_tagging_loss=0.005823, over 16400.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09643, pruned_loss=0.01679, audio_tagging_loss=0.009959, over 3054706.71 frames. ], batch size: 62, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:01:08,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1512153.3333333333, ans=0.0 2023-11-21 13:01:15,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1512220.0, ans=0.125 2023-11-21 13:01:32,573 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226850 2023-11-21 13:01:35,325 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:01:41,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1512353.3333333333, ans=0.125 2023-11-21 13:01:42,506 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.081e+01 8.812e+01 9.594e+01 2.017e+02, threshold=1.762e+02, percent-clipped=1.0 2023-11-21 13:01:47,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1512353.3333333333, ans=0.125 2023-11-21 13:02:03,533 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10450, loss[loss=0.07412, simple_loss=0.09237, pruned_loss=0.01681, audio_tagging_loss=0.01112, over 15453.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09605, pruned_loss=0.01677, audio_tagging_loss=0.009927, over 3048007.20 frames. ], batch size: 57, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:02:03,797 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:02:21,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1512553.3333333333, ans=0.0 2023-11-21 13:02:25,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1512553.3333333333, ans=0.0 2023-11-21 13:02:36,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226900 2023-11-21 13:03:08,419 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10500, loss[loss=0.09444, simple_loss=0.124, pruned_loss=0.02271, audio_tagging_loss=0.009713, over 16916.00 frames. ], tot_loss[loss=0.07429, simple_loss=0.09571, pruned_loss=0.01664, audio_tagging_loss=0.009799, over 3049817.56 frames. ], batch size: 62, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:03:09,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1512820.0, ans=0.125 2023-11-21 13:03:38,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 226950 2023-11-21 13:03:40,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1512953.3333333333, ans=0.125 2023-11-21 13:03:41,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1512953.3333333333, ans=0.0 2023-11-21 13:03:49,557 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.276e+01 8.777e+01 9.449e+01 1.286e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 13:04:11,141 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10550, loss[loss=0.07196, simple_loss=0.09108, pruned_loss=0.01539, audio_tagging_loss=0.01103, over 14803.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09604, pruned_loss=0.01665, audio_tagging_loss=0.009699, over 3044728.91 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:04:11,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1513153.3333333333, ans=0.0 2023-11-21 13:04:13,293 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.10 vs. limit=15.0 2023-11-21 13:04:13,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1513153.3333333333, ans=0.1 2023-11-21 13:04:23,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1513220.0, ans=0.125 2023-11-21 13:04:36,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1513286.6666666667, ans=0.125 2023-11-21 13:04:43,700 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227000 2023-11-21 13:04:56,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1513353.3333333333, ans=0.125 2023-11-21 13:05:14,600 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10600, loss[loss=0.06939, simple_loss=0.08418, pruned_loss=0.01783, audio_tagging_loss=0.009469, over 15620.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09577, pruned_loss=0.01651, audio_tagging_loss=0.009678, over 3046673.73 frames. ], batch size: 62, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:05:27,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1513553.3333333333, ans=0.125 2023-11-21 13:05:47,388 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227050 2023-11-21 13:05:57,030 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.166e+01 8.856e+01 9.761e+01 1.795e+02, threshold=1.771e+02, percent-clipped=1.0 2023-11-21 13:06:07,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1513753.3333333333, ans=0.1 2023-11-21 13:06:09,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1513753.3333333333, ans=0.125 2023-11-21 13:06:19,704 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10650, loss[loss=0.08272, simple_loss=0.1157, pruned_loss=0.0179, audio_tagging_loss=0.006947, over 15811.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09577, pruned_loss=0.01658, audio_tagging_loss=0.009549, over 3043731.37 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:06:21,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1513820.0, ans=0.125 2023-11-21 13:06:45,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1513953.3333333333, ans=0.1 2023-11-21 13:06:50,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227100 2023-11-21 13:06:55,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten.whitening_limit, batch_count=1514020.0, ans=15.0 2023-11-21 13:06:56,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1514020.0, ans=0.125 2023-11-21 13:07:06,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1514020.0, ans=0.2 2023-11-21 13:07:22,129 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10700, loss[loss=0.06517, simple_loss=0.0805, pruned_loss=0.01342, audio_tagging_loss=0.01149, over 14664.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.09548, pruned_loss=0.01655, audio_tagging_loss=0.009529, over 3033085.42 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:07:24,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1514153.3333333333, ans=0.04949747468305833 2023-11-21 13:07:38,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1514220.0, ans=0.125 2023-11-21 13:07:45,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.64 vs. limit=10.0 2023-11-21 13:07:47,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1514286.6666666667, ans=0.2 2023-11-21 13:07:55,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227150 2023-11-21 13:08:04,772 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.492e+01 8.011e+01 8.687e+01 9.462e+01 1.243e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 13:08:23,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1514420.0, ans=0.125 2023-11-21 13:08:25,498 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10750, loss[loss=0.08753, simple_loss=0.1114, pruned_loss=0.02399, audio_tagging_loss=0.007854, over 15831.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09543, pruned_loss=0.0165, audio_tagging_loss=0.009531, over 3040305.37 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:08:31,897 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:08:47,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1514553.3333333333, ans=0.05 2023-11-21 13:08:56,953 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:08:57,921 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227200 2023-11-21 13:09:13,855 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.40 vs. limit=22.5 2023-11-21 13:09:30,345 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10800, loss[loss=0.044, simple_loss=0.04394, pruned_loss=0.009111, audio_tagging_loss=0.01292, over 16540.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09521, pruned_loss=0.01654, audio_tagging_loss=0.009565, over 3045518.76 frames. ], batch size: 64, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:09:54,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1514953.3333333333, ans=0.0 2023-11-21 13:10:01,786 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227250 2023-11-21 13:10:13,781 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 7.864e+01 8.499e+01 9.519e+01 1.176e+02, threshold=1.700e+02, percent-clipped=0.0 2023-11-21 13:10:16,992 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.38 vs. limit=15.0 2023-11-21 13:10:22,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1515086.6666666667, ans=0.125 2023-11-21 13:10:26,518 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.63 vs. limit=15.0 2023-11-21 13:10:31,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1515086.6666666667, ans=0.125 2023-11-21 13:10:31,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1515086.6666666667, ans=0.125 2023-11-21 13:10:34,452 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10850, loss[loss=0.09471, simple_loss=0.1287, pruned_loss=0.02152, audio_tagging_loss=0.008812, over 16054.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09633, pruned_loss=0.01675, audio_tagging_loss=0.00954, over 3040003.60 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:10:35,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1515153.3333333333, ans=0.125 2023-11-21 13:10:38,493 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:11:01,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1515286.6666666667, ans=0.125 2023-11-21 13:11:06,251 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227300 2023-11-21 13:11:06,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1515286.6666666667, ans=0.5 2023-11-21 13:11:08,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1515286.6666666667, ans=0.1 2023-11-21 13:11:11,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1515353.3333333333, ans=0.125 2023-11-21 13:11:13,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1515353.3333333333, ans=0.125 2023-11-21 13:11:16,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1515353.3333333333, ans=0.0 2023-11-21 13:11:33,356 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 13:11:36,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1515420.0, ans=0.0 2023-11-21 13:11:38,240 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10900, loss[loss=0.09009, simple_loss=0.1216, pruned_loss=0.01974, audio_tagging_loss=0.009563, over 15372.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09782, pruned_loss=0.01674, audio_tagging_loss=0.009467, over 3046385.90 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:12:01,467 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.09 vs. limit=10.0 2023-11-21 13:12:11,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227350 2023-11-21 13:12:16,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1515686.6666666667, ans=10.0 2023-11-21 13:12:22,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.236e+01 8.108e+01 8.684e+01 9.152e+01 1.151e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 13:12:38,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1515753.3333333333, ans=0.125 2023-11-21 13:12:40,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1515753.3333333333, ans=0.2 2023-11-21 13:12:42,787 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 10950, loss[loss=0.06716, simple_loss=0.08504, pruned_loss=0.01409, audio_tagging_loss=0.01054, over 15477.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.0967, pruned_loss=0.01638, audio_tagging_loss=0.009544, over 3044152.50 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:12:44,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.90 vs. limit=15.0 2023-11-21 13:13:02,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1515886.6666666667, ans=0.125 2023-11-21 13:13:10,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1515953.3333333333, ans=0.125 2023-11-21 13:13:14,782 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227400 2023-11-21 13:13:24,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1516020.0, ans=0.2 2023-11-21 13:13:29,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.94 vs. limit=22.5 2023-11-21 13:13:39,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1516086.6666666667, ans=0.0 2023-11-21 13:13:40,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1516086.6666666667, ans=0.0 2023-11-21 13:13:45,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.20 vs. limit=15.0 2023-11-21 13:13:48,221 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11000, loss[loss=0.06766, simple_loss=0.08952, pruned_loss=0.01435, audio_tagging_loss=0.008541, over 16350.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.0963, pruned_loss=0.01622, audio_tagging_loss=0.009592, over 3042934.75 frames. ], batch size: 60, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:13:51,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1516153.3333333333, ans=0.125 2023-11-21 13:13:58,137 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 13:14:01,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.25 vs. limit=15.0 2023-11-21 13:14:08,723 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.79 vs. limit=22.5 2023-11-21 13:14:20,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227450 2023-11-21 13:14:32,302 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.986e+01 8.066e+01 8.741e+01 9.512e+01 1.178e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 13:14:52,774 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11050, loss[loss=0.05711, simple_loss=0.07091, pruned_loss=0.01041, audio_tagging_loss=0.01125, over 15037.00 frames. ], tot_loss[loss=0.07363, simple_loss=0.0954, pruned_loss=0.01625, audio_tagging_loss=0.009683, over 3042013.36 frames. ], batch size: 57, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:15:12,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1516553.3333333333, ans=0.125 2023-11-21 13:15:17,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.37 vs. limit=22.5 2023-11-21 13:15:23,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1516620.0, ans=0.125 2023-11-21 13:15:25,287 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227500 2023-11-21 13:15:30,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.51 vs. limit=15.0 2023-11-21 13:15:37,336 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:15:41,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1516686.6666666667, ans=0.1 2023-11-21 13:15:44,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.82 vs. limit=10.0 2023-11-21 13:15:53,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1516753.3333333333, ans=0.125 2023-11-21 13:15:57,832 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11100, loss[loss=0.0817, simple_loss=0.1029, pruned_loss=0.02173, audio_tagging_loss=0.008523, over 14332.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.0961, pruned_loss=0.01632, audio_tagging_loss=0.009758, over 3044554.97 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:16:27,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1516953.3333333333, ans=0.1 2023-11-21 13:16:30,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227550 2023-11-21 13:16:31,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1516953.3333333333, ans=0.1 2023-11-21 13:16:39,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1517020.0, ans=0.0 2023-11-21 13:16:39,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1517020.0, ans=0.125 2023-11-21 13:16:41,849 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.109e+01 8.821e+01 9.588e+01 2.410e+02, threshold=1.764e+02, percent-clipped=1.0 2023-11-21 13:16:46,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1517020.0, ans=0.0 2023-11-21 13:16:59,887 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:17:03,333 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11150, loss[loss=0.06224, simple_loss=0.07528, pruned_loss=0.01358, audio_tagging_loss=0.01103, over 14373.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09575, pruned_loss=0.01625, audio_tagging_loss=0.009819, over 3045625.69 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0 2023-11-21 13:17:03,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.54 vs. limit=15.0 2023-11-21 13:17:04,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1517153.3333333333, ans=0.1 2023-11-21 13:17:07,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1517153.3333333333, ans=0.0 2023-11-21 13:17:12,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1517153.3333333333, ans=0.125 2023-11-21 13:17:30,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1517286.6666666667, ans=0.125 2023-11-21 13:17:35,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227600 2023-11-21 13:17:42,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1517353.3333333333, ans=0.0 2023-11-21 13:17:42,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1517353.3333333333, ans=0.0 2023-11-21 13:17:58,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1517420.0, ans=0.0 2023-11-21 13:18:00,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1517420.0, ans=0.125 2023-11-21 13:18:07,667 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11200, loss[loss=0.08194, simple_loss=0.1138, pruned_loss=0.01517, audio_tagging_loss=0.009868, over 15339.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09619, pruned_loss=0.01637, audio_tagging_loss=0.00988, over 3038666.70 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:18:17,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1517486.6666666667, ans=0.5 2023-11-21 13:18:40,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227650 2023-11-21 13:18:47,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1517686.6666666667, ans=0.2 2023-11-21 13:18:48,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1517686.6666666667, ans=0.125 2023-11-21 13:18:51,697 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.451e+01 8.180e+01 8.891e+01 9.538e+01 1.267e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-21 13:19:00,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1517753.3333333333, ans=0.125 2023-11-21 13:19:12,942 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11250, loss[loss=0.06939, simple_loss=0.0842, pruned_loss=0.01529, audio_tagging_loss=0.012, over 16142.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09555, pruned_loss=0.01615, audio_tagging_loss=0.009916, over 3048275.68 frames. ], batch size: 60, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:19:25,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1517886.6666666667, ans=0.07 2023-11-21 13:19:27,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1517886.6666666667, ans=0.1 2023-11-21 13:19:32,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1517886.6666666667, ans=0.07 2023-11-21 13:19:40,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1517953.3333333333, ans=0.125 2023-11-21 13:19:42,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1517953.3333333333, ans=0.0 2023-11-21 13:19:45,497 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227700 2023-11-21 13:20:08,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.81 vs. limit=22.5 2023-11-21 13:20:18,082 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11300, loss[loss=0.06499, simple_loss=0.08569, pruned_loss=0.01187, audio_tagging_loss=0.01028, over 14833.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.0961, pruned_loss=0.01634, audio_tagging_loss=0.009734, over 3050905.91 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:20:22,040 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:20:48,942 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227750 2023-11-21 13:21:01,532 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.039e+01 8.792e+01 9.406e+01 1.983e+02, threshold=1.758e+02, percent-clipped=1.0 2023-11-21 13:21:02,204 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.58 vs. limit=10.0 2023-11-21 13:21:03,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.10 vs. limit=15.0 2023-11-21 13:21:05,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1518353.3333333333, ans=0.0 2023-11-21 13:21:06,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1518353.3333333333, ans=0.2 2023-11-21 13:21:21,395 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11350, loss[loss=0.06603, simple_loss=0.09788, pruned_loss=0.01012, audio_tagging_loss=0.006973, over 14811.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09532, pruned_loss=0.01623, audio_tagging_loss=0.009719, over 3048578.03 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:21:27,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1518486.6666666667, ans=0.1 2023-11-21 13:21:50,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1518620.0, ans=0.125 2023-11-21 13:21:54,322 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227800 2023-11-21 13:22:00,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1518686.6666666667, ans=0.1 2023-11-21 13:22:04,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.15 vs. limit=22.5 2023-11-21 13:22:22,600 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:22:26,029 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11400, loss[loss=0.06537, simple_loss=0.0814, pruned_loss=0.01566, audio_tagging_loss=0.009011, over 14913.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09604, pruned_loss=0.01644, audio_tagging_loss=0.009543, over 3046925.05 frames. ], batch size: 57, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:22:42,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1518886.6666666667, ans=0.2 2023-11-21 13:22:45,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.39 vs. limit=15.0 2023-11-21 13:22:58,231 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227850 2023-11-21 13:23:09,538 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.020e+01 8.128e+01 8.790e+01 9.462e+01 1.169e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 13:23:30,980 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11450, loss[loss=0.07895, simple_loss=0.09778, pruned_loss=0.02076, audio_tagging_loss=0.009295, over 14682.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.0965, pruned_loss=0.01649, audio_tagging_loss=0.009484, over 3045019.78 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:23:58,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1519286.6666666667, ans=0.125 2023-11-21 13:24:02,296 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227900 2023-11-21 13:24:06,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1519286.6666666667, ans=0.125 2023-11-21 13:24:24,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.09 vs. limit=15.0 2023-11-21 13:24:29,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1519420.0, ans=0.0 2023-11-21 13:24:31,974 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0 2023-11-21 13:24:34,773 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11500, loss[loss=0.0736, simple_loss=0.09515, pruned_loss=0.0178, audio_tagging_loss=0.008227, over 14847.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09656, pruned_loss=0.01675, audio_tagging_loss=0.009479, over 3048817.62 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 32.0 2023-11-21 13:24:39,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1519486.6666666667, ans=0.0 2023-11-21 13:24:49,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1519553.3333333333, ans=0.04949747468305833 2023-11-21 13:24:53,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1519553.3333333333, ans=0.0 2023-11-21 13:25:01,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1519620.0, ans=0.0 2023-11-21 13:25:07,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 227950 2023-11-21 13:25:18,647 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.513e+01 8.149e+01 8.832e+01 9.671e+01 1.256e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-21 13:25:32,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0 2023-11-21 13:25:38,554 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11550, loss[loss=0.07356, simple_loss=0.09311, pruned_loss=0.01917, audio_tagging_loss=0.00783, over 16246.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.0971, pruned_loss=0.0169, audio_tagging_loss=0.0094, over 3048787.03 frames. ], batch size: 63, lr: 3.56e-03, grad_scale: 16.0 2023-11-21 13:25:44,903 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.04 vs. limit=15.0 2023-11-21 13:25:59,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1519886.6666666667, ans=0.1 2023-11-21 13:26:05,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1519953.3333333333, ans=0.2 2023-11-21 13:26:06,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1519953.3333333333, ans=0.125 2023-11-21 13:26:10,878 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228000 2023-11-21 13:26:11,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.23 vs. limit=10.0 2023-11-21 13:26:20,422 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 13:26:21,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.50 vs. limit=22.5 2023-11-21 13:26:39,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1520086.6666666667, ans=0.125 2023-11-21 13:26:47,344 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11600, loss[loss=0.07826, simple_loss=0.09666, pruned_loss=0.02039, audio_tagging_loss=0.009538, over 15053.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09654, pruned_loss=0.01662, audio_tagging_loss=0.009409, over 3047192.81 frames. ], batch size: 57, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:26:58,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1520220.0, ans=0.125 2023-11-21 13:27:09,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1520220.0, ans=0.125 2023-11-21 13:27:18,881 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228050 2023-11-21 13:27:20,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1520286.6666666667, ans=0.125 2023-11-21 13:27:23,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1520353.3333333333, ans=0.0 2023-11-21 13:27:26,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.53 vs. limit=15.0 2023-11-21 13:27:28,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1520353.3333333333, ans=0.2 2023-11-21 13:27:32,534 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.674e+01 8.168e+01 8.737e+01 9.633e+01 1.247e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 13:27:40,711 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:27:51,768 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11650, loss[loss=0.08778, simple_loss=0.1109, pruned_loss=0.02048, audio_tagging_loss=0.01185, over 14785.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09601, pruned_loss=0.01639, audio_tagging_loss=0.009476, over 3043875.13 frames. ], batch size: 55, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:27:59,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1520486.6666666667, ans=0.2 2023-11-21 13:28:02,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1520486.6666666667, ans=0.125 2023-11-21 13:28:15,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1520553.3333333333, ans=0.0 2023-11-21 13:28:20,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1520620.0, ans=0.125 2023-11-21 13:28:24,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228100 2023-11-21 13:28:30,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1520686.6666666667, ans=0.1 2023-11-21 13:28:33,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1520686.6666666667, ans=0.125 2023-11-21 13:28:36,175 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.46 vs. limit=6.0 2023-11-21 13:28:39,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1520686.6666666667, ans=0.09899494936611666 2023-11-21 13:28:54,837 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11700, loss[loss=0.07596, simple_loss=0.09598, pruned_loss=0.01923, audio_tagging_loss=0.008744, over 15282.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09658, pruned_loss=0.01667, audio_tagging_loss=0.009482, over 3044957.92 frames. ], batch size: 54, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:28:56,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1520820.0, ans=0.1 2023-11-21 13:29:19,606 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.89 vs. limit=15.0 2023-11-21 13:29:27,536 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228150 2023-11-21 13:29:30,778 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.39 vs. limit=10.0 2023-11-21 13:29:37,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1521020.0, ans=0.1 2023-11-21 13:29:39,518 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.045e+01 8.764e+01 9.289e+01 1.171e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-21 13:29:58,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1521153.3333333333, ans=0.125 2023-11-21 13:29:59,679 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11750, loss[loss=0.06245, simple_loss=0.07574, pruned_loss=0.01419, audio_tagging_loss=0.01039, over 14798.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09573, pruned_loss=0.01666, audio_tagging_loss=0.009579, over 3046973.95 frames. ], batch size: 56, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:30:01,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1521153.3333333333, ans=0.0 2023-11-21 13:30:15,561 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.85 vs. limit=15.0 2023-11-21 13:30:22,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0 2023-11-21 13:30:23,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1521286.6666666667, ans=0.125 2023-11-21 13:30:26,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1521286.6666666667, ans=0.0 2023-11-21 13:30:30,700 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228200 2023-11-21 13:30:33,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1521286.6666666667, ans=0.125 2023-11-21 13:30:46,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1521353.3333333333, ans=0.1 2023-11-21 13:31:03,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1521486.6666666667, ans=0.1 2023-11-21 13:31:03,811 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11800, loss[loss=0.08652, simple_loss=0.1149, pruned_loss=0.01921, audio_tagging_loss=0.009853, over 15084.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09559, pruned_loss=0.01666, audio_tagging_loss=0.009647, over 3044581.91 frames. ], batch size: 57, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:31:05,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1521486.6666666667, ans=0.125 2023-11-21 13:31:08,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.29 vs. limit=22.5 2023-11-21 13:31:09,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1521486.6666666667, ans=0.0 2023-11-21 13:31:10,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1521486.6666666667, ans=0.125 2023-11-21 13:31:29,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.09 vs. limit=12.0 2023-11-21 13:31:35,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228250 2023-11-21 13:31:40,568 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.45 vs. limit=15.0 2023-11-21 13:31:46,937 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.55 vs. limit=15.0 2023-11-21 13:31:48,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.292e+01 8.361e+01 9.006e+01 1.004e+02 1.264e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-21 13:32:03,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1521753.3333333333, ans=0.0 2023-11-21 13:32:03,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1521753.3333333333, ans=0.125 2023-11-21 13:32:07,052 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11850, loss[loss=0.06922, simple_loss=0.08365, pruned_loss=0.01363, audio_tagging_loss=0.01376, over 14381.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09603, pruned_loss=0.0169, audio_tagging_loss=0.009645, over 3042767.86 frames. ], batch size: 54, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:32:19,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1521886.6666666667, ans=0.0 2023-11-21 13:32:32,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.98 vs. limit=15.0 2023-11-21 13:32:34,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.97 vs. limit=22.5 2023-11-21 13:32:40,341 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228300 2023-11-21 13:33:11,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1522153.3333333333, ans=0.125 2023-11-21 13:33:11,966 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11900, loss[loss=0.07792, simple_loss=0.108, pruned_loss=0.01536, audio_tagging_loss=0.008558, over 15702.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09547, pruned_loss=0.01675, audio_tagging_loss=0.009719, over 3037229.91 frames. ], batch size: 57, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:33:25,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1522220.0, ans=0.0 2023-11-21 13:33:27,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1522220.0, ans=0.0 2023-11-21 13:33:36,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1522286.6666666667, ans=0.125 2023-11-21 13:33:42,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1522286.6666666667, ans=0.1 2023-11-21 13:33:43,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228350 2023-11-21 13:33:55,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1522353.3333333333, ans=0.5 2023-11-21 13:33:56,596 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.221e+01 8.725e+01 9.501e+01 1.211e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-21 13:34:16,699 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 11950, loss[loss=0.07778, simple_loss=0.1019, pruned_loss=0.0197, audio_tagging_loss=0.007137, over 14507.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.09654, pruned_loss=0.01694, audio_tagging_loss=0.009737, over 3041259.12 frames. ], batch size: 57, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:34:19,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1522486.6666666667, ans=0.125 2023-11-21 13:34:30,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1522553.3333333333, ans=0.125 2023-11-21 13:34:35,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1522553.3333333333, ans=0.2 2023-11-21 13:34:47,875 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228400 2023-11-21 13:34:55,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1522686.6666666667, ans=0.0 2023-11-21 13:35:18,689 INFO [train_asr.py:1221] (3/4) Epoch 19, batch 12000, loss[loss=0.07922, simple_loss=0.09596, pruned_loss=0.02014, audio_tagging_loss=0.0111, over 15073.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09594, pruned_loss=0.0168, audio_tagging_loss=0.009811, over 3042781.42 frames. ], batch size: 57, lr: 3.56e-03, grad_scale: 32.0 2023-11-21 13:35:18,690 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 13:35:41,315 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([3.6042, 3.3371, 2.9150, 3.3169], device='cuda:3') 2023-11-21 13:36:02,007 INFO [train_asr.py:1253] (3/4) Epoch 19, validation: loss=0.06018, simple_loss=0.05227, pruned_loss=0.005307, audio_tagging_loss=0.02874, over 4681554.00 frames. 2023-11-21 13:36:02,008 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 13:36:06,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1522820.0, ans=0.125 2023-11-21 13:36:19,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1522886.6666666667, ans=0.0 2023-11-21 13:36:24,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1522953.3333333333, ans=0.125 2023-11-21 13:36:28,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1522953.3333333333, ans=0.125 2023-11-21 13:37:05,066 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 0, loss[loss=0.07278, simple_loss=0.06243, pruned_loss=0.01285, audio_tagging_loss=0.02872, over 14021.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.06243, pruned_loss=0.01285, audio_tagging_loss=0.02872, over 14021.00 frames. ], batch size: 56, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:37:05,068 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 13:37:33,927 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.5928, 3.4227, 3.8357, 3.2549], device='cuda:3') 2023-11-21 13:37:41,799 INFO [train_asr.py:1253] (3/4) Epoch 20, validation: loss=0.05938, simple_loss=0.0523, pruned_loss=0.005287, audio_tagging_loss=0.02794, over 4681554.00 frames. 2023-11-21 13:37:41,800 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 13:37:43,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228450 2023-11-21 13:37:43,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1522980.0, ans=0.1 2023-11-21 13:37:55,102 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.213e+01 9.034e+01 9.702e+01 1.198e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-21 13:38:02,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.62 vs. limit=15.0 2023-11-21 13:38:13,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1523113.3333333333, ans=0.125 2023-11-21 13:38:20,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1523180.0, ans=0.1 2023-11-21 13:38:21,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1523180.0, ans=0.125 2023-11-21 13:38:24,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1523180.0, ans=0.05 2023-11-21 13:38:25,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1523180.0, ans=0.125 2023-11-21 13:38:39,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1523246.6666666667, ans=0.1 2023-11-21 13:38:44,848 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 50, loss[loss=0.07716, simple_loss=0.08307, pruned_loss=0.01657, audio_tagging_loss=0.01905, over 14355.00 frames. ], tot_loss[loss=0.08315, simple_loss=0.09487, pruned_loss=0.01673, audio_tagging_loss=0.01899, over 684060.77 frames. ], batch size: 56, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:38:46,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228500 2023-11-21 13:38:52,994 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.35 vs. limit=15.0 2023-11-21 13:39:19,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1523446.6666666667, ans=0.1 2023-11-21 13:39:22,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1523446.6666666667, ans=0.1 2023-11-21 13:39:27,405 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.43 vs. limit=15.0 2023-11-21 13:39:49,072 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 100, loss[loss=0.0772, simple_loss=0.09192, pruned_loss=0.01511, audio_tagging_loss=0.01613, over 14785.00 frames. ], tot_loss[loss=0.08398, simple_loss=0.09796, pruned_loss=0.01715, audio_tagging_loss=0.01785, over 1211332.43 frames. ], batch size: 56, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:39:50,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228550 2023-11-21 13:39:50,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1523646.6666666667, ans=0.0 2023-11-21 13:40:01,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1523713.3333333333, ans=0.125 2023-11-21 13:40:04,257 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.140e+01 8.678e+01 9.460e+01 1.029e+02 1.398e+02, threshold=1.892e+02, percent-clipped=0.0 2023-11-21 13:40:13,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1523713.3333333333, ans=0.0 2023-11-21 13:40:23,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1523780.0, ans=0.1 2023-11-21 13:40:25,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1523780.0, ans=0.125 2023-11-21 13:40:35,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.95 vs. limit=12.0 2023-11-21 13:40:41,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1523913.3333333333, ans=0.125 2023-11-21 13:40:44,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1523913.3333333333, ans=0.125 2023-11-21 13:40:54,004 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 150, loss[loss=0.1172, simple_loss=0.143, pruned_loss=0.03766, audio_tagging_loss=0.008006, over 14532.00 frames. ], tot_loss[loss=0.08214, simple_loss=0.09818, pruned_loss=0.01697, audio_tagging_loss=0.01607, over 1616512.64 frames. ], batch size: 54, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:40:56,003 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228600 2023-11-21 13:41:18,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1524113.3333333333, ans=0.125 2023-11-21 13:41:58,550 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 200, loss[loss=0.08484, simple_loss=0.09741, pruned_loss=0.02391, audio_tagging_loss=0.01223, over 15383.00 frames. ], tot_loss[loss=0.07979, simple_loss=0.09778, pruned_loss=0.01675, audio_tagging_loss=0.01415, over 1934873.47 frames. ], batch size: 57, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:41:59,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228650 2023-11-21 13:42:12,011 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.304e+01 8.770e+01 9.399e+01 1.399e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 13:42:23,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1524446.6666666667, ans=0.07 2023-11-21 13:42:25,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1524446.6666666667, ans=0.2 2023-11-21 13:42:29,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1524446.6666666667, ans=0.0 2023-11-21 13:42:39,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1524513.3333333333, ans=0.125 2023-11-21 13:42:41,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1524513.3333333333, ans=0.125 2023-11-21 13:43:02,037 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 250, loss[loss=0.0707, simple_loss=0.08961, pruned_loss=0.01616, audio_tagging_loss=0.009734, over 15860.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09654, pruned_loss=0.01656, audio_tagging_loss=0.01289, over 2183392.58 frames. ], batch size: 60, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:43:03,942 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228700 2023-11-21 13:43:22,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1524713.3333333333, ans=0.1 2023-11-21 13:43:27,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1524780.0, ans=0.0 2023-11-21 13:43:53,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1524913.3333333333, ans=0.0 2023-11-21 13:43:58,984 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:44:07,313 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 300, loss[loss=0.06983, simple_loss=0.08838, pruned_loss=0.01391, audio_tagging_loss=0.01173, over 15605.00 frames. ], tot_loss[loss=0.07846, simple_loss=0.09899, pruned_loss=0.0171, audio_tagging_loss=0.01187, over 2384546.30 frames. ], batch size: 58, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:44:08,647 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228750 2023-11-21 13:44:12,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.96 vs. limit=6.0 2023-11-21 13:44:15,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1524980.0, ans=0.1 2023-11-21 13:44:18,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.45 vs. limit=12.0 2023-11-21 13:44:21,329 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.166e+01 8.999e+01 9.797e+01 1.140e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-21 13:45:08,969 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.68 vs. limit=5.0 2023-11-21 13:45:11,670 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 350, loss[loss=0.06949, simple_loss=0.08795, pruned_loss=0.0177, audio_tagging_loss=0.007822, over 15407.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.09836, pruned_loss=0.01685, audio_tagging_loss=0.0112, over 2536091.39 frames. ], batch size: 58, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:45:12,922 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228800 2023-11-21 13:45:25,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1525380.0, ans=0.0 2023-11-21 13:45:37,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1525446.6666666667, ans=0.2 2023-11-21 13:45:42,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1525446.6666666667, ans=0.125 2023-11-21 13:45:53,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1525513.3333333333, ans=0.5 2023-11-21 13:46:03,631 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:46:04,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1525580.0, ans=0.1 2023-11-21 13:46:15,878 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 400, loss[loss=0.07388, simple_loss=0.1038, pruned_loss=0.01346, audio_tagging_loss=0.00853, over 15549.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09796, pruned_loss=0.01679, audio_tagging_loss=0.01079, over 2648925.46 frames. ], batch size: 57, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:46:17,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228850 2023-11-21 13:46:24,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.05 vs. limit=22.5 2023-11-21 13:46:26,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1525646.6666666667, ans=0.95 2023-11-21 13:46:30,455 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.562e+01 8.083e+01 8.649e+01 9.202e+01 1.163e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-21 13:46:35,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1525713.3333333333, ans=0.0 2023-11-21 13:46:47,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.96 vs. limit=15.0 2023-11-21 13:46:50,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1525780.0, ans=0.0 2023-11-21 13:47:15,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1525913.3333333333, ans=0.0 2023-11-21 13:47:16,186 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.91 vs. limit=15.0 2023-11-21 13:47:20,802 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 450, loss[loss=0.06548, simple_loss=0.0794, pruned_loss=0.01454, audio_tagging_loss=0.01123, over 15347.00 frames. ], tot_loss[loss=0.07497, simple_loss=0.096, pruned_loss=0.01646, audio_tagging_loss=0.0105, over 2734789.11 frames. ], batch size: 59, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:47:22,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228900 2023-11-21 13:47:50,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1526113.3333333333, ans=0.0 2023-11-21 13:48:04,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=22.5 2023-11-21 13:48:24,762 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 500, loss[loss=0.08421, simple_loss=0.1159, pruned_loss=0.01992, audio_tagging_loss=0.006324, over 15162.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09692, pruned_loss=0.01662, audio_tagging_loss=0.01024, over 2795928.08 frames. ], batch size: 56, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:48:26,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 228950 2023-11-21 13:48:38,759 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.882e+01 7.917e+01 8.636e+01 9.467e+01 1.294e+02, threshold=1.727e+02, percent-clipped=0.0 2023-11-21 13:48:45,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.51 vs. limit=10.0 2023-11-21 13:48:50,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1526446.6666666667, ans=0.07 2023-11-21 13:48:53,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1526446.6666666667, ans=0.125 2023-11-21 13:49:28,873 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 550, loss[loss=0.07483, simple_loss=0.09836, pruned_loss=0.01421, audio_tagging_loss=0.01144, over 15727.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09567, pruned_loss=0.01644, audio_tagging_loss=0.01014, over 2852679.91 frames. ], batch size: 58, lr: 3.47e-03, grad_scale: 32.0 2023-11-21 13:49:30,149 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229000 2023-11-21 13:49:33,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1526646.6666666667, ans=0.1 2023-11-21 13:49:36,596 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0 2023-11-21 13:50:10,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1526846.6666666667, ans=0.2 2023-11-21 13:50:30,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1526913.3333333333, ans=0.2 2023-11-21 13:50:33,696 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 600, loss[loss=0.07176, simple_loss=0.09179, pruned_loss=0.01708, audio_tagging_loss=0.008787, over 15767.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09497, pruned_loss=0.01646, audio_tagging_loss=0.01011, over 2888788.61 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 13:50:35,089 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229050 2023-11-21 13:50:39,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1526980.0, ans=0.0 2023-11-21 13:50:47,731 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.121e+01 7.930e+01 8.613e+01 9.327e+01 1.199e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 13:50:53,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1527046.6666666667, ans=0.125 2023-11-21 13:50:54,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1527046.6666666667, ans=0.125 2023-11-21 13:51:07,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1527113.3333333333, ans=0.0 2023-11-21 13:51:14,319 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.19 vs. limit=15.0 2023-11-21 13:51:19,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1527180.0, ans=0.1 2023-11-21 13:51:22,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1527180.0, ans=0.1 2023-11-21 13:51:38,019 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 650, loss[loss=0.07261, simple_loss=0.09342, pruned_loss=0.01711, audio_tagging_loss=0.008789, over 15510.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09543, pruned_loss=0.01671, audio_tagging_loss=0.01002, over 2930288.43 frames. ], batch size: 64, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 13:51:38,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.13 vs. limit=10.0 2023-11-21 13:51:39,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229100 2023-11-21 13:52:14,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1527446.6666666667, ans=0.0 2023-11-21 13:52:41,375 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 700, loss[loss=0.0697, simple_loss=0.09962, pruned_loss=0.01282, audio_tagging_loss=0.007073, over 14160.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09504, pruned_loss=0.01638, audio_tagging_loss=0.01002, over 2959753.79 frames. ], batch size: 53, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 13:52:43,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229150 2023-11-21 13:52:54,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1527713.3333333333, ans=0.125 2023-11-21 13:52:56,214 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.031e+01 8.760e+01 9.624e+01 1.233e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-21 13:53:17,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1527780.0, ans=0.0 2023-11-21 13:53:22,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1527846.6666666667, ans=0.125 2023-11-21 13:53:23,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1527846.6666666667, ans=0.0 2023-11-21 13:53:30,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1527846.6666666667, ans=0.125 2023-11-21 13:53:45,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1527913.3333333333, ans=0.04949747468305833 2023-11-21 13:53:47,392 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 750, loss[loss=0.07355, simple_loss=0.09421, pruned_loss=0.01752, audio_tagging_loss=0.008919, over 14661.00 frames. ], tot_loss[loss=0.07372, simple_loss=0.09498, pruned_loss=0.01628, audio_tagging_loss=0.009946, over 2978294.74 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 13:53:48,684 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229200 2023-11-21 13:53:48,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1527980.0, ans=0.0 2023-11-21 13:53:48,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1527980.0, ans=0.125 2023-11-21 13:53:52,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1527980.0, ans=15.0 2023-11-21 13:54:00,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1528046.6666666667, ans=0.125 2023-11-21 13:54:06,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1528046.6666666667, ans=0.125 2023-11-21 13:54:17,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1528113.3333333333, ans=0.1 2023-11-21 13:54:20,017 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:54:21,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.92 vs. limit=15.0 2023-11-21 13:54:33,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1528180.0, ans=0.0 2023-11-21 13:54:33,979 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.37 vs. limit=10.0 2023-11-21 13:54:48,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1528246.6666666667, ans=0.0 2023-11-21 13:54:48,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1528246.6666666667, ans=0.1 2023-11-21 13:54:52,690 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 800, loss[loss=0.06704, simple_loss=0.08589, pruned_loss=0.01072, audio_tagging_loss=0.01337, over 15165.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09575, pruned_loss=0.01648, audio_tagging_loss=0.009901, over 2992597.11 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 13:54:54,015 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229250 2023-11-21 13:55:06,305 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.111e+01 8.814e+01 9.591e+01 1.302e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-21 13:55:21,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=1528446.6666666667, ans=0.02 2023-11-21 13:55:44,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1528580.0, ans=0.125 2023-11-21 13:55:44,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1528580.0, ans=0.125 2023-11-21 13:55:55,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1528580.0, ans=0.0 2023-11-21 13:55:57,283 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 850, loss[loss=0.07127, simple_loss=0.08836, pruned_loss=0.0149, audio_tagging_loss=0.01219, over 14921.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09595, pruned_loss=0.01648, audio_tagging_loss=0.009904, over 3006588.36 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 13:55:58,572 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229300 2023-11-21 13:56:06,602 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=12.0 2023-11-21 13:56:12,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1528713.3333333333, ans=0.0 2023-11-21 13:57:00,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1528913.3333333333, ans=0.125 2023-11-21 13:57:02,840 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 900, loss[loss=0.08887, simple_loss=0.1168, pruned_loss=0.02307, audio_tagging_loss=0.007397, over 14690.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09663, pruned_loss=0.01668, audio_tagging_loss=0.009866, over 3017059.56 frames. ], batch size: 52, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 13:57:04,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229350 2023-11-21 13:57:08,522 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=12.0 2023-11-21 13:57:13,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1528980.0, ans=0.0 2023-11-21 13:57:19,059 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.780e+01 8.022e+01 8.539e+01 9.514e+01 1.359e+02, threshold=1.708e+02, percent-clipped=0.0 2023-11-21 13:57:23,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1529046.6666666667, ans=0.2 2023-11-21 13:57:44,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1529180.0, ans=0.0 2023-11-21 13:57:56,822 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 13:57:58,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1529246.6666666667, ans=0.07 2023-11-21 13:58:08,496 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 950, loss[loss=0.0697, simple_loss=0.09707, pruned_loss=0.01482, audio_tagging_loss=0.006343, over 14799.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09678, pruned_loss=0.01662, audio_tagging_loss=0.009731, over 3022278.28 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 13:58:09,952 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229400 2023-11-21 13:58:16,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1529313.3333333333, ans=0.125 2023-11-21 13:58:23,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.74 vs. limit=12.0 2023-11-21 13:58:27,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1529380.0, ans=0.125 2023-11-21 13:58:51,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1529513.3333333333, ans=0.0 2023-11-21 13:58:54,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1529513.3333333333, ans=0.2 2023-11-21 13:59:12,410 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1000, loss[loss=0.07684, simple_loss=0.0994, pruned_loss=0.01554, audio_tagging_loss=0.01159, over 13981.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.09671, pruned_loss=0.01651, audio_tagging_loss=0.009653, over 3025362.55 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 13:59:13,741 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229450 2023-11-21 13:59:15,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1529646.6666666667, ans=0.125 2023-11-21 13:59:20,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.28 vs. limit=15.0 2023-11-21 13:59:27,765 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.761e+01 8.099e+01 8.790e+01 9.899e+01 1.209e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 13:59:34,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1529713.3333333333, ans=0.125 2023-11-21 13:59:41,900 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:00:16,823 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1050, loss[loss=0.09408, simple_loss=0.1221, pruned_loss=0.02327, audio_tagging_loss=0.009766, over 15982.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.0964, pruned_loss=0.0166, audio_tagging_loss=0.009518, over 3028824.12 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 14:00:18,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229500 2023-11-21 14:00:32,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1530046.6666666667, ans=0.125 2023-11-21 14:00:36,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.26 vs. limit=12.0 2023-11-21 14:00:43,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1530113.3333333333, ans=0.125 2023-11-21 14:00:47,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1530113.3333333333, ans=0.125 2023-11-21 14:01:09,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1530246.6666666667, ans=0.2 2023-11-21 14:01:23,282 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1100, loss[loss=0.08624, simple_loss=0.106, pruned_loss=0.02219, audio_tagging_loss=0.01106, over 14552.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09627, pruned_loss=0.0167, audio_tagging_loss=0.009528, over 3029011.69 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 14:01:24,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229550 2023-11-21 14:01:26,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=15.0 2023-11-21 14:01:27,044 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:01:34,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1530380.0, ans=0.125 2023-11-21 14:01:38,190 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.405e+01 8.211e+01 9.088e+01 9.888e+01 2.167e+02, threshold=1.818e+02, percent-clipped=1.0 2023-11-21 14:02:28,035 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1150, loss[loss=0.05562, simple_loss=0.06992, pruned_loss=0.01019, audio_tagging_loss=0.01046, over 13424.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09635, pruned_loss=0.01662, audio_tagging_loss=0.009503, over 3036138.81 frames. ], batch size: 53, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 14:02:29,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229600 2023-11-21 14:02:36,993 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 14:02:37,451 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.36 vs. limit=15.0 2023-11-21 14:03:08,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1530846.6666666667, ans=0.125 2023-11-21 14:03:15,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.85 vs. limit=22.5 2023-11-21 14:03:16,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1530846.6666666667, ans=0.0 2023-11-21 14:03:31,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1530980.0, ans=0.2 2023-11-21 14:03:32,058 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1200, loss[loss=0.08551, simple_loss=0.11, pruned_loss=0.01945, audio_tagging_loss=0.01108, over 14896.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09707, pruned_loss=0.01672, audio_tagging_loss=0.009414, over 3043647.12 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:03:33,341 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229650 2023-11-21 14:03:41,923 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.01 vs. limit=15.0 2023-11-21 14:03:42,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1530980.0, ans=0.125 2023-11-21 14:03:48,979 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.148e+01 8.693e+01 9.555e+01 1.704e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 14:03:51,071 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.42 vs. limit=22.5 2023-11-21 14:04:05,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1531113.3333333333, ans=0.0 2023-11-21 14:04:38,652 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1250, loss[loss=0.07313, simple_loss=0.1033, pruned_loss=0.0137, audio_tagging_loss=0.007784, over 16582.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09748, pruned_loss=0.01659, audio_tagging_loss=0.009331, over 3045587.91 frames. ], batch size: 63, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:04:40,025 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229700 2023-11-21 14:04:47,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1531313.3333333333, ans=0.125 2023-11-21 14:04:56,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1531380.0, ans=0.04949747468305833 2023-11-21 14:05:07,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.42 vs. limit=15.0 2023-11-21 14:05:39,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1531580.0, ans=0.0 2023-11-21 14:05:43,346 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1300, loss[loss=0.07582, simple_loss=0.09893, pruned_loss=0.01883, audio_tagging_loss=0.007524, over 15093.00 frames. ], tot_loss[loss=0.07442, simple_loss=0.09697, pruned_loss=0.01659, audio_tagging_loss=0.009347, over 3039561.76 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:05:44,647 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229750 2023-11-21 14:05:48,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1531646.6666666667, ans=0.0 2023-11-21 14:05:58,189 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.180e+01 8.668e+01 9.567e+01 1.145e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-21 14:06:09,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1531780.0, ans=0.5 2023-11-21 14:06:11,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1531780.0, ans=0.125 2023-11-21 14:06:18,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1531780.0, ans=0.09899494936611666 2023-11-21 14:06:18,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1531780.0, ans=0.0 2023-11-21 14:06:24,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1531846.6666666667, ans=0.0 2023-11-21 14:06:38,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1531913.3333333333, ans=0.0 2023-11-21 14:06:48,133 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1350, loss[loss=0.06684, simple_loss=0.07928, pruned_loss=0.01389, audio_tagging_loss=0.01332, over 14034.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.09676, pruned_loss=0.0167, audio_tagging_loss=0.009449, over 3037547.31 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:06:49,474 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229800 2023-11-21 14:07:05,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1532046.6666666667, ans=0.0 2023-11-21 14:07:05,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.32 vs. limit=10.0 2023-11-21 14:07:06,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1532046.6666666667, ans=0.125 2023-11-21 14:07:34,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1532180.0, ans=0.0 2023-11-21 14:07:35,733 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:07:39,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0 2023-11-21 14:07:40,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1532246.6666666667, ans=0.125 2023-11-21 14:07:55,134 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1400, loss[loss=0.06284, simple_loss=0.07931, pruned_loss=0.01368, audio_tagging_loss=0.009502, over 15385.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09672, pruned_loss=0.01678, audio_tagging_loss=0.009482, over 3045593.79 frames. ], batch size: 58, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:07:56,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229850 2023-11-21 14:08:10,532 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.288e+01 8.939e+01 9.468e+01 2.052e+02, threshold=1.788e+02, percent-clipped=1.0 2023-11-21 14:08:23,519 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=12.0 2023-11-21 14:08:24,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1532446.6666666667, ans=0.0 2023-11-21 14:08:26,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1532446.6666666667, ans=0.125 2023-11-21 14:08:53,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.64 vs. limit=15.0 2023-11-21 14:08:59,481 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1450, loss[loss=0.08331, simple_loss=0.106, pruned_loss=0.01882, audio_tagging_loss=0.01147, over 13965.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.0957, pruned_loss=0.01656, audio_tagging_loss=0.00959, over 3040765.65 frames. ], batch size: 53, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:09:00,863 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229900 2023-11-21 14:09:09,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.whiten.whitening_limit, batch_count=1532646.6666666667, ans=12.0 2023-11-21 14:09:30,769 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.28 vs. limit=5.0 2023-11-21 14:09:34,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1532780.0, ans=0.125 2023-11-21 14:09:35,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1532780.0, ans=0.125 2023-11-21 14:09:36,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1532780.0, ans=0.0 2023-11-21 14:09:36,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1532780.0, ans=0.125 2023-11-21 14:09:40,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1532846.6666666667, ans=0.0 2023-11-21 14:09:44,657 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0 2023-11-21 14:09:51,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1532913.3333333333, ans=0.125 2023-11-21 14:09:51,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=22.5 2023-11-21 14:09:57,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1532913.3333333333, ans=0.0 2023-11-21 14:10:01,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1532913.3333333333, ans=0.125 2023-11-21 14:10:03,586 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1500, loss[loss=0.081, simple_loss=0.105, pruned_loss=0.01778, audio_tagging_loss=0.01071, over 16211.00 frames. ], tot_loss[loss=0.07429, simple_loss=0.09582, pruned_loss=0.01664, audio_tagging_loss=0.009745, over 3041137.49 frames. ], batch size: 62, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:10:03,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1532980.0, ans=0.125 2023-11-21 14:10:04,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 229950 2023-11-21 14:10:18,863 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.143e+01 8.696e+01 9.361e+01 1.676e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 14:10:24,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1533046.6666666667, ans=0.0 2023-11-21 14:11:08,124 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1550, loss[loss=0.06695, simple_loss=0.08895, pruned_loss=0.0116, audio_tagging_loss=0.01087, over 14544.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09577, pruned_loss=0.01679, audio_tagging_loss=0.009821, over 3043443.90 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:11:10,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230000 2023-11-21 14:11:38,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1533446.6666666667, ans=0.2 2023-11-21 14:12:14,365 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1600, loss[loss=0.09519, simple_loss=0.1131, pruned_loss=0.02603, audio_tagging_loss=0.01259, over 14804.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09602, pruned_loss=0.01671, audio_tagging_loss=0.009736, over 3049779.41 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:12:16,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230050 2023-11-21 14:12:26,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1533713.3333333333, ans=10.0 2023-11-21 14:12:30,018 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.321e+01 8.893e+01 9.688e+01 1.461e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-21 14:12:35,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1533713.3333333333, ans=0.125 2023-11-21 14:12:36,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1533713.3333333333, ans=0.0 2023-11-21 14:12:36,824 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.01 vs. limit=10.0 2023-11-21 14:12:42,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1533780.0, ans=0.125 2023-11-21 14:12:45,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1533780.0, ans=0.125 2023-11-21 14:12:50,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1533780.0, ans=0.09899494936611666 2023-11-21 14:13:05,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1533913.3333333333, ans=0.125 2023-11-21 14:13:14,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1533913.3333333333, ans=0.0 2023-11-21 14:13:19,684 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1650, loss[loss=0.07854, simple_loss=0.1014, pruned_loss=0.0185, audio_tagging_loss=0.009331, over 16311.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09585, pruned_loss=0.01669, audio_tagging_loss=0.009771, over 3046484.63 frames. ], batch size: 61, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:13:21,096 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230100 2023-11-21 14:13:24,190 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.97 vs. limit=22.5 2023-11-21 14:13:26,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1533980.0, ans=0.125 2023-11-21 14:13:28,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1533980.0, ans=0.125 2023-11-21 14:13:35,274 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 14:13:42,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.57 vs. limit=22.5 2023-11-21 14:14:24,715 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1700, loss[loss=0.07979, simple_loss=0.0997, pruned_loss=0.01852, audio_tagging_loss=0.01143, over 16606.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.09461, pruned_loss=0.01632, audio_tagging_loss=0.009812, over 3041824.87 frames. ], batch size: 63, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:14:26,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230150 2023-11-21 14:14:37,645 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.63 vs. limit=15.0 2023-11-21 14:14:40,778 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.297e+01 8.260e+01 8.837e+01 9.595e+01 1.453e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 14:14:43,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1534380.0, ans=0.0 2023-11-21 14:14:46,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1534380.0, ans=0.2 2023-11-21 14:14:51,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1534446.6666666667, ans=0.0 2023-11-21 14:15:01,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1534446.6666666667, ans=6.0 2023-11-21 14:15:06,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1534513.3333333333, ans=0.035 2023-11-21 14:15:08,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1534513.3333333333, ans=0.0 2023-11-21 14:15:18,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1534580.0, ans=0.1 2023-11-21 14:15:24,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1534580.0, ans=10.0 2023-11-21 14:15:30,388 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1750, loss[loss=0.05578, simple_loss=0.06923, pruned_loss=0.009981, audio_tagging_loss=0.01119, over 16450.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09464, pruned_loss=0.01623, audio_tagging_loss=0.009731, over 3044795.68 frames. ], batch size: 62, lr: 3.46e-03, grad_scale: 32.0 2023-11-21 14:15:31,710 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230200 2023-11-21 14:15:34,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1534646.6666666667, ans=0.0 2023-11-21 14:15:39,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1534646.6666666667, ans=0.1 2023-11-21 14:15:42,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1534713.3333333333, ans=0.125 2023-11-21 14:15:54,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1534780.0, ans=0.09899494936611666 2023-11-21 14:15:58,311 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.22 vs. limit=15.0 2023-11-21 14:16:09,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1534846.6666666667, ans=0.125 2023-11-21 14:16:13,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.60 vs. limit=22.5 2023-11-21 14:16:29,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1534913.3333333333, ans=0.125 2023-11-21 14:16:34,751 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1800, loss[loss=0.07482, simple_loss=0.1037, pruned_loss=0.01348, audio_tagging_loss=0.009468, over 15621.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09535, pruned_loss=0.0163, audio_tagging_loss=0.009595, over 3051077.16 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 14:16:36,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230250 2023-11-21 14:16:36,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.29 vs. limit=6.0 2023-11-21 14:16:51,336 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.097e+01 8.882e+01 9.608e+01 1.325e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-21 14:17:00,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1535113.3333333333, ans=0.0 2023-11-21 14:17:00,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1535113.3333333333, ans=0.0 2023-11-21 14:17:13,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1535180.0, ans=0.2 2023-11-21 14:17:16,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.35 vs. limit=12.0 2023-11-21 14:17:39,502 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1850, loss[loss=0.06961, simple_loss=0.09212, pruned_loss=0.01484, audio_tagging_loss=0.008715, over 15565.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09552, pruned_loss=0.01635, audio_tagging_loss=0.009499, over 3045452.62 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 14:17:39,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1535313.3333333333, ans=0.2 2023-11-21 14:17:40,792 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230300 2023-11-21 14:17:43,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1535313.3333333333, ans=0.125 2023-11-21 14:17:48,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1535313.3333333333, ans=0.2 2023-11-21 14:17:53,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1535380.0, ans=0.0 2023-11-21 14:17:53,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1535380.0, ans=0.0 2023-11-21 14:17:54,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1535380.0, ans=0.0 2023-11-21 14:18:07,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1535446.6666666667, ans=0.1 2023-11-21 14:18:29,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1535513.3333333333, ans=0.1 2023-11-21 14:18:45,261 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1900, loss[loss=0.08084, simple_loss=0.105, pruned_loss=0.01742, audio_tagging_loss=0.01092, over 15042.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09484, pruned_loss=0.01638, audio_tagging_loss=0.009565, over 3043183.45 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 16.0 2023-11-21 14:18:46,565 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230350 2023-11-21 14:18:46,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1535646.6666666667, ans=0.1 2023-11-21 14:18:46,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1535646.6666666667, ans=0.1 2023-11-21 14:18:54,797 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.44 vs. limit=22.5 2023-11-21 14:19:01,199 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 7.941e+01 8.654e+01 9.429e+01 1.278e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 14:19:02,785 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.890e-02 2023-11-21 14:19:13,616 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.02 vs. limit=12.0 2023-11-21 14:19:22,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1535846.6666666667, ans=0.125 2023-11-21 14:19:25,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.14 vs. limit=15.0 2023-11-21 14:19:41,040 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.83 vs. limit=12.0 2023-11-21 14:19:49,114 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 1950, loss[loss=0.05717, simple_loss=0.07799, pruned_loss=0.00797, audio_tagging_loss=0.01021, over 13881.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09483, pruned_loss=0.01646, audio_tagging_loss=0.009592, over 3039280.82 frames. ], batch size: 54, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:19:50,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230400 2023-11-21 14:20:03,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1536046.6666666667, ans=0.125 2023-11-21 14:20:06,190 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.03 vs. limit=15.0 2023-11-21 14:20:07,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0 2023-11-21 14:20:26,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.85 vs. limit=15.0 2023-11-21 14:20:44,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1536246.6666666667, ans=0.2 2023-11-21 14:20:54,217 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2000, loss[loss=0.1008, simple_loss=0.1312, pruned_loss=0.02646, audio_tagging_loss=0.008679, over 16111.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.0942, pruned_loss=0.0164, audio_tagging_loss=0.009637, over 3040280.45 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:20:55,547 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230450 2023-11-21 14:20:56,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1536313.3333333333, ans=0.125 2023-11-21 14:21:03,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1536313.3333333333, ans=0.125 2023-11-21 14:21:03,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1536313.3333333333, ans=0.125 2023-11-21 14:21:10,600 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 7.989e+01 8.739e+01 9.353e+01 1.086e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 14:21:17,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1536380.0, ans=15.0 2023-11-21 14:21:58,509 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2050, loss[loss=0.0776, simple_loss=0.1033, pruned_loss=0.01933, audio_tagging_loss=0.006614, over 15469.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09529, pruned_loss=0.01667, audio_tagging_loss=0.00954, over 3045402.25 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:21:59,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230500 2023-11-21 14:22:19,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1536713.3333333333, ans=0.0 2023-11-21 14:22:50,388 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.71 vs. limit=22.5 2023-11-21 14:23:01,832 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2100, loss[loss=0.09323, simple_loss=0.1291, pruned_loss=0.02002, audio_tagging_loss=0.00863, over 15989.00 frames. ], tot_loss[loss=0.07433, simple_loss=0.09615, pruned_loss=0.01682, audio_tagging_loss=0.009436, over 3045018.55 frames. ], batch size: 54, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:23:03,176 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230550 2023-11-21 14:23:12,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1536980.0, ans=0.125 2023-11-21 14:23:15,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1537046.6666666667, ans=0.1 2023-11-21 14:23:20,272 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.887e+01 8.079e+01 8.846e+01 9.402e+01 1.190e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 14:23:59,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1537246.6666666667, ans=0.125 2023-11-21 14:24:05,864 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2150, loss[loss=0.08225, simple_loss=0.1057, pruned_loss=0.01915, audio_tagging_loss=0.01027, over 14737.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09694, pruned_loss=0.01685, audio_tagging_loss=0.00944, over 3043462.91 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:24:07,683 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230600 2023-11-21 14:24:15,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.10 vs. limit=15.0 2023-11-21 14:24:19,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.90 vs. limit=15.0 2023-11-21 14:24:29,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1537380.0, ans=0.04949747468305833 2023-11-21 14:24:45,355 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:25:11,560 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2200, loss[loss=0.07877, simple_loss=0.1046, pruned_loss=0.01788, audio_tagging_loss=0.008601, over 15366.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09728, pruned_loss=0.01673, audio_tagging_loss=0.009459, over 3040546.26 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:25:12,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230650 2023-11-21 14:25:16,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1537646.6666666667, ans=0.125 2023-11-21 14:25:28,621 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.037e+01 8.654e+01 9.441e+01 1.350e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 14:25:41,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1537780.0, ans=0.07 2023-11-21 14:25:46,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1537780.0, ans=0.125 2023-11-21 14:25:50,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1537846.6666666667, ans=0.125 2023-11-21 14:26:09,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1537913.3333333333, ans=10.0 2023-11-21 14:26:14,309 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2250, loss[loss=0.06997, simple_loss=0.09178, pruned_loss=0.01341, audio_tagging_loss=0.01067, over 14456.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.0974, pruned_loss=0.01672, audio_tagging_loss=0.009425, over 3035306.37 frames. ], batch size: 53, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:26:15,611 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230700 2023-11-21 14:27:17,394 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2300, loss[loss=0.06426, simple_loss=0.08634, pruned_loss=0.01059, audio_tagging_loss=0.01051, over 14870.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09729, pruned_loss=0.01686, audio_tagging_loss=0.009431, over 3039169.34 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:27:18,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230750 2023-11-21 14:27:21,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1538313.3333333333, ans=0.125 2023-11-21 14:27:23,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1538313.3333333333, ans=0.2 2023-11-21 14:27:31,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1538380.0, ans=0.2 2023-11-21 14:27:36,130 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.774e+01 7.885e+01 8.573e+01 9.171e+01 1.137e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-21 14:27:41,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.37 vs. limit=22.5 2023-11-21 14:27:45,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1538446.6666666667, ans=0.0 2023-11-21 14:28:05,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1538513.3333333333, ans=0.125 2023-11-21 14:28:13,480 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:28:16,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1538580.0, ans=0.125 2023-11-21 14:28:22,105 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2350, loss[loss=0.08749, simple_loss=0.1207, pruned_loss=0.01781, audio_tagging_loss=0.009322, over 15603.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.09722, pruned_loss=0.01677, audio_tagging_loss=0.00957, over 3031574.95 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:28:23,394 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230800 2023-11-21 14:28:29,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1538646.6666666667, ans=0.0 2023-11-21 14:28:49,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1538780.0, ans=0.0 2023-11-21 14:28:50,254 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.71 vs. limit=10.0 2023-11-21 14:29:26,628 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2400, loss[loss=0.06047, simple_loss=0.0708, pruned_loss=0.01021, audio_tagging_loss=0.01486, over 15289.00 frames. ], tot_loss[loss=0.07523, simple_loss=0.09744, pruned_loss=0.01672, audio_tagging_loss=0.009792, over 3035496.30 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:29:28,042 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230850 2023-11-21 14:29:33,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1538980.0, ans=0.5 2023-11-21 14:29:40,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.83 vs. limit=22.5 2023-11-21 14:29:42,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1539046.6666666667, ans=0.1 2023-11-21 14:29:43,573 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.345e+01 8.985e+01 9.638e+01 1.227e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-21 14:29:49,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1539046.6666666667, ans=0.125 2023-11-21 14:29:57,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1539113.3333333333, ans=0.125 2023-11-21 14:30:17,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1539246.6666666667, ans=0.0 2023-11-21 14:30:18,173 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.09 vs. limit=22.5 2023-11-21 14:30:24,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1539246.6666666667, ans=0.125 2023-11-21 14:30:26,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1539246.6666666667, ans=0.125 2023-11-21 14:30:29,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1539313.3333333333, ans=0.125 2023-11-21 14:30:30,101 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2450, loss[loss=0.09781, simple_loss=0.1314, pruned_loss=0.02504, audio_tagging_loss=0.00707, over 14502.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.09731, pruned_loss=0.01667, audio_tagging_loss=0.009815, over 3036737.36 frames. ], batch size: 54, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:30:31,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230900 2023-11-21 14:30:36,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1539313.3333333333, ans=0.0 2023-11-21 14:30:44,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1539380.0, ans=0.0 2023-11-21 14:30:49,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1539380.0, ans=0.2 2023-11-21 14:30:58,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1539446.6666666667, ans=0.125 2023-11-21 14:31:35,093 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2500, loss[loss=0.077, simple_loss=0.09388, pruned_loss=0.02273, audio_tagging_loss=0.007327, over 15494.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.097, pruned_loss=0.01657, audio_tagging_loss=0.009748, over 3039091.95 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:31:37,012 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 230950 2023-11-21 14:31:39,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1539646.6666666667, ans=0.125 2023-11-21 14:31:46,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1539646.6666666667, ans=0.125 2023-11-21 14:31:53,378 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 7.942e+01 8.529e+01 9.362e+01 1.301e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-21 14:32:03,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1539780.0, ans=0.5 2023-11-21 14:32:09,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1539780.0, ans=0.125 2023-11-21 14:32:40,135 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2550, loss[loss=0.07567, simple_loss=0.08838, pruned_loss=0.02024, audio_tagging_loss=0.01124, over 13694.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09668, pruned_loss=0.01653, audio_tagging_loss=0.009677, over 3042149.73 frames. ], batch size: 53, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:32:41,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231000 2023-11-21 14:32:42,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.07 vs. limit=22.5 2023-11-21 14:32:47,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1539980.0, ans=0.0 2023-11-21 14:32:49,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1539980.0, ans=15.0 2023-11-21 14:32:50,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1539980.0, ans=0.2 2023-11-21 14:32:55,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1540046.6666666667, ans=0.1 2023-11-21 14:32:56,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.72 vs. limit=10.0 2023-11-21 14:33:00,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1540046.6666666667, ans=0.125 2023-11-21 14:33:05,669 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 14:33:08,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.35 vs. limit=22.5 2023-11-21 14:33:11,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1540113.3333333333, ans=0.0 2023-11-21 14:33:15,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1540113.3333333333, ans=0.1 2023-11-21 14:33:23,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1540180.0, ans=0.125 2023-11-21 14:33:25,949 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.39 vs. limit=10.0 2023-11-21 14:33:44,013 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2600, loss[loss=0.07406, simple_loss=0.08817, pruned_loss=0.01788, audio_tagging_loss=0.01209, over 15478.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09637, pruned_loss=0.01652, audio_tagging_loss=0.009452, over 3044491.15 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:33:45,321 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231050 2023-11-21 14:33:56,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.73 vs. limit=15.0 2023-11-21 14:34:03,325 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.933e+01 8.665e+01 9.563e+01 1.438e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 14:34:08,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1540446.6666666667, ans=0.125 2023-11-21 14:34:11,509 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.81 vs. limit=15.0 2023-11-21 14:34:27,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1540513.3333333333, ans=0.125 2023-11-21 14:34:41,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.34 vs. limit=22.5 2023-11-21 14:34:43,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1540580.0, ans=0.0 2023-11-21 14:34:47,265 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2650, loss[loss=0.08961, simple_loss=0.1127, pruned_loss=0.02284, audio_tagging_loss=0.01042, over 15939.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.09642, pruned_loss=0.01663, audio_tagging_loss=0.00938, over 3049030.03 frames. ], batch size: 63, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:34:48,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231100 2023-11-21 14:35:06,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1540713.3333333333, ans=0.0 2023-11-21 14:35:19,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1540780.0, ans=0.025 2023-11-21 14:35:45,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.54 vs. limit=15.0 2023-11-21 14:35:48,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1540913.3333333333, ans=0.0 2023-11-21 14:35:51,750 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2700, loss[loss=0.09831, simple_loss=0.1328, pruned_loss=0.02575, audio_tagging_loss=0.006134, over 15266.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09673, pruned_loss=0.0167, audio_tagging_loss=0.00934, over 3050917.10 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:35:53,044 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231150 2023-11-21 14:36:10,602 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 8.073e+01 8.815e+01 9.662e+01 1.358e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-21 14:36:21,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1541113.3333333333, ans=0.0 2023-11-21 14:36:24,014 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0 2023-11-21 14:36:26,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1541113.3333333333, ans=0.125 2023-11-21 14:36:34,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1541180.0, ans=0.125 2023-11-21 14:36:39,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1541180.0, ans=0.0 2023-11-21 14:36:44,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1541246.6666666667, ans=0.125 2023-11-21 14:36:45,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1541246.6666666667, ans=0.0 2023-11-21 14:36:55,377 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2750, loss[loss=0.06322, simple_loss=0.07586, pruned_loss=0.01336, audio_tagging_loss=0.01193, over 14644.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09699, pruned_loss=0.01675, audio_tagging_loss=0.009404, over 3055723.17 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:36:56,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231200 2023-11-21 14:37:05,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1541313.3333333333, ans=0.1 2023-11-21 14:37:06,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1541313.3333333333, ans=0.035 2023-11-21 14:37:07,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1541380.0, ans=0.1 2023-11-21 14:37:37,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=1541513.3333333333, ans=0.1 2023-11-21 14:37:46,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1541580.0, ans=10.0 2023-11-21 14:37:50,358 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:37:54,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.40 vs. limit=15.0 2023-11-21 14:37:59,543 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2800, loss[loss=0.05601, simple_loss=0.07677, pruned_loss=0.01033, audio_tagging_loss=0.007287, over 15314.00 frames. ], tot_loss[loss=0.07442, simple_loss=0.09703, pruned_loss=0.01663, audio_tagging_loss=0.009272, over 3050953.18 frames. ], batch size: 61, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:38:00,898 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231250 2023-11-21 14:38:07,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1541646.6666666667, ans=0.09899494936611666 2023-11-21 14:38:12,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1541713.3333333333, ans=0.0 2023-11-21 14:38:19,382 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.133e+01 8.565e+01 9.239e+01 1.235e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-21 14:38:37,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1541846.6666666667, ans=0.0 2023-11-21 14:38:41,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1541846.6666666667, ans=0.125 2023-11-21 14:38:44,540 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.28 vs. limit=15.0 2023-11-21 14:38:44,574 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.65 vs. limit=6.0 2023-11-21 14:38:56,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1541913.3333333333, ans=0.0 2023-11-21 14:39:03,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1541980.0, ans=0.125 2023-11-21 14:39:04,746 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2850, loss[loss=0.08462, simple_loss=0.1153, pruned_loss=0.02058, audio_tagging_loss=0.00637, over 14570.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09671, pruned_loss=0.01645, audio_tagging_loss=0.009292, over 3050330.00 frames. ], batch size: 53, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:39:06,080 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231300 2023-11-21 14:39:25,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1542046.6666666667, ans=0.125 2023-11-21 14:39:29,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.56 vs. limit=6.0 2023-11-21 14:39:36,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1542113.3333333333, ans=0.125 2023-11-21 14:39:53,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.71 vs. limit=10.0 2023-11-21 14:39:57,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1542246.6666666667, ans=0.1 2023-11-21 14:40:09,151 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2900, loss[loss=0.08722, simple_loss=0.1179, pruned_loss=0.01971, audio_tagging_loss=0.008574, over 16261.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09608, pruned_loss=0.01638, audio_tagging_loss=0.009335, over 3049391.23 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:40:10,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231350 2023-11-21 14:40:16,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1542313.3333333333, ans=0.125 2023-11-21 14:40:16,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1542313.3333333333, ans=0.125 2023-11-21 14:40:22,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1542380.0, ans=0.2 2023-11-21 14:40:29,512 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.134e+01 8.745e+01 9.473e+01 1.198e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-21 14:40:35,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1542446.6666666667, ans=0.09899494936611666 2023-11-21 14:40:41,446 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.33 vs. limit=6.0 2023-11-21 14:40:41,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1542446.6666666667, ans=0.125 2023-11-21 14:40:42,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1542446.6666666667, ans=0.04949747468305833 2023-11-21 14:40:42,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5 2023-11-21 14:40:50,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.09 vs. limit=6.0 2023-11-21 14:40:51,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1542513.3333333333, ans=0.0 2023-11-21 14:40:52,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1542513.3333333333, ans=0.125 2023-11-21 14:41:00,813 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0 2023-11-21 14:41:11,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.89 vs. limit=22.5 2023-11-21 14:41:12,927 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 2950, loss[loss=0.07818, simple_loss=0.1031, pruned_loss=0.0187, audio_tagging_loss=0.007931, over 15097.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09727, pruned_loss=0.01667, audio_tagging_loss=0.009417, over 3051568.12 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:41:14,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231400 2023-11-21 14:41:52,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1542846.6666666667, ans=0.1 2023-11-21 14:42:18,045 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3000, loss[loss=0.05815, simple_loss=0.07024, pruned_loss=0.009453, audio_tagging_loss=0.01357, over 14820.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09692, pruned_loss=0.0166, audio_tagging_loss=0.009595, over 3050208.41 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:42:18,046 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 14:42:56,760 INFO [train_asr.py:1253] (3/4) Epoch 20, validation: loss=0.05942, simple_loss=0.05225, pruned_loss=0.00524, audio_tagging_loss=0.02805, over 4681554.00 frames. 2023-11-21 14:42:56,761 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 14:42:58,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231450 2023-11-21 14:43:07,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1542980.0, ans=0.2 2023-11-21 14:43:08,184 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0 2023-11-21 14:43:10,951 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.70 vs. limit=15.0 2023-11-21 14:43:17,879 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.247e+01 8.896e+01 9.640e+01 1.197e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-21 14:43:24,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1543113.3333333333, ans=0.125 2023-11-21 14:43:30,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543113.3333333333, ans=0.1 2023-11-21 14:43:41,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543180.0, ans=0.1 2023-11-21 14:43:49,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.66 vs. limit=15.0 2023-11-21 14:44:00,892 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3050, loss[loss=0.06857, simple_loss=0.0776, pruned_loss=0.01492, audio_tagging_loss=0.01485, over 13837.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09685, pruned_loss=0.01674, audio_tagging_loss=0.009658, over 3038601.96 frames. ], batch size: 53, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:44:02,279 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231500 2023-11-21 14:44:06,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543313.3333333333, ans=0.1 2023-11-21 14:44:08,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1543313.3333333333, ans=0.0 2023-11-21 14:44:17,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1543380.0, ans=0.0 2023-11-21 14:44:24,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1543380.0, ans=0.0 2023-11-21 14:44:35,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543446.6666666667, ans=0.1 2023-11-21 14:44:37,566 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:44:39,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1543513.3333333333, ans=0.1 2023-11-21 14:44:42,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1543513.3333333333, ans=0.0 2023-11-21 14:44:45,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543513.3333333333, ans=0.1 2023-11-21 14:44:48,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1543513.3333333333, ans=0.09899494936611666 2023-11-21 14:45:00,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1543580.0, ans=0.125 2023-11-21 14:45:05,761 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3100, loss[loss=0.07337, simple_loss=0.09241, pruned_loss=0.01711, audio_tagging_loss=0.01005, over 14573.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09767, pruned_loss=0.01677, audio_tagging_loss=0.009635, over 3040678.97 frames. ], batch size: 54, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:45:07,048 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231550 2023-11-21 14:45:13,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1543646.6666666667, ans=0.125 2023-11-21 14:45:13,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1543646.6666666667, ans=0.0 2023-11-21 14:45:17,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1543713.3333333333, ans=0.95 2023-11-21 14:45:17,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1543713.3333333333, ans=0.0 2023-11-21 14:45:25,357 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.074e+01 8.626e+01 9.315e+01 1.250e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-21 14:45:31,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1543780.0, ans=0.125 2023-11-21 14:45:31,494 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.22 vs. limit=15.0 2023-11-21 14:45:33,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1543780.0, ans=0.125 2023-11-21 14:45:34,051 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.76 vs. limit=12.0 2023-11-21 14:45:36,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1543780.0, ans=0.125 2023-11-21 14:45:39,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1543780.0, ans=0.015 2023-11-21 14:45:39,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1543780.0, ans=0.2 2023-11-21 14:45:47,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.44 vs. limit=15.0 2023-11-21 14:45:55,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543913.3333333333, ans=0.1 2023-11-21 14:46:00,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.67 vs. limit=15.0 2023-11-21 14:46:08,547 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3150, loss[loss=0.08115, simple_loss=0.1141, pruned_loss=0.017, audio_tagging_loss=0.007118, over 15191.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09777, pruned_loss=0.01683, audio_tagging_loss=0.009651, over 3036379.11 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 16.0 2023-11-21 14:46:09,930 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231600 2023-11-21 14:46:16,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1543980.0, ans=0.07 2023-11-21 14:46:22,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1544046.6666666667, ans=0.125 2023-11-21 14:46:38,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1544113.3333333333, ans=0.1 2023-11-21 14:46:44,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1544113.3333333333, ans=0.2 2023-11-21 14:47:02,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.93 vs. limit=22.5 2023-11-21 14:47:13,996 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3200, loss[loss=0.0742, simple_loss=0.08882, pruned_loss=0.01909, audio_tagging_loss=0.0107, over 14341.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09784, pruned_loss=0.01677, audio_tagging_loss=0.009674, over 3035185.34 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:47:15,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231650 2023-11-21 14:47:16,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1544313.3333333333, ans=0.125 2023-11-21 14:47:18,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1544313.3333333333, ans=0.0 2023-11-21 14:47:24,140 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 14:47:31,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1544380.0, ans=0.125 2023-11-21 14:47:34,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1544380.0, ans=0.125 2023-11-21 14:47:34,898 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.766e+01 7.989e+01 8.860e+01 9.564e+01 1.510e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-21 14:47:54,770 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 14:48:06,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.09 vs. limit=10.0 2023-11-21 14:48:10,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1544580.0, ans=0.0 2023-11-21 14:48:19,246 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3250, loss[loss=0.08641, simple_loss=0.1121, pruned_loss=0.02252, audio_tagging_loss=0.007839, over 15027.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.0971, pruned_loss=0.01677, audio_tagging_loss=0.009741, over 3041243.67 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 32.0 2023-11-21 14:48:20,595 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231700 2023-11-21 14:48:33,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1544713.3333333333, ans=0.1 2023-11-21 14:49:00,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0 2023-11-21 14:49:20,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1544913.3333333333, ans=0.125 2023-11-21 14:49:22,563 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3300, loss[loss=0.06012, simple_loss=0.0769, pruned_loss=0.009605, audio_tagging_loss=0.01207, over 15696.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.0966, pruned_loss=0.01657, audio_tagging_loss=0.009757, over 3037419.75 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:49:23,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231750 2023-11-21 14:49:27,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1544980.0, ans=0.2 2023-11-21 14:49:30,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1544980.0, ans=0.125 2023-11-21 14:49:32,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1544980.0, ans=0.2 2023-11-21 14:49:43,238 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.162e+01 8.724e+01 9.354e+01 1.405e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-21 14:49:44,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1545046.6666666667, ans=0.0 2023-11-21 14:49:56,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1545113.3333333333, ans=0.0 2023-11-21 14:50:10,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1545180.0, ans=0.2 2023-11-21 14:50:25,995 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3350, loss[loss=0.06087, simple_loss=0.0773, pruned_loss=0.01067, audio_tagging_loss=0.01156, over 15775.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09708, pruned_loss=0.01668, audio_tagging_loss=0.009726, over 3046574.28 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:50:27,351 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231800 2023-11-21 14:51:07,365 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.75 vs. limit=22.5 2023-11-21 14:51:10,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.61 vs. limit=15.0 2023-11-21 14:51:20,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.29 vs. limit=15.0 2023-11-21 14:51:28,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1545580.0, ans=0.025 2023-11-21 14:51:30,454 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3400, loss[loss=0.07936, simple_loss=0.1031, pruned_loss=0.01914, audio_tagging_loss=0.008671, over 16173.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09698, pruned_loss=0.01665, audio_tagging_loss=0.009589, over 3045902.54 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:51:31,719 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231850 2023-11-21 14:51:33,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1545646.6666666667, ans=0.125 2023-11-21 14:51:34,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1545646.6666666667, ans=0.1 2023-11-21 14:51:39,545 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.15 vs. limit=22.5 2023-11-21 14:51:40,636 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.94 vs. limit=22.5 2023-11-21 14:51:47,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1545713.3333333333, ans=0.125 2023-11-21 14:51:49,682 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.184e+01 8.749e+01 9.223e+01 1.235e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 14:51:51,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1545713.3333333333, ans=0.1 2023-11-21 14:52:05,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1545780.0, ans=0.125 2023-11-21 14:52:28,785 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 14:52:28,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1545913.3333333333, ans=0.1 2023-11-21 14:52:33,409 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3450, loss[loss=0.07905, simple_loss=0.104, pruned_loss=0.01639, audio_tagging_loss=0.01065, over 14434.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09716, pruned_loss=0.01673, audio_tagging_loss=0.009538, over 3044854.29 frames. ], batch size: 54, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:52:34,775 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231900 2023-11-21 14:52:58,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1546113.3333333333, ans=0.2 2023-11-21 14:52:58,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1546113.3333333333, ans=0.0 2023-11-21 14:53:03,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1546113.3333333333, ans=0.125 2023-11-21 14:53:23,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1546246.6666666667, ans=0.125 2023-11-21 14:53:26,564 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.98 vs. limit=6.0 2023-11-21 14:53:28,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1546246.6666666667, ans=0.0 2023-11-21 14:53:36,607 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3500, loss[loss=0.08066, simple_loss=0.1063, pruned_loss=0.01775, audio_tagging_loss=0.009777, over 14598.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09609, pruned_loss=0.01644, audio_tagging_loss=0.009507, over 3038142.71 frames. ], batch size: 55, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 14:53:38,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 231950 2023-11-21 14:53:40,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1546313.3333333333, ans=0.125 2023-11-21 14:53:59,552 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.228e+01 8.776e+01 9.674e+01 1.230e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 14:54:08,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1546446.6666666667, ans=0.0 2023-11-21 14:54:10,763 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:54:27,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1546580.0, ans=0.125 2023-11-21 14:54:32,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.71 vs. limit=22.5 2023-11-21 14:54:36,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1546580.0, ans=0.1 2023-11-21 14:54:41,980 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3550, loss[loss=0.06919, simple_loss=0.08354, pruned_loss=0.01489, audio_tagging_loss=0.01253, over 15633.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09663, pruned_loss=0.0167, audio_tagging_loss=0.009485, over 3039267.32 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 14:54:43,356 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232000 2023-11-21 14:54:50,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1546646.6666666667, ans=0.125 2023-11-21 14:55:23,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1546846.6666666667, ans=0.1 2023-11-21 14:55:24,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1546846.6666666667, ans=0.125 2023-11-21 14:55:32,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1546846.6666666667, ans=0.125 2023-11-21 14:55:41,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1546913.3333333333, ans=0.0 2023-11-21 14:55:43,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1546913.3333333333, ans=0.0 2023-11-21 14:55:47,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1546913.3333333333, ans=0.2 2023-11-21 14:55:49,449 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3600, loss[loss=0.0659, simple_loss=0.08355, pruned_loss=0.01417, audio_tagging_loss=0.009949, over 15188.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09627, pruned_loss=0.01669, audio_tagging_loss=0.009428, over 3035559.64 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:55:50,731 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232050 2023-11-21 14:55:54,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1546980.0, ans=0.0 2023-11-21 14:56:10,954 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.980e+01 8.692e+01 9.366e+01 1.171e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-21 14:56:53,204 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3650, loss[loss=0.08001, simple_loss=0.104, pruned_loss=0.02202, audio_tagging_loss=0.005968, over 15493.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09531, pruned_loss=0.01633, audio_tagging_loss=0.009482, over 3039511.49 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:56:54,518 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232100 2023-11-21 14:57:31,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1547513.3333333333, ans=0.1 2023-11-21 14:57:58,558 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3700, loss[loss=0.08854, simple_loss=0.1191, pruned_loss=0.02117, audio_tagging_loss=0.007809, over 16506.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09662, pruned_loss=0.01661, audio_tagging_loss=0.00933, over 3050090.41 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:57:59,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232150 2023-11-21 14:58:12,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1547713.3333333333, ans=0.125 2023-11-21 14:58:18,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.70 vs. limit=12.0 2023-11-21 14:58:20,230 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.897e+01 8.102e+01 8.765e+01 9.489e+01 1.317e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-21 14:58:38,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1547846.6666666667, ans=0.125 2023-11-21 14:58:38,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1547846.6666666667, ans=0.1 2023-11-21 14:59:02,970 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3750, loss[loss=0.07436, simple_loss=0.08908, pruned_loss=0.02011, audio_tagging_loss=0.009717, over 15954.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09657, pruned_loss=0.01667, audio_tagging_loss=0.009413, over 3050979.37 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 14:59:04,272 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232200 2023-11-21 14:59:09,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1547980.0, ans=0.125 2023-11-21 14:59:17,548 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.42 vs. limit=6.0 2023-11-21 14:59:28,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1548113.3333333333, ans=0.125 2023-11-21 14:59:29,136 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.77 vs. limit=10.0 2023-11-21 14:59:30,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1548113.3333333333, ans=0.2 2023-11-21 14:59:46,983 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 14:59:55,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1548246.6666666667, ans=0.07 2023-11-21 14:59:59,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.48 vs. limit=10.0 2023-11-21 15:00:06,898 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3800, loss[loss=0.05436, simple_loss=0.06489, pruned_loss=0.01248, audio_tagging_loss=0.009439, over 15218.00 frames. ], tot_loss[loss=0.07442, simple_loss=0.09659, pruned_loss=0.01667, audio_tagging_loss=0.009458, over 3054523.01 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:00:08,172 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232250 2023-11-21 15:00:29,252 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.346e+01 8.333e+01 9.130e+01 9.759e+01 1.207e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-21 15:00:52,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1548513.3333333333, ans=0.125 2023-11-21 15:01:10,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1548646.6666666667, ans=0.125 2023-11-21 15:01:11,475 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3850, loss[loss=0.08832, simple_loss=0.1113, pruned_loss=0.01922, audio_tagging_loss=0.01347, over 14845.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09579, pruned_loss=0.0165, audio_tagging_loss=0.009582, over 3050369.04 frames. ], batch size: 54, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:01:12,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232300 2023-11-21 15:01:12,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1548646.6666666667, ans=0.125 2023-11-21 15:01:20,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1548646.6666666667, ans=0.0 2023-11-21 15:01:28,481 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:01:58,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.53 vs. limit=10.0 2023-11-21 15:02:15,391 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3900, loss[loss=0.06974, simple_loss=0.08904, pruned_loss=0.01643, audio_tagging_loss=0.008787, over 16092.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09646, pruned_loss=0.01652, audio_tagging_loss=0.009488, over 3051231.01 frames. ], batch size: 61, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:02:16,695 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232350 2023-11-21 15:02:36,390 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.138e+01 8.613e+01 9.443e+01 1.181e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 15:02:40,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1549113.3333333333, ans=0.125 2023-11-21 15:03:00,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1549180.0, ans=0.125 2023-11-21 15:03:05,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1549246.6666666667, ans=0.0 2023-11-21 15:03:19,004 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 3950, loss[loss=0.08787, simple_loss=0.1197, pruned_loss=0.0191, audio_tagging_loss=0.008918, over 16581.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09615, pruned_loss=0.01654, audio_tagging_loss=0.009593, over 3060774.48 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:03:20,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232400 2023-11-21 15:03:32,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1549380.0, ans=0.125 2023-11-21 15:04:09,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1549580.0, ans=0.125 2023-11-21 15:04:09,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1549580.0, ans=0.0 2023-11-21 15:04:23,709 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4000, loss[loss=0.06331, simple_loss=0.06947, pruned_loss=0.01551, audio_tagging_loss=0.01306, over 15831.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09627, pruned_loss=0.01677, audio_tagging_loss=0.009712, over 3055271.83 frames. ], batch size: 62, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:04:24,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1549646.6666666667, ans=0.0 2023-11-21 15:04:25,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232450 2023-11-21 15:04:28,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1549646.6666666667, ans=0.125 2023-11-21 15:04:37,894 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.09 vs. limit=15.0 2023-11-21 15:04:43,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1549713.3333333333, ans=0.09899494936611666 2023-11-21 15:04:44,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1549713.3333333333, ans=0.2 2023-11-21 15:04:45,498 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.188e+01 8.912e+01 9.535e+01 1.152e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-21 15:04:53,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1549780.0, ans=0.1 2023-11-21 15:05:28,171 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4050, loss[loss=0.08765, simple_loss=0.113, pruned_loss=0.02113, audio_tagging_loss=0.01003, over 13990.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09738, pruned_loss=0.01711, audio_tagging_loss=0.009673, over 3052337.90 frames. ], batch size: 53, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:05:29,467 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232500 2023-11-21 15:05:30,625 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 15:05:38,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1549980.0, ans=0.125 2023-11-21 15:05:38,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.96 vs. limit=15.0 2023-11-21 15:06:05,967 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.56 vs. limit=22.5 2023-11-21 15:06:14,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1550180.0, ans=0.1 2023-11-21 15:06:32,485 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4100, loss[loss=0.07897, simple_loss=0.1011, pruned_loss=0.01927, audio_tagging_loss=0.009173, over 15498.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09837, pruned_loss=0.01719, audio_tagging_loss=0.00967, over 3054235.08 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:06:33,871 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232550 2023-11-21 15:06:49,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1550380.0, ans=0.1 2023-11-21 15:06:51,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1550380.0, ans=0.2 2023-11-21 15:06:55,662 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.215e+01 8.893e+01 9.542e+01 1.222e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-21 15:06:57,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1550446.6666666667, ans=0.125 2023-11-21 15:07:03,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1550446.6666666667, ans=0.0 2023-11-21 15:07:24,359 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.85 vs. limit=12.0 2023-11-21 15:07:36,352 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4150, loss[loss=0.04679, simple_loss=0.0624, pruned_loss=0.007553, audio_tagging_loss=0.008036, over 14199.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.09819, pruned_loss=0.01697, audio_tagging_loss=0.009607, over 3048238.02 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:07:37,683 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232600 2023-11-21 15:07:41,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1550646.6666666667, ans=0.0 2023-11-21 15:08:05,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1550780.0, ans=0.125 2023-11-21 15:08:18,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1550846.6666666667, ans=0.1 2023-11-21 15:08:22,763 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 15:08:39,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1550913.3333333333, ans=0.1 2023-11-21 15:08:41,865 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4200, loss[loss=0.06597, simple_loss=0.08564, pruned_loss=0.01391, audio_tagging_loss=0.009237, over 16260.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.0977, pruned_loss=0.01685, audio_tagging_loss=0.009502, over 3052251.41 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:08:43,219 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232650 2023-11-21 15:08:53,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1551046.6666666667, ans=0.125 2023-11-21 15:09:03,666 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.544e+01 8.218e+01 8.944e+01 9.900e+01 1.172e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-21 15:09:06,411 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.69 vs. limit=15.0 2023-11-21 15:09:09,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1551113.3333333333, ans=0.07 2023-11-21 15:09:17,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1551113.3333333333, ans=0.125 2023-11-21 15:09:21,951 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:09:29,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1551180.0, ans=0.2 2023-11-21 15:09:33,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1551246.6666666667, ans=0.125 2023-11-21 15:09:40,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1551246.6666666667, ans=0.1 2023-11-21 15:09:43,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1551246.6666666667, ans=0.0 2023-11-21 15:09:45,358 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4250, loss[loss=0.08927, simple_loss=0.1154, pruned_loss=0.02292, audio_tagging_loss=0.008627, over 15904.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09818, pruned_loss=0.01692, audio_tagging_loss=0.009447, over 3057077.20 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:09:46,709 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232700 2023-11-21 15:10:18,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1551446.6666666667, ans=0.125 2023-11-21 15:10:19,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.15 vs. limit=6.0 2023-11-21 15:10:25,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1551513.3333333333, ans=0.125 2023-11-21 15:10:25,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1551513.3333333333, ans=0.1 2023-11-21 15:10:26,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1551513.3333333333, ans=0.125 2023-11-21 15:10:27,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1551513.3333333333, ans=0.1 2023-11-21 15:10:35,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1551513.3333333333, ans=0.125 2023-11-21 15:10:37,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1551580.0, ans=0.125 2023-11-21 15:10:49,920 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4300, loss[loss=0.08427, simple_loss=0.113, pruned_loss=0.02027, audio_tagging_loss=0.007497, over 15334.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09916, pruned_loss=0.01706, audio_tagging_loss=0.009356, over 3054793.90 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:10:51,782 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232750 2023-11-21 15:10:56,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1551646.6666666667, ans=0.2 2023-11-21 15:11:11,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1551713.3333333333, ans=0.0 2023-11-21 15:11:13,427 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.861e+01 8.351e+01 9.107e+01 1.007e+02 1.335e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-21 15:11:20,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.80 vs. limit=10.0 2023-11-21 15:11:23,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.32 vs. limit=15.0 2023-11-21 15:11:54,995 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4350, loss[loss=0.08028, simple_loss=0.1073, pruned_loss=0.01679, audio_tagging_loss=0.00985, over 15387.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09921, pruned_loss=0.01706, audio_tagging_loss=0.009325, over 3047092.19 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:11:56,277 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232800 2023-11-21 15:12:22,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1552113.3333333333, ans=0.125 2023-11-21 15:12:41,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1552180.0, ans=0.1 2023-11-21 15:12:52,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1552246.6666666667, ans=0.0 2023-11-21 15:12:58,324 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4400, loss[loss=0.07817, simple_loss=0.09263, pruned_loss=0.02009, audio_tagging_loss=0.01177, over 15925.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.09851, pruned_loss=0.01693, audio_tagging_loss=0.009307, over 3050471.44 frames. ], batch size: 61, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:12:59,779 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232850 2023-11-21 15:13:10,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1552380.0, ans=0.125 2023-11-21 15:13:12,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.72 vs. limit=15.0 2023-11-21 15:13:13,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff2.min_abs, batch_count=1552380.0, ans=0.1 2023-11-21 15:13:18,976 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.26 vs. limit=15.0 2023-11-21 15:13:19,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1552380.0, ans=0.0 2023-11-21 15:13:20,842 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.302e+01 8.149e+01 8.654e+01 9.425e+01 1.207e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 15:14:01,248 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4450, loss[loss=0.0637, simple_loss=0.07987, pruned_loss=0.01432, audio_tagging_loss=0.009446, over 15658.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.09752, pruned_loss=0.01667, audio_tagging_loss=0.009384, over 3047602.12 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:14:01,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1552646.6666666667, ans=0.0 2023-11-21 15:14:02,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232900 2023-11-21 15:14:31,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.01 vs. limit=15.0 2023-11-21 15:14:33,119 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:14:48,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1552846.6666666667, ans=0.125 2023-11-21 15:14:51,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1552913.3333333333, ans=0.125 2023-11-21 15:15:06,405 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4500, loss[loss=0.06537, simple_loss=0.08487, pruned_loss=0.01483, audio_tagging_loss=0.008108, over 15849.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09752, pruned_loss=0.01669, audio_tagging_loss=0.009303, over 3046762.00 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0 2023-11-21 15:15:06,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1552980.0, ans=0.1 2023-11-21 15:15:07,755 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 232950 2023-11-21 15:15:14,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1552980.0, ans=0.125 2023-11-21 15:15:24,899 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.85 vs. limit=10.0 2023-11-21 15:15:30,002 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.048e+01 8.795e+01 9.628e+01 1.621e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 15:15:43,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1553180.0, ans=0.1 2023-11-21 15:15:49,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1553180.0, ans=0.0 2023-11-21 15:15:58,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1553246.6666666667, ans=0.125 2023-11-21 15:16:07,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1553246.6666666667, ans=0.0 2023-11-21 15:16:10,802 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4550, loss[loss=0.08113, simple_loss=0.1032, pruned_loss=0.02103, audio_tagging_loss=0.008525, over 15473.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09761, pruned_loss=0.01682, audio_tagging_loss=0.009342, over 3050941.51 frames. ], batch size: 55, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:16:12,072 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233000 2023-11-21 15:16:12,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1553313.3333333333, ans=0.2 2023-11-21 15:16:16,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1553313.3333333333, ans=0.125 2023-11-21 15:16:40,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1553446.6666666667, ans=0.0 2023-11-21 15:16:54,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1553513.3333333333, ans=0.125 2023-11-21 15:17:00,259 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 15:17:04,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1553580.0, ans=0.0 2023-11-21 15:17:09,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1553580.0, ans=0.1 2023-11-21 15:17:15,280 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4600, loss[loss=0.04878, simple_loss=0.06266, pruned_loss=0.008857, audio_tagging_loss=0.008593, over 16303.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09699, pruned_loss=0.01682, audio_tagging_loss=0.009537, over 3050180.39 frames. ], batch size: 61, lr: 3.44e-03, grad_scale: 16.0 2023-11-21 15:17:16,657 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233050 2023-11-21 15:17:40,752 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.142e+01 8.759e+01 9.508e+01 1.262e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-21 15:17:52,622 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.92 vs. limit=15.0 2023-11-21 15:17:54,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1553846.6666666667, ans=0.0 2023-11-21 15:18:13,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1553913.3333333333, ans=0.1 2023-11-21 15:18:16,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1553913.3333333333, ans=0.2 2023-11-21 15:18:21,093 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4650, loss[loss=0.06927, simple_loss=0.09526, pruned_loss=0.01177, audio_tagging_loss=0.009871, over 15502.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09635, pruned_loss=0.01661, audio_tagging_loss=0.009706, over 3052169.56 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 16.0 2023-11-21 15:18:22,994 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233100 2023-11-21 15:18:23,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1553980.0, ans=0.125 2023-11-21 15:18:32,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.18 vs. limit=15.0 2023-11-21 15:18:54,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1554113.3333333333, ans=22.5 2023-11-21 15:19:21,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1554246.6666666667, ans=0.2 2023-11-21 15:19:22,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1554246.6666666667, ans=0.1 2023-11-21 15:19:26,199 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4700, loss[loss=0.05896, simple_loss=0.06878, pruned_loss=0.0111, audio_tagging_loss=0.01347, over 14603.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09477, pruned_loss=0.01617, audio_tagging_loss=0.009823, over 3048990.71 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0 2023-11-21 15:19:27,484 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233150 2023-11-21 15:19:33,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1554313.3333333333, ans=0.125 2023-11-21 15:19:37,520 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:19:49,367 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.239e+01 8.978e+01 9.572e+01 1.151e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-21 15:20:01,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1554446.6666666667, ans=0.1 2023-11-21 15:20:29,866 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4750, loss[loss=0.07445, simple_loss=0.101, pruned_loss=0.01514, audio_tagging_loss=0.008793, over 15400.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09526, pruned_loss=0.01616, audio_tagging_loss=0.009786, over 3048188.25 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0 2023-11-21 15:20:31,161 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233200 2023-11-21 15:20:34,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1554646.6666666667, ans=0.125 2023-11-21 15:20:35,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1554646.6666666667, ans=0.125 2023-11-21 15:20:35,976 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.56 vs. limit=15.0 2023-11-21 15:20:36,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1554646.6666666667, ans=0.125 2023-11-21 15:20:52,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1554713.3333333333, ans=0.125 2023-11-21 15:21:11,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1554846.6666666667, ans=0.0 2023-11-21 15:21:34,022 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4800, loss[loss=0.05856, simple_loss=0.06745, pruned_loss=0.01421, audio_tagging_loss=0.01063, over 14893.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09482, pruned_loss=0.01613, audio_tagging_loss=0.00982, over 3041421.28 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:21:35,965 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233250 2023-11-21 15:21:37,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1554980.0, ans=0.0 2023-11-21 15:21:51,747 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.86 vs. limit=15.0 2023-11-21 15:21:59,669 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.234e+01 8.749e+01 9.624e+01 1.145e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 15:22:10,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.04 vs. limit=15.0 2023-11-21 15:22:31,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1555246.6666666667, ans=0.1 2023-11-21 15:22:40,340 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4850, loss[loss=0.05812, simple_loss=0.07376, pruned_loss=0.01261, audio_tagging_loss=0.008625, over 16353.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09478, pruned_loss=0.01615, audio_tagging_loss=0.009964, over 3039323.82 frames. ], batch size: 62, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:22:41,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233300 2023-11-21 15:23:06,720 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.41 vs. limit=15.0 2023-11-21 15:23:40,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1555580.0, ans=0.125 2023-11-21 15:23:43,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1555646.6666666667, ans=0.0 2023-11-21 15:23:43,871 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4900, loss[loss=0.08071, simple_loss=0.0965, pruned_loss=0.02156, audio_tagging_loss=0.0109, over 15204.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09572, pruned_loss=0.01631, audio_tagging_loss=0.009828, over 3040049.32 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:23:45,296 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233350 2023-11-21 15:23:49,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1555646.6666666667, ans=0.0 2023-11-21 15:24:08,598 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.805e+01 8.037e+01 8.614e+01 9.424e+01 1.262e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 15:24:15,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.55 vs. limit=15.0 2023-11-21 15:24:22,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1555846.6666666667, ans=0.2 2023-11-21 15:24:35,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1555913.3333333333, ans=0.025 2023-11-21 15:24:37,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1555913.3333333333, ans=0.0 2023-11-21 15:24:48,027 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 4950, loss[loss=0.1128, simple_loss=0.1421, pruned_loss=0.03205, audio_tagging_loss=0.009685, over 15050.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09594, pruned_loss=0.01632, audio_tagging_loss=0.009728, over 3034915.21 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:24:49,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233400 2023-11-21 15:24:53,454 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.63 vs. limit=10.0 2023-11-21 15:24:56,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1555980.0, ans=0.125 2023-11-21 15:24:56,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1555980.0, ans=0.125 2023-11-21 15:25:11,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1556046.6666666667, ans=0.125 2023-11-21 15:25:18,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1556113.3333333333, ans=0.0 2023-11-21 15:25:23,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1556113.3333333333, ans=0.1 2023-11-21 15:25:53,387 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5000, loss[loss=0.05866, simple_loss=0.07799, pruned_loss=0.009833, audio_tagging_loss=0.009832, over 14298.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09569, pruned_loss=0.01624, audio_tagging_loss=0.009675, over 3030157.00 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:25:54,617 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233450 2023-11-21 15:26:06,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1556380.0, ans=0.0 2023-11-21 15:26:17,067 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.408e+01 8.018e+01 8.729e+01 9.400e+01 1.250e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 15:26:25,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1556446.6666666667, ans=0.125 2023-11-21 15:26:49,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1556580.0, ans=0.125 2023-11-21 15:26:54,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1556580.0, ans=0.0 2023-11-21 15:26:57,655 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5050, loss[loss=0.07969, simple_loss=0.1154, pruned_loss=0.01475, audio_tagging_loss=0.007253, over 14835.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.0959, pruned_loss=0.01621, audio_tagging_loss=0.009606, over 3041294.00 frames. ], batch size: 53, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:26:58,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233500 2023-11-21 15:27:18,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1556713.3333333333, ans=0.125 2023-11-21 15:27:51,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.66 vs. limit=15.0 2023-11-21 15:27:58,594 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.25 vs. limit=15.0 2023-11-21 15:28:00,936 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.40 vs. limit=22.5 2023-11-21 15:28:01,659 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5100, loss[loss=0.08163, simple_loss=0.1118, pruned_loss=0.01693, audio_tagging_loss=0.008815, over 15891.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09547, pruned_loss=0.01619, audio_tagging_loss=0.009613, over 3039016.05 frames. ], batch size: 59, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:28:02,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233550 2023-11-21 15:28:12,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1556980.0, ans=0.125 2023-11-21 15:28:26,719 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.860e+01 8.610e+01 9.200e+01 1.229e+02, threshold=1.722e+02, percent-clipped=0.0 2023-11-21 15:28:30,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1557113.3333333333, ans=0.125 2023-11-21 15:28:38,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1557113.3333333333, ans=0.2 2023-11-21 15:28:43,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1557180.0, ans=0.0 2023-11-21 15:28:44,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1557180.0, ans=0.125 2023-11-21 15:28:51,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1557246.6666666667, ans=0.125 2023-11-21 15:28:56,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1557246.6666666667, ans=0.1 2023-11-21 15:28:59,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1557246.6666666667, ans=0.0 2023-11-21 15:29:00,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1557246.6666666667, ans=0.125 2023-11-21 15:29:06,979 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5150, loss[loss=0.08244, simple_loss=0.1137, pruned_loss=0.01928, audio_tagging_loss=0.006297, over 15665.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.097, pruned_loss=0.01664, audio_tagging_loss=0.009531, over 3038378.22 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:29:08,280 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233600 2023-11-21 15:29:09,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1557313.3333333333, ans=0.1 2023-11-21 15:29:14,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.87 vs. limit=6.0 2023-11-21 15:29:41,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1557446.6666666667, ans=0.125 2023-11-21 15:29:54,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1557513.3333333333, ans=0.125 2023-11-21 15:30:11,323 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5200, loss[loss=0.08197, simple_loss=0.108, pruned_loss=0.01872, audio_tagging_loss=0.009246, over 15437.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09662, pruned_loss=0.0165, audio_tagging_loss=0.009542, over 3041235.14 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:30:12,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233650 2023-11-21 15:30:25,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1557713.3333333333, ans=0.0 2023-11-21 15:30:35,766 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.165e+01 8.844e+01 9.636e+01 1.346e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 15:31:03,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1557913.3333333333, ans=0.2 2023-11-21 15:31:03,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1557913.3333333333, ans=0.125 2023-11-21 15:31:06,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1557913.3333333333, ans=0.0 2023-11-21 15:31:13,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1557913.3333333333, ans=0.1 2023-11-21 15:31:15,901 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5250, loss[loss=0.06387, simple_loss=0.08131, pruned_loss=0.01207, audio_tagging_loss=0.01115, over 14130.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09598, pruned_loss=0.01646, audio_tagging_loss=0.009544, over 3040139.05 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:31:17,265 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233700 2023-11-21 15:31:28,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1558046.6666666667, ans=0.1 2023-11-21 15:31:37,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1558046.6666666667, ans=0.0 2023-11-21 15:32:02,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1558180.0, ans=0.1 2023-11-21 15:32:12,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1558246.6666666667, ans=0.125 2023-11-21 15:32:19,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1558246.6666666667, ans=0.0 2023-11-21 15:32:21,264 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5300, loss[loss=0.1104, simple_loss=0.1407, pruned_loss=0.03082, audio_tagging_loss=0.009291, over 15613.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09634, pruned_loss=0.0167, audio_tagging_loss=0.009428, over 3035824.36 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:32:22,558 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233750 2023-11-21 15:32:35,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1558380.0, ans=0.1 2023-11-21 15:32:45,204 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.789e+01 8.165e+01 8.716e+01 9.457e+01 1.108e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 15:32:59,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0 2023-11-21 15:33:10,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1558513.3333333333, ans=0.04949747468305833 2023-11-21 15:33:25,922 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5350, loss[loss=0.1066, simple_loss=0.1402, pruned_loss=0.02751, audio_tagging_loss=0.009001, over 15042.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09656, pruned_loss=0.01672, audio_tagging_loss=0.009463, over 3038165.33 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:33:26,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1558646.6666666667, ans=0.1 2023-11-21 15:33:26,547 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.49 vs. limit=15.0 2023-11-21 15:33:27,194 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233800 2023-11-21 15:33:37,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1558713.3333333333, ans=0.1 2023-11-21 15:33:43,358 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.92 vs. limit=6.0 2023-11-21 15:34:07,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1558846.6666666667, ans=0.0 2023-11-21 15:34:12,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1558846.6666666667, ans=0.0 2023-11-21 15:34:19,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1558913.3333333333, ans=0.0 2023-11-21 15:34:20,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1558913.3333333333, ans=0.1 2023-11-21 15:34:30,877 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5400, loss[loss=0.04703, simple_loss=0.05514, pruned_loss=0.00817, audio_tagging_loss=0.01129, over 14252.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09533, pruned_loss=0.01644, audio_tagging_loss=0.009505, over 3038413.68 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 16.0 2023-11-21 15:34:32,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233850 2023-11-21 15:34:48,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1559046.6666666667, ans=0.0 2023-11-21 15:34:53,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1559046.6666666667, ans=0.125 2023-11-21 15:34:56,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1559113.3333333333, ans=0.1 2023-11-21 15:34:57,300 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.221e+01 8.112e+01 8.686e+01 9.399e+01 1.125e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 15:35:03,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1559113.3333333333, ans=0.125 2023-11-21 15:35:11,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1559180.0, ans=0.125 2023-11-21 15:35:15,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1559180.0, ans=0.1 2023-11-21 15:35:36,469 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5450, loss[loss=0.0823, simple_loss=0.1111, pruned_loss=0.01862, audio_tagging_loss=0.008139, over 15932.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09456, pruned_loss=0.01647, audio_tagging_loss=0.009637, over 3044281.55 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 16.0 2023-11-21 15:35:37,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233900 2023-11-21 15:35:42,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1559313.3333333333, ans=0.0 2023-11-21 15:35:43,712 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0 2023-11-21 15:35:46,340 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.62 vs. limit=15.0 2023-11-21 15:35:56,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1559380.0, ans=0.1 2023-11-21 15:35:57,468 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.63 vs. limit=15.0 2023-11-21 15:36:12,442 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=12.0 2023-11-21 15:36:13,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1559513.3333333333, ans=0.125 2023-11-21 15:36:36,606 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.19 vs. limit=15.0 2023-11-21 15:36:40,857 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5500, loss[loss=0.09995, simple_loss=0.12, pruned_loss=0.03079, audio_tagging_loss=0.009191, over 15570.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09522, pruned_loss=0.01661, audio_tagging_loss=0.009671, over 3045880.19 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 16.0 2023-11-21 15:36:42,167 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 233950 2023-11-21 15:36:47,671 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.44 vs. limit=15.0 2023-11-21 15:37:05,835 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.151e+01 8.405e+01 9.190e+01 9.926e+01 1.565e+02, threshold=1.838e+02, percent-clipped=0.0 2023-11-21 15:37:09,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1559780.0, ans=0.125 2023-11-21 15:37:10,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1559780.0, ans=0.0 2023-11-21 15:37:12,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1559780.0, ans=0.2 2023-11-21 15:37:29,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1559846.6666666667, ans=0.0 2023-11-21 15:37:42,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.66 vs. limit=15.0 2023-11-21 15:37:44,555 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5550, loss[loss=0.07125, simple_loss=0.08428, pruned_loss=0.01681, audio_tagging_loss=0.01231, over 15094.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09533, pruned_loss=0.01645, audio_tagging_loss=0.009766, over 3042540.82 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 16.0 2023-11-21 15:37:45,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234000 2023-11-21 15:38:15,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1560113.3333333333, ans=0.125 2023-11-21 15:38:16,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1560113.3333333333, ans=0.0 2023-11-21 15:38:22,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1560113.3333333333, ans=0.2 2023-11-21 15:38:22,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1560113.3333333333, ans=0.025 2023-11-21 15:38:28,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1560180.0, ans=0.125 2023-11-21 15:38:51,271 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5600, loss[loss=0.07844, simple_loss=0.1094, pruned_loss=0.01572, audio_tagging_loss=0.008027, over 14095.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.0956, pruned_loss=0.01639, audio_tagging_loss=0.009783, over 3044411.80 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:38:52,694 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234050 2023-11-21 15:38:55,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1560313.3333333333, ans=0.05 2023-11-21 15:39:05,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1560380.0, ans=0.0 2023-11-21 15:39:09,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1560380.0, ans=0.125 2023-11-21 15:39:16,694 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 7.943e+01 8.570e+01 9.599e+01 1.261e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 15:39:37,654 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 15:39:39,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1560513.3333333333, ans=0.0 2023-11-21 15:39:41,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1560513.3333333333, ans=0.0 2023-11-21 15:39:49,763 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0 2023-11-21 15:39:56,435 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5650, loss[loss=0.06434, simple_loss=0.08309, pruned_loss=0.01208, audio_tagging_loss=0.01071, over 15764.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09589, pruned_loss=0.01654, audio_tagging_loss=0.009773, over 3055459.32 frames. ], batch size: 61, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:39:57,799 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234100 2023-11-21 15:40:03,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1560646.6666666667, ans=0.0 2023-11-21 15:40:17,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1560713.3333333333, ans=0.125 2023-11-21 15:40:41,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1560846.6666666667, ans=0.1 2023-11-21 15:40:41,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1560846.6666666667, ans=0.1 2023-11-21 15:41:00,843 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5700, loss[loss=0.07232, simple_loss=0.09384, pruned_loss=0.01702, audio_tagging_loss=0.008388, over 15410.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09603, pruned_loss=0.01656, audio_tagging_loss=0.009687, over 3044446.95 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:41:01,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1560980.0, ans=0.2 2023-11-21 15:41:02,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234150 2023-11-21 15:41:06,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1560980.0, ans=0.0 2023-11-21 15:41:27,211 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.266e+01 8.051e+01 8.513e+01 9.127e+01 1.242e+02, threshold=1.703e+02, percent-clipped=0.0 2023-11-21 15:41:30,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1561113.3333333333, ans=0.125 2023-11-21 15:41:31,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1561113.3333333333, ans=0.1 2023-11-21 15:41:31,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1561113.3333333333, ans=0.2 2023-11-21 15:41:38,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1561113.3333333333, ans=0.04949747468305833 2023-11-21 15:41:39,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1561180.0, ans=0.125 2023-11-21 15:41:57,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1561246.6666666667, ans=0.0 2023-11-21 15:42:05,391 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5750, loss[loss=0.05252, simple_loss=0.06492, pruned_loss=0.008782, audio_tagging_loss=0.01128, over 15583.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09627, pruned_loss=0.01674, audio_tagging_loss=0.009524, over 3049099.09 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:42:05,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1561313.3333333333, ans=0.0 2023-11-21 15:42:06,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234200 2023-11-21 15:42:26,225 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:42:41,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1561446.6666666667, ans=0.0 2023-11-21 15:42:43,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1561513.3333333333, ans=0.1 2023-11-21 15:43:11,844 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5800, loss[loss=0.07383, simple_loss=0.09492, pruned_loss=0.01418, audio_tagging_loss=0.01218, over 15198.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09624, pruned_loss=0.01668, audio_tagging_loss=0.009456, over 3061438.59 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:43:13,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234250 2023-11-21 15:43:36,364 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.121e+01 8.819e+01 9.541e+01 1.299e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-21 15:43:37,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.07 vs. limit=15.0 2023-11-21 15:43:37,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.52 vs. limit=15.0 2023-11-21 15:43:44,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1561780.0, ans=0.0 2023-11-21 15:44:03,405 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.03 vs. limit=15.0 2023-11-21 15:44:11,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1561913.3333333333, ans=0.07 2023-11-21 15:44:16,109 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5850, loss[loss=0.06294, simple_loss=0.08374, pruned_loss=0.0114, audio_tagging_loss=0.009674, over 15565.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.09603, pruned_loss=0.01649, audio_tagging_loss=0.00942, over 3063120.17 frames. ], batch size: 60, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:44:17,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234300 2023-11-21 15:44:47,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1562113.3333333333, ans=0.125 2023-11-21 15:44:49,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1562113.3333333333, ans=0.0 2023-11-21 15:44:58,767 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.00 vs. limit=22.5 2023-11-21 15:44:59,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1562180.0, ans=0.125 2023-11-21 15:45:05,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1562180.0, ans=0.125 2023-11-21 15:45:20,796 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5900, loss[loss=0.07787, simple_loss=0.09211, pruned_loss=0.02214, audio_tagging_loss=0.009674, over 14954.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09679, pruned_loss=0.01661, audio_tagging_loss=0.009362, over 3062658.66 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:45:22,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234350 2023-11-21 15:45:23,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1562313.3333333333, ans=0.1 2023-11-21 15:45:47,511 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.953e+01 8.033e+01 8.577e+01 9.444e+01 1.216e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-21 15:45:49,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1562446.6666666667, ans=0.125 2023-11-21 15:45:57,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.20 vs. limit=15.0 2023-11-21 15:46:21,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1562580.0, ans=0.125 2023-11-21 15:46:26,267 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0 2023-11-21 15:46:26,929 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 5950, loss[loss=0.0649, simple_loss=0.07839, pruned_loss=0.01832, audio_tagging_loss=0.007386, over 14715.00 frames. ], tot_loss[loss=0.074, simple_loss=0.0966, pruned_loss=0.01648, audio_tagging_loss=0.009216, over 3070056.98 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0 2023-11-21 15:46:27,563 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.64 vs. limit=15.0 2023-11-21 15:46:28,274 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234400 2023-11-21 15:46:38,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1562713.3333333333, ans=0.125 2023-11-21 15:46:45,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1562713.3333333333, ans=0.1 2023-11-21 15:46:45,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1562713.3333333333, ans=10.0 2023-11-21 15:47:00,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.01 vs. limit=22.5 2023-11-21 15:47:29,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1562913.3333333333, ans=0.125 2023-11-21 15:47:31,479 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6000, loss[loss=0.07438, simple_loss=0.0944, pruned_loss=0.01843, audio_tagging_loss=0.00875, over 15049.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09634, pruned_loss=0.01651, audio_tagging_loss=0.009263, over 3068261.91 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:47:31,480 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 15:48:12,894 INFO [train_asr.py:1253] (3/4) Epoch 20, validation: loss=0.06068, simple_loss=0.0522, pruned_loss=0.005214, audio_tagging_loss=0.02937, over 4681554.00 frames. 2023-11-21 15:48:12,895 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 15:48:14,243 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234450 2023-11-21 15:48:33,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1563046.6666666667, ans=0.125 2023-11-21 15:48:38,429 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 7.928e+01 8.696e+01 9.354e+01 1.094e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 15:48:43,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1563113.3333333333, ans=0.0 2023-11-21 15:48:58,825 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 15:49:13,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1563246.6666666667, ans=0.0 2023-11-21 15:49:13,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1563246.6666666667, ans=0.1 2023-11-21 15:49:17,906 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6050, loss[loss=0.09529, simple_loss=0.1284, pruned_loss=0.02248, audio_tagging_loss=0.008618, over 16114.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09628, pruned_loss=0.01649, audio_tagging_loss=0.0093, over 3070347.39 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:49:19,205 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234500 2023-11-21 15:49:21,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.73 vs. limit=15.0 2023-11-21 15:49:35,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1563380.0, ans=0.1 2023-11-21 15:49:45,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1563446.6666666667, ans=0.125 2023-11-21 15:49:53,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1563446.6666666667, ans=0.09899494936611666 2023-11-21 15:50:21,447 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6100, loss[loss=0.06423, simple_loss=0.07889, pruned_loss=0.01551, audio_tagging_loss=0.009281, over 15252.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09635, pruned_loss=0.01653, audio_tagging_loss=0.00935, over 3064526.07 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:50:22,753 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234550 2023-11-21 15:50:38,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1563713.3333333333, ans=0.1 2023-11-21 15:50:47,222 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.060e+01 8.562e+01 9.373e+01 1.321e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-21 15:51:06,210 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=12.0 2023-11-21 15:51:09,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.39 vs. limit=22.5 2023-11-21 15:51:25,612 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6150, loss[loss=0.09844, simple_loss=0.1324, pruned_loss=0.02437, audio_tagging_loss=0.007872, over 14847.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.0958, pruned_loss=0.01634, audio_tagging_loss=0.009367, over 3054263.01 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:51:27,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234600 2023-11-21 15:51:30,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1563980.0, ans=0.125 2023-11-21 15:51:41,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1564046.6666666667, ans=0.125 2023-11-21 15:51:41,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1564046.6666666667, ans=0.04949747468305833 2023-11-21 15:51:55,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1564113.3333333333, ans=0.125 2023-11-21 15:52:00,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1564113.3333333333, ans=0.0 2023-11-21 15:52:10,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1564180.0, ans=0.2 2023-11-21 15:52:11,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.56 vs. limit=15.0 2023-11-21 15:52:30,447 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:52:31,536 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6200, loss[loss=0.06089, simple_loss=0.06531, pruned_loss=0.01194, audio_tagging_loss=0.0163, over 14393.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09541, pruned_loss=0.01625, audio_tagging_loss=0.00962, over 3057220.42 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:52:32,800 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234650 2023-11-21 15:52:35,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.74 vs. limit=15.0 2023-11-21 15:52:36,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1564313.3333333333, ans=0.125 2023-11-21 15:52:37,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1564313.3333333333, ans=0.2 2023-11-21 15:52:41,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.94 vs. limit=15.0 2023-11-21 15:52:55,834 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.24 vs. limit=15.0 2023-11-21 15:52:56,120 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 7.947e+01 8.423e+01 9.173e+01 1.216e+02, threshold=1.685e+02, percent-clipped=0.0 2023-11-21 15:53:12,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1564513.3333333333, ans=0.1 2023-11-21 15:53:21,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1564580.0, ans=0.2 2023-11-21 15:53:30,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1564580.0, ans=0.125 2023-11-21 15:53:35,441 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6250, loss[loss=0.0668, simple_loss=0.08608, pruned_loss=0.01407, audio_tagging_loss=0.009683, over 15949.00 frames. ], tot_loss[loss=0.07423, simple_loss=0.09615, pruned_loss=0.01649, audio_tagging_loss=0.009664, over 3053252.46 frames. ], batch size: 60, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:53:36,818 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234700 2023-11-21 15:54:03,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1564780.0, ans=0.125 2023-11-21 15:54:17,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1564846.6666666667, ans=0.125 2023-11-21 15:54:22,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1564846.6666666667, ans=15.0 2023-11-21 15:54:23,316 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:54:35,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1564913.3333333333, ans=0.09899494936611666 2023-11-21 15:54:39,043 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6300, loss[loss=0.06926, simple_loss=0.09339, pruned_loss=0.01421, audio_tagging_loss=0.008361, over 15548.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.0962, pruned_loss=0.0165, audio_tagging_loss=0.009757, over 3052242.95 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:54:40,894 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234750 2023-11-21 15:54:46,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1564980.0, ans=0.0 2023-11-21 15:55:05,689 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.628e+01 8.110e+01 8.746e+01 9.579e+01 1.135e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-21 15:55:10,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1565113.3333333333, ans=0.2 2023-11-21 15:55:15,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1565113.3333333333, ans=0.125 2023-11-21 15:55:17,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.09 vs. limit=15.0 2023-11-21 15:55:19,387 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0 2023-11-21 15:55:36,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1565246.6666666667, ans=0.125 2023-11-21 15:55:36,678 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.31 vs. limit=6.0 2023-11-21 15:55:45,081 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6350, loss[loss=0.06564, simple_loss=0.07981, pruned_loss=0.01307, audio_tagging_loss=0.01267, over 14336.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09534, pruned_loss=0.01618, audio_tagging_loss=0.009836, over 3043470.88 frames. ], batch size: 54, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:55:46,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234800 2023-11-21 15:55:54,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1565313.3333333333, ans=0.05 2023-11-21 15:56:49,992 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6400, loss[loss=0.07629, simple_loss=0.09788, pruned_loss=0.01629, audio_tagging_loss=0.01106, over 16018.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09482, pruned_loss=0.01609, audio_tagging_loss=0.009944, over 3037432.75 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:56:51,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234850 2023-11-21 15:57:03,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.82 vs. limit=15.0 2023-11-21 15:57:05,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1565713.3333333333, ans=0.0 2023-11-21 15:57:06,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1565713.3333333333, ans=0.125 2023-11-21 15:57:08,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1565713.3333333333, ans=0.125 2023-11-21 15:57:16,949 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.183e+01 8.791e+01 9.502e+01 1.187e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 15:57:23,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1565780.0, ans=0.0 2023-11-21 15:57:42,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1565913.3333333333, ans=0.0 2023-11-21 15:57:44,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.17 vs. limit=15.0 2023-11-21 15:57:46,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1565913.3333333333, ans=0.125 2023-11-21 15:57:55,253 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6450, loss[loss=0.09649, simple_loss=0.1289, pruned_loss=0.02324, audio_tagging_loss=0.008772, over 14962.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09506, pruned_loss=0.01609, audio_tagging_loss=0.009954, over 3038628.58 frames. ], batch size: 54, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:57:56,577 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234900 2023-11-21 15:58:05,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1565980.0, ans=0.015 2023-11-21 15:58:11,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1566046.6666666667, ans=0.0 2023-11-21 15:58:20,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1566113.3333333333, ans=0.1 2023-11-21 15:58:35,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1566180.0, ans=0.0 2023-11-21 15:58:54,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1566246.6666666667, ans=0.125 2023-11-21 15:58:56,053 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.75 vs. limit=15.0 2023-11-21 15:58:59,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1566246.6666666667, ans=0.125 2023-11-21 15:59:00,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1566313.3333333333, ans=0.09899494936611666 2023-11-21 15:59:01,489 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6500, loss[loss=0.07005, simple_loss=0.1007, pruned_loss=0.01301, audio_tagging_loss=0.006705, over 15517.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09526, pruned_loss=0.01609, audio_tagging_loss=0.009832, over 3041071.77 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 15:59:02,865 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 234950 2023-11-21 15:59:22,190 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 15:59:22,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1566380.0, ans=0.2 2023-11-21 15:59:26,835 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.005e+01 8.613e+01 9.254e+01 1.234e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 15:59:38,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1566446.6666666667, ans=0.0 2023-11-21 15:59:55,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1566580.0, ans=0.1 2023-11-21 16:00:06,210 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6550, loss[loss=0.08238, simple_loss=0.1087, pruned_loss=0.01988, audio_tagging_loss=0.008139, over 16477.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09501, pruned_loss=0.01603, audio_tagging_loss=0.009731, over 3047103.95 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:00:07,641 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235000 2023-11-21 16:00:28,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1566713.3333333333, ans=0.0 2023-11-21 16:00:30,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1566713.3333333333, ans=0.07 2023-11-21 16:00:47,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1566846.6666666667, ans=0.0 2023-11-21 16:00:57,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1566913.3333333333, ans=0.125 2023-11-21 16:01:11,114 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6600, loss[loss=0.07441, simple_loss=0.08633, pruned_loss=0.01754, audio_tagging_loss=0.0137, over 13646.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.09453, pruned_loss=0.01593, audio_tagging_loss=0.009684, over 3044009.54 frames. ], batch size: 53, lr: 3.42e-03, grad_scale: 16.0 2023-11-21 16:01:13,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235050 2023-11-21 16:01:17,220 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.79 vs. limit=15.0 2023-11-21 16:01:19,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1566980.0, ans=0.125 2023-11-21 16:01:25,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1567046.6666666667, ans=0.035 2023-11-21 16:01:33,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1567046.6666666667, ans=0.0 2023-11-21 16:01:39,698 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.039e+01 8.543e+01 9.389e+01 1.376e+02, threshold=1.709e+02, percent-clipped=0.0 2023-11-21 16:01:40,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1567113.3333333333, ans=0.0 2023-11-21 16:01:42,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1567113.3333333333, ans=0.125 2023-11-21 16:01:45,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.29 vs. limit=10.0 2023-11-21 16:01:55,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1567180.0, ans=0.125 2023-11-21 16:02:14,273 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0 2023-11-21 16:02:17,520 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6650, loss[loss=0.06142, simple_loss=0.08056, pruned_loss=0.01193, audio_tagging_loss=0.00921, over 14588.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09523, pruned_loss=0.01606, audio_tagging_loss=0.009636, over 3042752.06 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 16.0 2023-11-21 16:02:18,829 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235100 2023-11-21 16:02:20,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1567313.3333333333, ans=0.125 2023-11-21 16:02:30,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1567380.0, ans=0.125 2023-11-21 16:02:40,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1567380.0, ans=0.0 2023-11-21 16:02:43,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1567446.6666666667, ans=0.2 2023-11-21 16:02:54,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1567513.3333333333, ans=0.1 2023-11-21 16:03:06,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1567513.3333333333, ans=0.0 2023-11-21 16:03:12,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1567580.0, ans=0.125 2023-11-21 16:03:22,491 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6700, loss[loss=0.06274, simple_loss=0.08764, pruned_loss=0.01084, audio_tagging_loss=0.00808, over 15161.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09497, pruned_loss=0.0161, audio_tagging_loss=0.009553, over 3045239.15 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 16.0 2023-11-21 16:03:23,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.12 vs. limit=15.0 2023-11-21 16:03:23,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235150 2023-11-21 16:03:27,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1567646.6666666667, ans=0.95 2023-11-21 16:03:49,607 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.083e+01 8.674e+01 9.269e+01 1.242e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 16:03:56,092 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.34 vs. limit=12.0 2023-11-21 16:03:58,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1567780.0, ans=10.0 2023-11-21 16:04:26,636 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6750, loss[loss=0.05833, simple_loss=0.06933, pruned_loss=0.009594, audio_tagging_loss=0.01407, over 14158.00 frames. ], tot_loss[loss=0.07311, simple_loss=0.09479, pruned_loss=0.01606, audio_tagging_loss=0.009649, over 3045562.39 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 16.0 2023-11-21 16:04:27,964 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235200 2023-11-21 16:04:42,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1568046.6666666667, ans=0.0 2023-11-21 16:04:45,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1568046.6666666667, ans=0.1 2023-11-21 16:04:50,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1568046.6666666667, ans=0.2 2023-11-21 16:05:06,583 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0 2023-11-21 16:05:17,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1568246.6666666667, ans=0.1 2023-11-21 16:05:29,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1568246.6666666667, ans=0.125 2023-11-21 16:05:31,729 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6800, loss[loss=0.04989, simple_loss=0.05892, pruned_loss=0.01065, audio_tagging_loss=0.00978, over 14566.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09552, pruned_loss=0.01609, audio_tagging_loss=0.009603, over 3041681.91 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:05:33,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235250 2023-11-21 16:05:35,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1568313.3333333333, ans=0.125 2023-11-21 16:05:57,884 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.054e+01 8.686e+01 9.528e+01 1.643e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 16:05:58,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1568446.6666666667, ans=0.95 2023-11-21 16:06:01,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1568446.6666666667, ans=0.0 2023-11-21 16:06:11,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1568513.3333333333, ans=0.2 2023-11-21 16:06:35,965 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6850, loss[loss=0.0511, simple_loss=0.05964, pruned_loss=0.008576, audio_tagging_loss=0.01271, over 14929.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09602, pruned_loss=0.01627, audio_tagging_loss=0.009525, over 3040687.87 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:06:37,351 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235300 2023-11-21 16:07:27,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1568913.3333333333, ans=0.2 2023-11-21 16:07:30,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1568913.3333333333, ans=0.0 2023-11-21 16:07:39,650 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6900, loss[loss=0.05562, simple_loss=0.07224, pruned_loss=0.01081, audio_tagging_loss=0.008696, over 14628.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09635, pruned_loss=0.01627, audio_tagging_loss=0.009419, over 3043342.66 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:07:41,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235350 2023-11-21 16:08:06,946 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 7.983e+01 8.656e+01 9.439e+01 1.131e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 16:08:24,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1569180.0, ans=0.0 2023-11-21 16:08:27,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1569180.0, ans=0.125 2023-11-21 16:08:30,046 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 16:08:41,911 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.19 vs. limit=22.5 2023-11-21 16:08:44,338 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 6950, loss[loss=0.06677, simple_loss=0.09661, pruned_loss=0.01164, audio_tagging_loss=0.006825, over 15302.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09575, pruned_loss=0.0162, audio_tagging_loss=0.009519, over 3042430.57 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:08:45,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235400 2023-11-21 16:09:02,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1569380.0, ans=0.125 2023-11-21 16:09:02,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.07 vs. limit=15.0 2023-11-21 16:09:25,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1569513.3333333333, ans=10.0 2023-11-21 16:09:32,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1569513.3333333333, ans=0.1 2023-11-21 16:09:44,429 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.66 vs. limit=22.5 2023-11-21 16:09:49,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.63 vs. limit=22.5 2023-11-21 16:09:50,390 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7000, loss[loss=0.08747, simple_loss=0.1127, pruned_loss=0.02079, audio_tagging_loss=0.01032, over 16155.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09494, pruned_loss=0.01609, audio_tagging_loss=0.009585, over 3046788.00 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:09:51,698 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235450 2023-11-21 16:10:05,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1569713.3333333333, ans=0.05 2023-11-21 16:10:05,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1569713.3333333333, ans=0.125 2023-11-21 16:10:12,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1569713.3333333333, ans=0.0 2023-11-21 16:10:15,858 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.127e+01 8.792e+01 9.421e+01 1.149e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 16:10:24,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=1569780.0, ans=0.1 2023-11-21 16:10:34,475 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.27 vs. limit=15.0 2023-11-21 16:10:43,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.64 vs. limit=12.0 2023-11-21 16:10:53,811 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7050, loss[loss=0.07368, simple_loss=0.08605, pruned_loss=0.01865, audio_tagging_loss=0.01201, over 14429.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09509, pruned_loss=0.01616, audio_tagging_loss=0.009611, over 3044704.02 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:10:55,104 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235500 2023-11-21 16:10:58,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1569980.0, ans=0.0 2023-11-21 16:11:07,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1570046.6666666667, ans=0.0 2023-11-21 16:11:07,539 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.95 vs. limit=12.0 2023-11-21 16:11:14,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1570046.6666666667, ans=0.125 2023-11-21 16:11:30,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1570113.3333333333, ans=22.5 2023-11-21 16:11:34,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=12.0 2023-11-21 16:11:56,997 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7100, loss[loss=0.07839, simple_loss=0.1084, pruned_loss=0.01823, audio_tagging_loss=0.005973, over 15481.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09549, pruned_loss=0.01615, audio_tagging_loss=0.009691, over 3049515.85 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 16.0 2023-11-21 16:11:58,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235550 2023-11-21 16:12:14,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1570380.0, ans=0.1 2023-11-21 16:12:25,213 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.128e+01 8.670e+01 9.348e+01 1.083e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-21 16:12:26,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1570446.6666666667, ans=0.125 2023-11-21 16:12:33,413 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.94 vs. limit=22.5 2023-11-21 16:12:37,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.29 vs. limit=15.0 2023-11-21 16:12:41,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1570513.3333333333, ans=0.07 2023-11-21 16:12:48,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1570580.0, ans=0.1 2023-11-21 16:13:01,311 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7150, loss[loss=0.06915, simple_loss=0.08555, pruned_loss=0.01575, audio_tagging_loss=0.01062, over 15145.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09635, pruned_loss=0.01635, audio_tagging_loss=0.009767, over 3056003.29 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 16.0 2023-11-21 16:13:03,321 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235600 2023-11-21 16:13:08,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1570646.6666666667, ans=0.0 2023-11-21 16:13:12,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1570646.6666666667, ans=0.1 2023-11-21 16:13:28,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.29 vs. limit=6.0 2023-11-21 16:13:51,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1570913.3333333333, ans=0.2 2023-11-21 16:14:05,655 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7200, loss[loss=0.06807, simple_loss=0.08337, pruned_loss=0.01529, audio_tagging_loss=0.01109, over 14777.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09678, pruned_loss=0.01644, audio_tagging_loss=0.00975, over 3060217.77 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:14:05,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1570980.0, ans=0.125 2023-11-21 16:14:06,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1570980.0, ans=0.025 2023-11-21 16:14:06,994 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235650 2023-11-21 16:14:07,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1570980.0, ans=0.125 2023-11-21 16:14:08,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1570980.0, ans=0.125 2023-11-21 16:14:22,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.85 vs. limit=22.5 2023-11-21 16:14:33,171 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.636e+01 8.075e+01 8.939e+01 9.765e+01 1.346e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-21 16:14:55,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1571246.6666666667, ans=0.125 2023-11-21 16:15:01,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1571246.6666666667, ans=0.125 2023-11-21 16:15:06,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1571246.6666666667, ans=0.125 2023-11-21 16:15:08,519 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7250, loss[loss=0.06613, simple_loss=0.08353, pruned_loss=0.01289, audio_tagging_loss=0.01147, over 15113.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09674, pruned_loss=0.01644, audio_tagging_loss=0.009735, over 3058135.42 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:15:09,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235700 2023-11-21 16:16:13,776 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7300, loss[loss=0.07008, simple_loss=0.08497, pruned_loss=0.01584, audio_tagging_loss=0.01175, over 13434.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09701, pruned_loss=0.01657, audio_tagging_loss=0.009707, over 3049053.37 frames. ], batch size: 52, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:16:15,119 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235750 2023-11-21 16:16:33,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1571713.3333333333, ans=0.2 2023-11-21 16:16:33,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.30 vs. limit=15.0 2023-11-21 16:16:37,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1571713.3333333333, ans=0.0 2023-11-21 16:16:37,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.73 vs. limit=12.0 2023-11-21 16:16:42,201 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.221e+01 8.789e+01 9.369e+01 1.391e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 16:16:53,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1571846.6666666667, ans=0.0 2023-11-21 16:17:02,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1571846.6666666667, ans=0.1 2023-11-21 16:17:02,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1571846.6666666667, ans=0.125 2023-11-21 16:17:19,080 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7350, loss[loss=0.08284, simple_loss=0.1147, pruned_loss=0.01841, audio_tagging_loss=0.007087, over 15280.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.09815, pruned_loss=0.01673, audio_tagging_loss=0.009442, over 3052974.99 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0 2023-11-21 16:17:20,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235800 2023-11-21 16:17:24,473 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:17:33,276 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.55 vs. limit=15.0 2023-11-21 16:17:56,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1572180.0, ans=0.125 2023-11-21 16:18:11,162 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.00 vs. limit=22.5 2023-11-21 16:18:22,175 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7400, loss[loss=0.07335, simple_loss=0.09762, pruned_loss=0.01496, audio_tagging_loss=0.009575, over 14261.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09736, pruned_loss=0.01655, audio_tagging_loss=0.00935, over 3045136.26 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 32.0 2023-11-21 16:18:23,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235850 2023-11-21 16:18:50,702 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.093e+01 8.770e+01 9.313e+01 1.126e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 16:19:17,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1572580.0, ans=0.0 2023-11-21 16:19:26,623 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7450, loss[loss=0.09316, simple_loss=0.1044, pruned_loss=0.02651, audio_tagging_loss=0.01446, over 14545.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09775, pruned_loss=0.01686, audio_tagging_loss=0.00933, over 3050573.26 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 32.0 2023-11-21 16:19:27,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235900 2023-11-21 16:19:32,202 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:19:32,363 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:19:38,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1572713.3333333333, ans=0.125 2023-11-21 16:19:58,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.37 vs. limit=22.5 2023-11-21 16:19:59,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1572780.0, ans=0.0 2023-11-21 16:20:10,158 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0 2023-11-21 16:20:15,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1572846.6666666667, ans=0.125 2023-11-21 16:20:28,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1572913.3333333333, ans=0.125 2023-11-21 16:20:30,983 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7500, loss[loss=0.06057, simple_loss=0.07083, pruned_loss=0.01406, audio_tagging_loss=0.0111, over 14267.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09734, pruned_loss=0.01681, audio_tagging_loss=0.009305, over 3051086.58 frames. ], batch size: 53, lr: 3.41e-03, grad_scale: 32.0 2023-11-21 16:20:32,338 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 235950 2023-11-21 16:20:33,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1572980.0, ans=0.125 2023-11-21 16:20:41,959 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.36 vs. limit=15.0 2023-11-21 16:20:43,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1573046.6666666667, ans=0.125 2023-11-21 16:20:51,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1573046.6666666667, ans=0.1 2023-11-21 16:20:59,390 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 8.351e+01 8.857e+01 9.417e+01 1.207e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-21 16:21:10,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1573180.0, ans=0.5 2023-11-21 16:21:34,636 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7550, loss[loss=0.07301, simple_loss=0.09454, pruned_loss=0.01583, audio_tagging_loss=0.009917, over 14933.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.0976, pruned_loss=0.01698, audio_tagging_loss=0.009265, over 3060881.50 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 32.0 2023-11-21 16:21:35,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236000 2023-11-21 16:21:37,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1573313.3333333333, ans=0.5 2023-11-21 16:21:42,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1573313.3333333333, ans=0.1 2023-11-21 16:22:04,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.04 vs. limit=15.0 2023-11-21 16:22:16,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1573513.3333333333, ans=0.1 2023-11-21 16:22:24,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1573513.3333333333, ans=0.125 2023-11-21 16:22:30,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1573580.0, ans=0.04949747468305833 2023-11-21 16:22:41,876 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7600, loss[loss=0.04845, simple_loss=0.05219, pruned_loss=0.01055, audio_tagging_loss=0.01181, over 15356.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09723, pruned_loss=0.01694, audio_tagging_loss=0.009196, over 3057826.24 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 32.0 2023-11-21 16:22:43,191 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236050 2023-11-21 16:23:00,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1573713.3333333333, ans=0.125 2023-11-21 16:23:05,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1573780.0, ans=0.125 2023-11-21 16:23:07,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1573780.0, ans=0.125 2023-11-21 16:23:09,948 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.352e+01 8.154e+01 8.768e+01 9.478e+01 1.233e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 16:23:39,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1573913.3333333333, ans=0.125 2023-11-21 16:23:46,826 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7650, loss[loss=0.05314, simple_loss=0.06018, pruned_loss=0.009934, audio_tagging_loss=0.01312, over 14711.00 frames. ], tot_loss[loss=0.0746, simple_loss=0.09703, pruned_loss=0.0168, audio_tagging_loss=0.009288, over 3057457.82 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:23:48,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236100 2023-11-21 16:23:58,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1574046.6666666667, ans=0.07 2023-11-21 16:24:00,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1574046.6666666667, ans=0.125 2023-11-21 16:24:06,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.32 vs. limit=15.0 2023-11-21 16:24:42,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1574246.6666666667, ans=0.125 2023-11-21 16:24:45,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1574246.6666666667, ans=0.0 2023-11-21 16:24:48,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.64 vs. limit=15.0 2023-11-21 16:24:52,297 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7700, loss[loss=0.06427, simple_loss=0.07909, pruned_loss=0.01223, audio_tagging_loss=0.01249, over 14947.00 frames. ], tot_loss[loss=0.07469, simple_loss=0.0972, pruned_loss=0.01673, audio_tagging_loss=0.009357, over 3055100.93 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:24:53,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236150 2023-11-21 16:25:00,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1574313.3333333333, ans=0.125 2023-11-21 16:25:05,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1574380.0, ans=0.125 2023-11-21 16:25:22,845 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.998e+01 8.716e+01 9.349e+01 1.459e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 16:25:26,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1574446.6666666667, ans=0.125 2023-11-21 16:25:29,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1574446.6666666667, ans=0.0 2023-11-21 16:25:34,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.60 vs. limit=15.0 2023-11-21 16:25:57,890 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7750, loss[loss=0.07767, simple_loss=0.1103, pruned_loss=0.01519, audio_tagging_loss=0.007341, over 15917.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09701, pruned_loss=0.01685, audio_tagging_loss=0.009445, over 3056645.47 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:25:59,178 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236200 2023-11-21 16:26:02,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1574646.6666666667, ans=0.125 2023-11-21 16:26:05,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.51 vs. limit=12.0 2023-11-21 16:26:10,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1574713.3333333333, ans=0.0 2023-11-21 16:26:12,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1574713.3333333333, ans=0.1 2023-11-21 16:26:14,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1574713.3333333333, ans=0.125 2023-11-21 16:26:15,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1574713.3333333333, ans=0.5 2023-11-21 16:26:16,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1574713.3333333333, ans=0.1 2023-11-21 16:26:17,841 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:26:30,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1574780.0, ans=0.07 2023-11-21 16:27:03,061 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7800, loss[loss=0.0819, simple_loss=0.09938, pruned_loss=0.01988, audio_tagging_loss=0.01233, over 14321.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09599, pruned_loss=0.01662, audio_tagging_loss=0.009542, over 3052513.86 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:27:04,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236250 2023-11-21 16:27:15,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1575046.6666666667, ans=0.125 2023-11-21 16:27:16,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1575046.6666666667, ans=0.2 2023-11-21 16:27:26,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1575046.6666666667, ans=0.0 2023-11-21 16:27:32,544 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.496e+01 8.260e+01 8.924e+01 9.487e+01 1.142e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-21 16:27:37,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1575113.3333333333, ans=0.0 2023-11-21 16:27:47,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1575180.0, ans=0.125 2023-11-21 16:27:54,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1575246.6666666667, ans=0.2 2023-11-21 16:27:56,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1575246.6666666667, ans=0.125 2023-11-21 16:28:06,793 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7850, loss[loss=0.06952, simple_loss=0.09, pruned_loss=0.01572, audio_tagging_loss=0.008806, over 14889.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09535, pruned_loss=0.01642, audio_tagging_loss=0.009572, over 3051253.45 frames. ], batch size: 58, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:28:07,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1575313.3333333333, ans=0.1 2023-11-21 16:28:08,197 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236300 2023-11-21 16:28:13,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1575313.3333333333, ans=0.2 2023-11-21 16:28:33,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.18 vs. limit=15.0 2023-11-21 16:28:41,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1575446.6666666667, ans=0.125 2023-11-21 16:29:01,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.30 vs. limit=12.0 2023-11-21 16:29:12,433 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7900, loss[loss=0.0929, simple_loss=0.1232, pruned_loss=0.022, audio_tagging_loss=0.00932, over 15504.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09542, pruned_loss=0.01646, audio_tagging_loss=0.009728, over 3056830.58 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:29:13,788 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236350 2023-11-21 16:29:17,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1575646.6666666667, ans=0.09899494936611666 2023-11-21 16:29:27,451 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.54 vs. limit=22.5 2023-11-21 16:29:42,402 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.695e+01 8.223e+01 8.741e+01 9.737e+01 2.619e+02, threshold=1.748e+02, percent-clipped=1.0 2023-11-21 16:29:46,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1575780.0, ans=0.1 2023-11-21 16:30:04,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1575913.3333333333, ans=0.125 2023-11-21 16:30:16,772 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 7950, loss[loss=0.05762, simple_loss=0.063, pruned_loss=0.01335, audio_tagging_loss=0.01277, over 14869.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09614, pruned_loss=0.01658, audio_tagging_loss=0.009807, over 3052304.43 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:30:18,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236400 2023-11-21 16:30:31,801 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 16:30:48,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=15.0 2023-11-21 16:30:51,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1576113.3333333333, ans=0.1 2023-11-21 16:30:56,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.10 vs. limit=22.5 2023-11-21 16:31:09,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1576246.6666666667, ans=0.1 2023-11-21 16:31:14,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1576246.6666666667, ans=0.0 2023-11-21 16:31:16,380 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.88 vs. limit=15.0 2023-11-21 16:31:20,514 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8000, loss[loss=0.09551, simple_loss=0.1316, pruned_loss=0.0223, audio_tagging_loss=0.007417, over 15333.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.0957, pruned_loss=0.01662, audio_tagging_loss=0.009869, over 3049488.89 frames. ], batch size: 54, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:31:21,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236450 2023-11-21 16:31:33,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1576380.0, ans=0.2 2023-11-21 16:31:44,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1576380.0, ans=0.2 2023-11-21 16:31:52,250 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.291e+01 7.990e+01 8.622e+01 9.652e+01 1.226e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-21 16:32:23,867 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8050, loss[loss=0.07989, simple_loss=0.09732, pruned_loss=0.0197, audio_tagging_loss=0.01154, over 14614.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09564, pruned_loss=0.01652, audio_tagging_loss=0.009866, over 3047756.83 frames. ], batch size: 53, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:32:24,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1576646.6666666667, ans=0.125 2023-11-21 16:32:26,474 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236500 2023-11-21 16:32:45,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1576713.3333333333, ans=0.0 2023-11-21 16:33:02,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.01 vs. limit=15.0 2023-11-21 16:33:04,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1576846.6666666667, ans=0.0 2023-11-21 16:33:10,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1576846.6666666667, ans=0.125 2023-11-21 16:33:10,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1576846.6666666667, ans=0.5 2023-11-21 16:33:28,735 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8100, loss[loss=0.06295, simple_loss=0.08209, pruned_loss=0.01358, audio_tagging_loss=0.008332, over 15069.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09549, pruned_loss=0.01661, audio_tagging_loss=0.00979, over 3047519.40 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:33:30,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236550 2023-11-21 16:33:31,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1576980.0, ans=0.125 2023-11-21 16:33:59,139 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.498e+01 8.221e+01 8.771e+01 9.447e+01 1.298e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 16:34:10,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.74 vs. limit=15.0 2023-11-21 16:34:13,586 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.31 vs. limit=15.0 2023-11-21 16:34:15,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1577180.0, ans=0.2 2023-11-21 16:34:31,735 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8150, loss[loss=0.05975, simple_loss=0.07508, pruned_loss=0.01335, audio_tagging_loss=0.008861, over 16213.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09656, pruned_loss=0.0166, audio_tagging_loss=0.009563, over 3051956.91 frames. ], batch size: 62, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:34:33,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236600 2023-11-21 16:34:34,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1577313.3333333333, ans=0.0 2023-11-21 16:34:42,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1577313.3333333333, ans=0.0 2023-11-21 16:35:02,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1577446.6666666667, ans=0.5 2023-11-21 16:35:29,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1577580.0, ans=0.2 2023-11-21 16:35:34,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1577646.6666666667, ans=0.2 2023-11-21 16:35:34,885 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8200, loss[loss=0.05523, simple_loss=0.06318, pruned_loss=0.01105, audio_tagging_loss=0.01259, over 15697.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09703, pruned_loss=0.01651, audio_tagging_loss=0.009428, over 3056699.21 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:35:34,953 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 16:35:36,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236650 2023-11-21 16:35:50,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1577713.3333333333, ans=0.125 2023-11-21 16:35:58,356 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.31 vs. limit=22.5 2023-11-21 16:36:07,542 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.532e+01 8.090e+01 8.732e+01 9.271e+01 1.131e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 16:36:13,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.79 vs. limit=15.0 2023-11-21 16:36:33,001 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.07 vs. limit=12.0 2023-11-21 16:36:40,221 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8250, loss[loss=0.106, simple_loss=0.1387, pruned_loss=0.02708, audio_tagging_loss=0.009595, over 15394.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09672, pruned_loss=0.01647, audio_tagging_loss=0.009373, over 3050456.40 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:36:41,526 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236700 2023-11-21 16:36:53,472 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.96 vs. limit=22.5 2023-11-21 16:37:42,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1578313.3333333333, ans=0.125 2023-11-21 16:37:43,366 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8300, loss[loss=0.04827, simple_loss=0.04905, pruned_loss=0.00847, audio_tagging_loss=0.01527, over 15454.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09576, pruned_loss=0.01637, audio_tagging_loss=0.009515, over 3049216.31 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:37:44,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236750 2023-11-21 16:37:46,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1578313.3333333333, ans=0.0 2023-11-21 16:37:46,554 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.41 vs. limit=10.0 2023-11-21 16:38:14,685 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.835e+01 8.193e+01 8.671e+01 9.431e+01 1.215e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-21 16:38:14,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1578446.6666666667, ans=0.125 2023-11-21 16:38:17,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1578446.6666666667, ans=0.0 2023-11-21 16:38:25,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1578513.3333333333, ans=0.125 2023-11-21 16:38:29,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1578513.3333333333, ans=0.0 2023-11-21 16:38:34,840 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:38:36,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1578580.0, ans=15.0 2023-11-21 16:38:39,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1578580.0, ans=0.125 2023-11-21 16:38:45,470 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8350, loss[loss=0.06466, simple_loss=0.09278, pruned_loss=0.01072, audio_tagging_loss=0.007555, over 15662.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09625, pruned_loss=0.01639, audio_tagging_loss=0.009451, over 3052765.91 frames. ], batch size: 58, lr: 3.41e-03, grad_scale: 8.0 2023-11-21 16:38:46,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236800 2023-11-21 16:38:48,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1578646.6666666667, ans=0.04949747468305833 2023-11-21 16:38:50,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1578646.6666666667, ans=0.125 2023-11-21 16:38:58,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1578713.3333333333, ans=0.125 2023-11-21 16:39:06,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.60 vs. limit=15.0 2023-11-21 16:39:09,448 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.49 vs. limit=22.5 2023-11-21 16:39:26,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1578846.6666666667, ans=0.125 2023-11-21 16:39:45,942 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:39:49,553 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8400, loss[loss=0.07139, simple_loss=0.08967, pruned_loss=0.01571, audio_tagging_loss=0.01084, over 15984.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09606, pruned_loss=0.01637, audio_tagging_loss=0.009468, over 3056026.00 frames. ], batch size: 61, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:39:50,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236850 2023-11-21 16:40:00,112 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.56 vs. limit=22.5 2023-11-21 16:40:08,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1579046.6666666667, ans=0.0 2023-11-21 16:40:20,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1579113.3333333333, ans=0.125 2023-11-21 16:40:21,096 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 7.886e+01 8.656e+01 9.499e+01 1.187e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 16:40:28,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1579180.0, ans=0.125 2023-11-21 16:40:36,264 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0 2023-11-21 16:40:53,311 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8450, loss[loss=0.04982, simple_loss=0.06223, pruned_loss=0.00928, audio_tagging_loss=0.009422, over 15850.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09581, pruned_loss=0.01645, audio_tagging_loss=0.009533, over 3050691.29 frames. ], batch size: 64, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:40:54,662 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236900 2023-11-21 16:40:58,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1579313.3333333333, ans=0.125 2023-11-21 16:41:02,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1579313.3333333333, ans=0.125 2023-11-21 16:41:07,549 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.17 vs. limit=22.5 2023-11-21 16:41:33,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1579513.3333333333, ans=0.1 2023-11-21 16:41:47,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1579580.0, ans=0.1 2023-11-21 16:41:56,720 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8500, loss[loss=0.05932, simple_loss=0.07531, pruned_loss=0.01254, audio_tagging_loss=0.009129, over 14949.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09579, pruned_loss=0.01627, audio_tagging_loss=0.009412, over 3054830.46 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:41:58,196 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 236950 2023-11-21 16:42:03,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1579646.6666666667, ans=0.125 2023-11-21 16:42:03,510 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.97 vs. limit=22.5 2023-11-21 16:42:06,594 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.20 vs. limit=12.0 2023-11-21 16:42:12,953 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:42:23,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1579780.0, ans=0.0 2023-11-21 16:42:24,988 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.16 vs. limit=6.0 2023-11-21 16:42:29,761 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.566e+01 7.919e+01 8.659e+01 9.250e+01 1.200e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 16:42:47,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1579913.3333333333, ans=0.125 2023-11-21 16:43:01,486 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8550, loss[loss=0.08468, simple_loss=0.1182, pruned_loss=0.01688, audio_tagging_loss=0.008695, over 15349.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09506, pruned_loss=0.01598, audio_tagging_loss=0.009531, over 3049851.73 frames. ], batch size: 58, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:43:02,742 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237000 2023-11-21 16:43:11,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1579980.0, ans=0.2 2023-11-21 16:43:42,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1580180.0, ans=0.1 2023-11-21 16:43:45,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1580180.0, ans=0.1 2023-11-21 16:43:55,405 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.94 vs. limit=15.0 2023-11-21 16:44:03,274 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.99 vs. limit=10.0 2023-11-21 16:44:06,844 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8600, loss[loss=0.09853, simple_loss=0.1341, pruned_loss=0.02492, audio_tagging_loss=0.006541, over 16712.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.0964, pruned_loss=0.01643, audio_tagging_loss=0.009548, over 3052790.13 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:44:08,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237050 2023-11-21 16:44:11,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1580313.3333333333, ans=0.125 2023-11-21 16:44:20,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1580380.0, ans=0.0 2023-11-21 16:44:37,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1580446.6666666667, ans=0.1 2023-11-21 16:44:38,175 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 8.198e+01 8.655e+01 9.570e+01 1.230e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 16:45:07,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1580580.0, ans=0.0 2023-11-21 16:45:09,711 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8650, loss[loss=0.08524, simple_loss=0.1063, pruned_loss=0.02373, audio_tagging_loss=0.008356, over 14122.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09732, pruned_loss=0.0167, audio_tagging_loss=0.009532, over 3050567.05 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:45:10,993 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237100 2023-11-21 16:45:12,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1580646.6666666667, ans=0.125 2023-11-21 16:45:24,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1580713.3333333333, ans=0.025 2023-11-21 16:46:12,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1580980.0, ans=0.125 2023-11-21 16:46:14,118 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8700, loss[loss=0.0846, simple_loss=0.1123, pruned_loss=0.01839, audio_tagging_loss=0.01006, over 16926.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.0971, pruned_loss=0.01667, audio_tagging_loss=0.009659, over 3046472.70 frames. ], batch size: 62, lr: 3.41e-03, grad_scale: 16.0 2023-11-21 16:46:15,401 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237150 2023-11-21 16:46:28,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1581046.6666666667, ans=15.0 2023-11-21 16:46:30,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1581046.6666666667, ans=0.125 2023-11-21 16:46:45,567 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.271e+01 8.853e+01 9.798e+01 1.350e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-21 16:46:49,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1581113.3333333333, ans=0.0 2023-11-21 16:46:55,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1581180.0, ans=0.0 2023-11-21 16:47:12,616 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.34 vs. limit=12.0 2023-11-21 16:47:17,472 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8750, loss[loss=0.1081, simple_loss=0.1394, pruned_loss=0.02973, audio_tagging_loss=0.008621, over 16308.00 frames. ], tot_loss[loss=0.07584, simple_loss=0.09831, pruned_loss=0.01702, audio_tagging_loss=0.009668, over 3044362.74 frames. ], batch size: 58, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 16:47:18,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237200 2023-11-21 16:48:01,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1581513.3333333333, ans=0.0 2023-11-21 16:48:09,622 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.57 vs. limit=15.0 2023-11-21 16:48:11,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1581580.0, ans=0.125 2023-11-21 16:48:21,629 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8800, loss[loss=0.08424, simple_loss=0.1043, pruned_loss=0.01666, audio_tagging_loss=0.01543, over 16200.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09813, pruned_loss=0.01709, audio_tagging_loss=0.009765, over 3050299.65 frames. ], batch size: 64, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:48:22,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237250 2023-11-21 16:48:30,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1581646.6666666667, ans=0.0 2023-11-21 16:48:37,277 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:48:46,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1581780.0, ans=0.0 2023-11-21 16:48:49,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1581780.0, ans=0.2 2023-11-21 16:48:53,811 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.845e+01 8.270e+01 9.064e+01 1.008e+02 1.291e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-21 16:48:56,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1581780.0, ans=0.2 2023-11-21 16:49:05,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.72 vs. limit=15.0 2023-11-21 16:49:25,645 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8850, loss[loss=0.05773, simple_loss=0.0793, pruned_loss=0.009755, audio_tagging_loss=0.008327, over 15290.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.09703, pruned_loss=0.01679, audio_tagging_loss=0.009762, over 3051461.82 frames. ], batch size: 58, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:49:26,909 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237300 2023-11-21 16:49:38,432 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 16:50:10,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1582180.0, ans=0.035 2023-11-21 16:50:30,403 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8900, loss[loss=0.09854, simple_loss=0.1291, pruned_loss=0.02726, audio_tagging_loss=0.006711, over 15233.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.0979, pruned_loss=0.01705, audio_tagging_loss=0.009653, over 3058161.12 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:50:31,682 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237350 2023-11-21 16:50:34,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1582313.3333333333, ans=0.0 2023-11-21 16:50:59,002 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:51:01,807 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 7.931e+01 8.617e+01 9.547e+01 1.329e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 16:51:11,213 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:51:33,473 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 8950, loss[loss=0.07706, simple_loss=0.09196, pruned_loss=0.01703, audio_tagging_loss=0.01404, over 15388.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09788, pruned_loss=0.01701, audio_tagging_loss=0.009522, over 3054583.40 frames. ], batch size: 58, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:51:33,699 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:51:34,811 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237400 2023-11-21 16:51:40,888 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:51:52,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0 2023-11-21 16:51:53,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.41 vs. limit=15.0 2023-11-21 16:52:23,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1582846.6666666667, ans=0.125 2023-11-21 16:52:24,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1582913.3333333333, ans=0.0 2023-11-21 16:52:38,587 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9000, loss[loss=0.08406, simple_loss=0.1061, pruned_loss=0.02164, audio_tagging_loss=0.009373, over 15123.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09772, pruned_loss=0.01711, audio_tagging_loss=0.009453, over 3046626.38 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:52:38,587 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 16:53:19,756 INFO [train_asr.py:1253] (3/4) Epoch 20, validation: loss=0.06046, simple_loss=0.05217, pruned_loss=0.005278, audio_tagging_loss=0.0291, over 4681554.00 frames. 2023-11-21 16:53:19,756 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 16:53:21,065 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237450 2023-11-21 16:53:23,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1582980.0, ans=0.0 2023-11-21 16:53:37,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.78 vs. limit=22.5 2023-11-21 16:53:52,349 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.325e+01 8.299e+01 8.982e+01 9.463e+01 1.306e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-21 16:54:01,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1583180.0, ans=0.0 2023-11-21 16:54:07,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1583180.0, ans=0.5 2023-11-21 16:54:22,595 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9050, loss[loss=0.06896, simple_loss=0.09419, pruned_loss=0.01165, audio_tagging_loss=0.01022, over 14467.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.0974, pruned_loss=0.01686, audio_tagging_loss=0.009428, over 3047707.51 frames. ], batch size: 53, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 16:54:23,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237500 2023-11-21 16:54:25,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1583313.3333333333, ans=0.0 2023-11-21 16:54:43,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1583380.0, ans=0.0 2023-11-21 16:54:53,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.93 vs. limit=15.0 2023-11-21 16:54:54,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1583446.6666666667, ans=0.2 2023-11-21 16:55:01,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1583513.3333333333, ans=0.0 2023-11-21 16:55:08,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1583513.3333333333, ans=0.1 2023-11-21 16:55:27,151 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9100, loss[loss=0.07426, simple_loss=0.09964, pruned_loss=0.01622, audio_tagging_loss=0.008218, over 15439.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09643, pruned_loss=0.01675, audio_tagging_loss=0.009409, over 3048696.53 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 16:55:28,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237550 2023-11-21 16:55:39,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1583713.3333333333, ans=0.125 2023-11-21 16:55:44,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1583713.3333333333, ans=0.2 2023-11-21 16:55:56,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1583780.0, ans=0.125 2023-11-21 16:56:00,156 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.100e+01 8.666e+01 9.594e+01 1.245e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 16:56:16,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1583846.6666666667, ans=0.2 2023-11-21 16:56:16,298 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 16:56:32,107 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9150, loss[loss=0.06003, simple_loss=0.08053, pruned_loss=0.009713, audio_tagging_loss=0.01005, over 16150.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09615, pruned_loss=0.01664, audio_tagging_loss=0.009442, over 3052305.73 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 16:56:33,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237600 2023-11-21 16:56:42,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1583980.0, ans=0.125 2023-11-21 16:57:07,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1584113.3333333333, ans=0.125 2023-11-21 16:57:15,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1584180.0, ans=0.125 2023-11-21 16:57:27,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1584246.6666666667, ans=0.125 2023-11-21 16:57:33,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1584246.6666666667, ans=0.0 2023-11-21 16:57:35,467 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9200, loss[loss=0.07505, simple_loss=0.09211, pruned_loss=0.01998, audio_tagging_loss=0.009008, over 15802.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09585, pruned_loss=0.01664, audio_tagging_loss=0.009407, over 3052715.72 frames. ], batch size: 61, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:57:36,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237650 2023-11-21 16:57:43,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1584313.3333333333, ans=0.125 2023-11-21 16:58:05,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1584446.6666666667, ans=0.2 2023-11-21 16:58:08,692 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.617e+01 8.012e+01 8.576e+01 9.231e+01 1.366e+02, threshold=1.715e+02, percent-clipped=0.0 2023-11-21 16:58:17,578 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=3.898e-02 2023-11-21 16:58:32,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.99 vs. limit=15.0 2023-11-21 16:58:37,790 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9250, loss[loss=0.09005, simple_loss=0.1167, pruned_loss=0.02373, audio_tagging_loss=0.007981, over 15545.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.09582, pruned_loss=0.01655, audio_tagging_loss=0.009325, over 3054857.92 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:58:39,704 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237700 2023-11-21 16:58:44,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1584646.6666666667, ans=0.1 2023-11-21 16:58:51,322 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.76 vs. limit=22.5 2023-11-21 16:58:58,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1584713.3333333333, ans=0.05 2023-11-21 16:59:19,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1584846.6666666667, ans=0.125 2023-11-21 16:59:42,839 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9300, loss[loss=0.07748, simple_loss=0.1013, pruned_loss=0.01634, audio_tagging_loss=0.01049, over 15742.00 frames. ], tot_loss[loss=0.07398, simple_loss=0.09617, pruned_loss=0.01657, audio_tagging_loss=0.009324, over 3053643.44 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 16:59:44,218 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237750 2023-11-21 16:59:51,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1584980.0, ans=0.0 2023-11-21 17:00:01,889 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.45 vs. limit=15.0 2023-11-21 17:00:10,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1585113.3333333333, ans=0.1 2023-11-21 17:00:14,599 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.079e+01 8.667e+01 9.296e+01 1.099e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 17:00:45,977 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9350, loss[loss=0.1005, simple_loss=0.1386, pruned_loss=0.02491, audio_tagging_loss=0.006241, over 15179.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09645, pruned_loss=0.01656, audio_tagging_loss=0.009319, over 3047225.24 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:00:47,282 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237800 2023-11-21 17:00:49,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.75 vs. limit=15.0 2023-11-21 17:01:10,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1585446.6666666667, ans=0.125 2023-11-21 17:01:35,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.84 vs. limit=15.0 2023-11-21 17:01:49,480 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9400, loss[loss=0.06795, simple_loss=0.08882, pruned_loss=0.0145, audio_tagging_loss=0.00904, over 14691.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.09733, pruned_loss=0.01679, audio_tagging_loss=0.009361, over 3041715.46 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:01:50,883 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237850 2023-11-21 17:02:17,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1585780.0, ans=0.125 2023-11-21 17:02:24,231 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.317e+01 9.011e+01 9.971e+01 1.419e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-21 17:02:24,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1585780.0, ans=0.0 2023-11-21 17:02:36,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1585846.6666666667, ans=0.125 2023-11-21 17:02:53,501 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:02:55,304 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9450, loss[loss=0.06698, simple_loss=0.07961, pruned_loss=0.01704, audio_tagging_loss=0.01013, over 14426.00 frames. ], tot_loss[loss=0.07501, simple_loss=0.09742, pruned_loss=0.01692, audio_tagging_loss=0.009383, over 3040849.69 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:02:56,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237900 2023-11-21 17:03:09,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1586046.6666666667, ans=0.125 2023-11-21 17:03:11,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1586046.6666666667, ans=0.125 2023-11-21 17:03:19,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1586113.3333333333, ans=0.125 2023-11-21 17:03:26,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1586113.3333333333, ans=0.1 2023-11-21 17:03:39,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1586180.0, ans=0.125 2023-11-21 17:03:43,891 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.32 vs. limit=6.0 2023-11-21 17:03:58,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1586313.3333333333, ans=0.1 2023-11-21 17:03:59,166 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9500, loss[loss=0.09096, simple_loss=0.121, pruned_loss=0.02241, audio_tagging_loss=0.008072, over 15308.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09712, pruned_loss=0.01681, audio_tagging_loss=0.009557, over 3044963.30 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:04:00,508 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 237950 2023-11-21 17:04:08,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1586313.3333333333, ans=0.125 2023-11-21 17:04:08,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.37 vs. limit=15.0 2023-11-21 17:04:31,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0 2023-11-21 17:04:31,962 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.163e+01 8.915e+01 9.675e+01 1.127e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-21 17:04:33,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1586446.6666666667, ans=0.125 2023-11-21 17:04:50,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.50 vs. limit=22.5 2023-11-21 17:05:00,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1586646.6666666667, ans=0.0 2023-11-21 17:05:01,661 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9550, loss[loss=0.07209, simple_loss=0.09385, pruned_loss=0.01637, audio_tagging_loss=0.008802, over 15321.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09679, pruned_loss=0.01681, audio_tagging_loss=0.009681, over 3046397.04 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:05:03,002 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238000 2023-11-21 17:05:06,360 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0 2023-11-21 17:05:10,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1586646.6666666667, ans=0.125 2023-11-21 17:05:19,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1586713.3333333333, ans=0.0 2023-11-21 17:05:30,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1586780.0, ans=0.0 2023-11-21 17:05:32,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1586780.0, ans=0.2 2023-11-21 17:05:49,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1586846.6666666667, ans=0.125 2023-11-21 17:05:57,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1586913.3333333333, ans=10.0 2023-11-21 17:06:06,210 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9600, loss[loss=0.07652, simple_loss=0.1051, pruned_loss=0.01603, audio_tagging_loss=0.007926, over 16356.00 frames. ], tot_loss[loss=0.07469, simple_loss=0.09671, pruned_loss=0.01663, audio_tagging_loss=0.00971, over 3048401.58 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:06:07,537 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238050 2023-11-21 17:06:14,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1586980.0, ans=0.025 2023-11-21 17:06:20,092 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.64 vs. limit=15.0 2023-11-21 17:06:38,577 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.410e+01 9.133e+01 9.513e+01 1.325e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-21 17:06:54,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1587180.0, ans=0.0 2023-11-21 17:07:07,175 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.88 vs. limit=22.5 2023-11-21 17:07:10,211 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9650, loss[loss=0.06911, simple_loss=0.09395, pruned_loss=0.01324, audio_tagging_loss=0.008889, over 15380.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09659, pruned_loss=0.01663, audio_tagging_loss=0.009641, over 3050498.32 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:07:11,563 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238100 2023-11-21 17:07:31,358 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 17:07:39,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1587446.6666666667, ans=0.125 2023-11-21 17:07:52,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1587513.3333333333, ans=0.125 2023-11-21 17:07:56,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1587513.3333333333, ans=0.125 2023-11-21 17:08:13,223 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9700, loss[loss=0.08858, simple_loss=0.1138, pruned_loss=0.02098, audio_tagging_loss=0.0107, over 14674.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09716, pruned_loss=0.01678, audio_tagging_loss=0.00951, over 3051656.26 frames. ], batch size: 53, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:08:13,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1587646.6666666667, ans=0.0 2023-11-21 17:08:14,577 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238150 2023-11-21 17:08:17,484 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.37 vs. limit=22.5 2023-11-21 17:08:46,892 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.277e+01 8.905e+01 9.605e+01 1.183e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-21 17:09:17,301 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9750, loss[loss=0.1081, simple_loss=0.1476, pruned_loss=0.02971, audio_tagging_loss=0.004614, over 15705.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09626, pruned_loss=0.0165, audio_tagging_loss=0.009454, over 3047880.77 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:09:17,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1587980.0, ans=0.125 2023-11-21 17:09:18,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238200 2023-11-21 17:09:43,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1588113.3333333333, ans=0.04949747468305833 2023-11-21 17:10:21,266 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9800, loss[loss=0.07383, simple_loss=0.1046, pruned_loss=0.01348, audio_tagging_loss=0.008061, over 15241.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09634, pruned_loss=0.01641, audio_tagging_loss=0.009408, over 3040759.09 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0 2023-11-21 17:10:22,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238250 2023-11-21 17:10:34,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1588380.0, ans=0.1 2023-11-21 17:10:36,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1588380.0, ans=0.05 2023-11-21 17:10:37,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1588380.0, ans=0.1 2023-11-21 17:10:40,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1588380.0, ans=0.125 2023-11-21 17:10:46,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1588446.6666666667, ans=0.125 2023-11-21 17:10:47,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1588446.6666666667, ans=0.0 2023-11-21 17:10:53,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1588446.6666666667, ans=0.1 2023-11-21 17:10:55,715 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.038e+01 8.658e+01 9.399e+01 1.112e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 17:11:00,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1588513.3333333333, ans=0.1 2023-11-21 17:11:05,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.90 vs. limit=15.0 2023-11-21 17:11:17,625 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:11:19,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1588580.0, ans=0.2 2023-11-21 17:11:24,848 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9850, loss[loss=0.06874, simple_loss=0.08283, pruned_loss=0.01481, audio_tagging_loss=0.01251, over 13867.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09594, pruned_loss=0.01637, audio_tagging_loss=0.009427, over 3049994.85 frames. ], batch size: 53, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 17:11:25,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1588646.6666666667, ans=0.0 2023-11-21 17:11:26,119 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238300 2023-11-21 17:11:27,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1588646.6666666667, ans=0.125 2023-11-21 17:11:28,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1588646.6666666667, ans=0.025 2023-11-21 17:11:47,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1588713.3333333333, ans=0.125 2023-11-21 17:12:06,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1588846.6666666667, ans=0.125 2023-11-21 17:12:21,453 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=22.5 2023-11-21 17:12:29,608 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9900, loss[loss=0.06721, simple_loss=0.08748, pruned_loss=0.01465, audio_tagging_loss=0.008824, over 15123.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.0958, pruned_loss=0.01629, audio_tagging_loss=0.00939, over 3048980.54 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 8.0 2023-11-21 17:12:30,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238350 2023-11-21 17:12:54,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1589113.3333333333, ans=0.125 2023-11-21 17:13:00,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1589113.3333333333, ans=0.125 2023-11-21 17:13:04,957 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 7.926e+01 8.564e+01 9.409e+01 1.276e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-21 17:13:11,911 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.24 vs. limit=15.0 2023-11-21 17:13:28,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1589246.6666666667, ans=0.0 2023-11-21 17:13:33,215 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 9950, loss[loss=0.06206, simple_loss=0.07337, pruned_loss=0.01542, audio_tagging_loss=0.009953, over 14896.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09526, pruned_loss=0.01625, audio_tagging_loss=0.009417, over 3053991.31 frames. ], batch size: 58, lr: 3.40e-03, grad_scale: 8.0 2023-11-21 17:13:34,512 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238400 2023-11-21 17:14:00,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.62 vs. limit=15.0 2023-11-21 17:14:05,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1589446.6666666667, ans=0.2 2023-11-21 17:14:32,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1589580.0, ans=0.125 2023-11-21 17:14:36,963 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10000, loss[loss=0.07828, simple_loss=0.1055, pruned_loss=0.01648, audio_tagging_loss=0.009063, over 14774.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09436, pruned_loss=0.01598, audio_tagging_loss=0.009413, over 3041329.70 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 17:14:38,308 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238450 2023-11-21 17:14:43,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0 2023-11-21 17:14:44,852 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=15.0 2023-11-21 17:14:50,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1589713.3333333333, ans=0.125 2023-11-21 17:15:13,232 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.031e+01 7.913e+01 8.716e+01 9.510e+01 1.205e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 17:15:13,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1589780.0, ans=0.1 2023-11-21 17:15:21,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.41 vs. limit=15.0 2023-11-21 17:15:22,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1589846.6666666667, ans=0.0 2023-11-21 17:15:23,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.45 vs. limit=15.0 2023-11-21 17:15:31,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1589913.3333333333, ans=0.2 2023-11-21 17:15:35,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1589913.3333333333, ans=0.125 2023-11-21 17:15:41,448 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10050, loss[loss=0.09275, simple_loss=0.1373, pruned_loss=0.01968, audio_tagging_loss=0.004398, over 15481.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09475, pruned_loss=0.01613, audio_tagging_loss=0.009423, over 3039888.90 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 17:15:42,738 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238500 2023-11-21 17:15:42,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1589980.0, ans=0.1 2023-11-21 17:15:46,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.14 vs. limit=15.0 2023-11-21 17:15:50,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1589980.0, ans=0.125 2023-11-21 17:15:50,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1589980.0, ans=0.125 2023-11-21 17:15:51,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1589980.0, ans=15.0 2023-11-21 17:15:54,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1590046.6666666667, ans=0.2 2023-11-21 17:15:59,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.32 vs. limit=15.0 2023-11-21 17:16:35,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1590246.6666666667, ans=0.0 2023-11-21 17:16:42,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1590246.6666666667, ans=0.125 2023-11-21 17:16:46,166 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10100, loss[loss=0.1014, simple_loss=0.1401, pruned_loss=0.02434, audio_tagging_loss=0.006996, over 15663.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09598, pruned_loss=0.01625, audio_tagging_loss=0.009384, over 3046033.76 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 16.0 2023-11-21 17:16:47,395 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238550 2023-11-21 17:16:56,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1590313.3333333333, ans=0.125 2023-11-21 17:17:22,050 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.174e+01 9.110e+01 9.882e+01 1.190e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-21 17:17:25,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1590513.3333333333, ans=0.125 2023-11-21 17:17:27,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1590513.3333333333, ans=0.0 2023-11-21 17:17:37,870 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:17:49,981 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10150, loss[loss=0.07368, simple_loss=0.0922, pruned_loss=0.01486, audio_tagging_loss=0.01272, over 16757.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09604, pruned_loss=0.01618, audio_tagging_loss=0.009479, over 3052835.70 frames. ], batch size: 63, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:17:51,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238600 2023-11-21 17:18:05,627 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 17:18:20,068 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:18:53,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.42 vs. limit=15.0 2023-11-21 17:18:54,021 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10200, loss[loss=0.08532, simple_loss=0.109, pruned_loss=0.02043, audio_tagging_loss=0.0104, over 14513.00 frames. ], tot_loss[loss=0.07433, simple_loss=0.09694, pruned_loss=0.01643, audio_tagging_loss=0.009431, over 3050383.79 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:18:54,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1590980.0, ans=0.1 2023-11-21 17:18:55,298 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238650 2023-11-21 17:19:01,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1590980.0, ans=0.1 2023-11-21 17:19:03,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1590980.0, ans=15.0 2023-11-21 17:19:04,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0 2023-11-21 17:19:07,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1591046.6666666667, ans=0.125 2023-11-21 17:19:17,860 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:19:17,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1591046.6666666667, ans=0.125 2023-11-21 17:19:29,938 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.636e+01 9.230e+01 9.917e+01 1.306e+02, threshold=1.846e+02, percent-clipped=0.0 2023-11-21 17:19:39,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1591180.0, ans=0.1 2023-11-21 17:19:39,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.10 vs. limit=15.0 2023-11-21 17:19:49,440 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.74 vs. limit=15.0 2023-11-21 17:19:50,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1591246.6666666667, ans=0.0 2023-11-21 17:19:51,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1591246.6666666667, ans=0.04949747468305833 2023-11-21 17:19:51,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1591246.6666666667, ans=0.125 2023-11-21 17:19:58,397 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10250, loss[loss=0.05556, simple_loss=0.06425, pruned_loss=0.01136, audio_tagging_loss=0.01207, over 14861.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09563, pruned_loss=0.01613, audio_tagging_loss=0.009539, over 3048398.41 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:19:59,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238700 2023-11-21 17:20:20,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1591380.0, ans=0.125 2023-11-21 17:20:22,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1591446.6666666667, ans=0.1 2023-11-21 17:20:45,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1591513.3333333333, ans=0.125 2023-11-21 17:20:51,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1591580.0, ans=0.125 2023-11-21 17:21:01,964 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10300, loss[loss=0.08492, simple_loss=0.1065, pruned_loss=0.02168, audio_tagging_loss=0.009998, over 15585.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09678, pruned_loss=0.01634, audio_tagging_loss=0.009523, over 3053561.48 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:21:03,219 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238750 2023-11-21 17:21:04,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1591646.6666666667, ans=0.2 2023-11-21 17:21:19,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1591713.3333333333, ans=0.0 2023-11-21 17:21:38,272 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 7.986e+01 8.562e+01 9.250e+01 1.387e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-21 17:22:05,304 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10350, loss[loss=0.08185, simple_loss=0.09589, pruned_loss=0.01905, audio_tagging_loss=0.01486, over 15712.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09667, pruned_loss=0.01629, audio_tagging_loss=0.009674, over 3056717.03 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:22:06,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238800 2023-11-21 17:22:09,235 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.13 vs. limit=15.0 2023-11-21 17:22:10,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0 2023-11-21 17:22:18,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=1592046.6666666667, ans=0.02 2023-11-21 17:22:35,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1592113.3333333333, ans=0.0 2023-11-21 17:22:36,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1592113.3333333333, ans=0.0 2023-11-21 17:23:07,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.10 vs. limit=15.0 2023-11-21 17:23:10,377 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10400, loss[loss=0.09126, simple_loss=0.1254, pruned_loss=0.01943, audio_tagging_loss=0.009111, over 16424.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09593, pruned_loss=0.01609, audio_tagging_loss=0.009793, over 3056501.40 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:23:11,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238850 2023-11-21 17:23:31,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.24 vs. limit=6.0 2023-11-21 17:23:46,371 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 7.924e+01 8.815e+01 9.464e+01 2.008e+02, threshold=1.763e+02, percent-clipped=1.0 2023-11-21 17:23:51,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1592513.3333333333, ans=0.2 2023-11-21 17:24:14,048 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10450, loss[loss=0.06891, simple_loss=0.08361, pruned_loss=0.01534, audio_tagging_loss=0.01177, over 15737.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.09479, pruned_loss=0.01594, audio_tagging_loss=0.009796, over 3054300.80 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:24:15,335 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238900 2023-11-21 17:24:33,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1592713.3333333333, ans=0.0 2023-11-21 17:24:36,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1592713.3333333333, ans=0.0 2023-11-21 17:25:12,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1592913.3333333333, ans=0.0 2023-11-21 17:25:14,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1592913.3333333333, ans=6.0 2023-11-21 17:25:16,687 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10500, loss[loss=0.0625, simple_loss=0.07831, pruned_loss=0.014, audio_tagging_loss=0.009346, over 15352.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.09592, pruned_loss=0.01617, audio_tagging_loss=0.009681, over 3051926.65 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:25:17,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 238950 2023-11-21 17:25:29,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1593046.6666666667, ans=0.2 2023-11-21 17:25:33,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1593046.6666666667, ans=0.0 2023-11-21 17:25:54,785 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.981e+01 8.666e+01 9.446e+01 1.198e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 17:25:58,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1593180.0, ans=0.0 2023-11-21 17:26:08,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1593246.6666666667, ans=0.125 2023-11-21 17:26:11,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1593246.6666666667, ans=0.0 2023-11-21 17:26:22,629 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10550, loss[loss=0.05356, simple_loss=0.07606, pruned_loss=0.007097, audio_tagging_loss=0.008431, over 15791.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09662, pruned_loss=0.01633, audio_tagging_loss=0.009551, over 3053252.41 frames. ], batch size: 60, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:26:23,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239000 2023-11-21 17:26:25,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1593313.3333333333, ans=0.125 2023-11-21 17:26:37,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1593380.0, ans=0.1 2023-11-21 17:26:52,295 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.39 vs. limit=15.0 2023-11-21 17:27:02,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.52 vs. limit=15.0 2023-11-21 17:27:03,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1593513.3333333333, ans=0.0 2023-11-21 17:27:06,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1593513.3333333333, ans=0.1 2023-11-21 17:27:26,970 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10600, loss[loss=0.07318, simple_loss=0.0942, pruned_loss=0.01666, audio_tagging_loss=0.009419, over 13916.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09687, pruned_loss=0.01643, audio_tagging_loss=0.009494, over 3054462.64 frames. ], batch size: 54, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:27:27,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1593646.6666666667, ans=0.0 2023-11-21 17:27:28,360 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239050 2023-11-21 17:27:33,053 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.12 vs. limit=15.0 2023-11-21 17:27:37,659 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.24 vs. limit=15.0 2023-11-21 17:27:46,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1593713.3333333333, ans=0.125 2023-11-21 17:28:04,037 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.229e+01 8.607e+01 9.453e+01 1.327e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-21 17:28:09,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1593846.6666666667, ans=0.0 2023-11-21 17:28:26,083 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.53 vs. limit=15.0 2023-11-21 17:28:30,406 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10650, loss[loss=0.09134, simple_loss=0.1221, pruned_loss=0.02235, audio_tagging_loss=0.007954, over 14449.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09686, pruned_loss=0.01649, audio_tagging_loss=0.009419, over 3053107.34 frames. ], batch size: 52, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:28:31,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239100 2023-11-21 17:28:56,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1594113.3333333333, ans=0.125 2023-11-21 17:29:02,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1594113.3333333333, ans=0.1 2023-11-21 17:29:14,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1594180.0, ans=0.0 2023-11-21 17:29:34,681 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10700, loss[loss=0.07055, simple_loss=0.08855, pruned_loss=0.01584, audio_tagging_loss=0.01043, over 15193.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09635, pruned_loss=0.01629, audio_tagging_loss=0.009421, over 3050417.64 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:29:35,998 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239150 2023-11-21 17:29:37,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1594313.3333333333, ans=0.2 2023-11-21 17:30:11,205 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.705e+01 8.191e+01 8.750e+01 9.714e+01 1.316e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 17:30:15,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.14 vs. limit=15.0 2023-11-21 17:30:38,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1594646.6666666667, ans=0.2 2023-11-21 17:30:39,193 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10750, loss[loss=0.05644, simple_loss=0.06967, pruned_loss=0.009267, audio_tagging_loss=0.01234, over 14077.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09691, pruned_loss=0.01648, audio_tagging_loss=0.00937, over 3056429.59 frames. ], batch size: 53, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:30:40,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239200 2023-11-21 17:31:17,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1594846.6666666667, ans=0.0 2023-11-21 17:31:21,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1594846.6666666667, ans=0.2 2023-11-21 17:31:31,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1594913.3333333333, ans=0.2 2023-11-21 17:31:43,347 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10800, loss[loss=0.06288, simple_loss=0.07699, pruned_loss=0.0155, audio_tagging_loss=0.008888, over 14990.00 frames. ], tot_loss[loss=0.07372, simple_loss=0.09615, pruned_loss=0.01627, audio_tagging_loss=0.009382, over 3050956.57 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 32.0 2023-11-21 17:31:44,696 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239250 2023-11-21 17:32:21,230 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.353e+01 8.126e+01 8.721e+01 9.296e+01 1.155e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-21 17:32:26,990 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.77 vs. limit=15.0 2023-11-21 17:32:35,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1595246.6666666667, ans=0.125 2023-11-21 17:32:36,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.10 vs. limit=12.0 2023-11-21 17:32:48,043 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10850, loss[loss=0.08335, simple_loss=0.1212, pruned_loss=0.0162, audio_tagging_loss=0.006562, over 17574.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.0963, pruned_loss=0.01626, audio_tagging_loss=0.009443, over 3055349.14 frames. ], batch size: 61, lr: 3.39e-03, grad_scale: 32.0 2023-11-21 17:32:49,334 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239300 2023-11-21 17:32:51,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1595313.3333333333, ans=0.1 2023-11-21 17:33:00,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1595380.0, ans=0.125 2023-11-21 17:33:13,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1595446.6666666667, ans=0.0 2023-11-21 17:33:15,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1595446.6666666667, ans=0.0 2023-11-21 17:33:26,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1595513.3333333333, ans=0.125 2023-11-21 17:33:29,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1595513.3333333333, ans=0.125 2023-11-21 17:33:46,331 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:33:51,759 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10900, loss[loss=0.06207, simple_loss=0.08434, pruned_loss=0.01026, audio_tagging_loss=0.009637, over 15886.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09628, pruned_loss=0.01629, audio_tagging_loss=0.009557, over 3048323.22 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 32.0 2023-11-21 17:33:53,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239350 2023-11-21 17:33:58,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1595646.6666666667, ans=0.035 2023-11-21 17:34:00,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1595646.6666666667, ans=0.2 2023-11-21 17:34:03,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1595713.3333333333, ans=0.1 2023-11-21 17:34:10,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1595713.3333333333, ans=0.0 2023-11-21 17:34:17,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1595780.0, ans=0.0 2023-11-21 17:34:28,930 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.501e+01 8.126e+01 8.588e+01 9.275e+01 1.184e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-21 17:34:30,873 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.21 vs. limit=15.0 2023-11-21 17:34:56,177 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 10950, loss[loss=0.06828, simple_loss=0.08853, pruned_loss=0.01301, audio_tagging_loss=0.01101, over 15657.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09661, pruned_loss=0.01629, audio_tagging_loss=0.009516, over 3046228.08 frames. ], batch size: 60, lr: 3.39e-03, grad_scale: 32.0 2023-11-21 17:34:57,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239400 2023-11-21 17:35:17,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1596046.6666666667, ans=0.1 2023-11-21 17:35:25,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1596113.3333333333, ans=0.0 2023-11-21 17:35:26,706 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.19 vs. limit=15.0 2023-11-21 17:35:28,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1596113.3333333333, ans=0.1 2023-11-21 17:35:42,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1596180.0, ans=0.1 2023-11-21 17:35:42,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1596180.0, ans=0.125 2023-11-21 17:35:47,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1596246.6666666667, ans=0.2 2023-11-21 17:35:56,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1596246.6666666667, ans=0.125 2023-11-21 17:36:00,163 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.84 vs. limit=10.0 2023-11-21 17:36:00,732 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11000, loss[loss=0.08485, simple_loss=0.1172, pruned_loss=0.01938, audio_tagging_loss=0.006887, over 15057.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.0967, pruned_loss=0.0164, audio_tagging_loss=0.009505, over 3043500.27 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:36:02,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239450 2023-11-21 17:36:05,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.14 vs. limit=15.0 2023-11-21 17:36:11,872 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:36:22,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.67 vs. limit=22.5 2023-11-21 17:36:28,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1596446.6666666667, ans=0.0 2023-11-21 17:36:37,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1596446.6666666667, ans=0.125 2023-11-21 17:36:40,591 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.031e+01 8.533e+01 9.347e+01 2.206e+02, threshold=1.707e+02, percent-clipped=1.0 2023-11-21 17:37:01,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1596580.0, ans=0.1 2023-11-21 17:37:06,724 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11050, loss[loss=0.05998, simple_loss=0.0734, pruned_loss=0.01043, audio_tagging_loss=0.01285, over 14351.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09623, pruned_loss=0.01644, audio_tagging_loss=0.009591, over 3049882.19 frames. ], batch size: 54, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:37:08,079 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239500 2023-11-21 17:37:10,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1596646.6666666667, ans=0.2 2023-11-21 17:37:27,762 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.65 vs. limit=10.0 2023-11-21 17:37:59,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1596913.3333333333, ans=0.125 2023-11-21 17:38:11,231 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11100, loss[loss=0.07534, simple_loss=0.09212, pruned_loss=0.02025, audio_tagging_loss=0.009035, over 15095.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09586, pruned_loss=0.01627, audio_tagging_loss=0.009631, over 3049227.80 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:38:12,537 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239550 2023-11-21 17:38:28,024 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 17:38:49,492 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.177e+01 8.722e+01 9.312e+01 1.121e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-21 17:38:49,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1597180.0, ans=0.1 2023-11-21 17:39:06,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1597246.6666666667, ans=0.125 2023-11-21 17:39:10,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1597246.6666666667, ans=0.0 2023-11-21 17:39:14,654 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11150, loss[loss=0.08144, simple_loss=0.1101, pruned_loss=0.01712, audio_tagging_loss=0.009251, over 14545.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09594, pruned_loss=0.01627, audio_tagging_loss=0.009671, over 3053890.30 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:39:15,942 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239600 2023-11-21 17:39:35,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1597380.0, ans=0.125 2023-11-21 17:39:40,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1597446.6666666667, ans=0.0 2023-11-21 17:39:45,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1597446.6666666667, ans=0.0 2023-11-21 17:40:02,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1597513.3333333333, ans=0.125 2023-11-21 17:40:10,152 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.92 vs. limit=15.0 2023-11-21 17:40:19,209 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11200, loss[loss=0.06761, simple_loss=0.08553, pruned_loss=0.01371, audio_tagging_loss=0.01114, over 16325.00 frames. ], tot_loss[loss=0.07363, simple_loss=0.09545, pruned_loss=0.01616, audio_tagging_loss=0.00975, over 3050422.17 frames. ], batch size: 62, lr: 3.39e-03, grad_scale: 32.0 2023-11-21 17:40:19,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1597646.6666666667, ans=0.125 2023-11-21 17:40:20,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239650 2023-11-21 17:40:58,062 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.528e+01 8.115e+01 8.678e+01 9.817e+01 1.232e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-21 17:41:04,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1597846.6666666667, ans=0.125 2023-11-21 17:41:17,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1597913.3333333333, ans=0.125 2023-11-21 17:41:23,464 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11250, loss[loss=0.07417, simple_loss=0.09297, pruned_loss=0.01441, audio_tagging_loss=0.01328, over 16429.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09449, pruned_loss=0.01602, audio_tagging_loss=0.009798, over 3054379.44 frames. ], batch size: 60, lr: 3.39e-03, grad_scale: 32.0 2023-11-21 17:41:24,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239700 2023-11-21 17:41:47,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1598046.6666666667, ans=0.125 2023-11-21 17:41:48,747 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.74 vs. limit=15.0 2023-11-21 17:42:24,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1598246.6666666667, ans=0.125 2023-11-21 17:42:27,198 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11300, loss[loss=0.05172, simple_loss=0.06576, pruned_loss=0.009922, audio_tagging_loss=0.008925, over 15128.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09502, pruned_loss=0.01607, audio_tagging_loss=0.009657, over 3054193.12 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:42:27,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1598313.3333333333, ans=0.05 2023-11-21 17:42:28,508 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239750 2023-11-21 17:42:41,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1598380.0, ans=0.125 2023-11-21 17:42:51,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1598446.6666666667, ans=0.0 2023-11-21 17:42:56,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1598446.6666666667, ans=0.125 2023-11-21 17:43:01,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1598446.6666666667, ans=0.0 2023-11-21 17:43:06,114 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.360e+01 8.263e+01 8.887e+01 9.511e+01 1.252e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-21 17:43:14,974 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.94 vs. limit=15.0 2023-11-21 17:43:19,764 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.21 vs. limit=12.0 2023-11-21 17:43:22,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.86 vs. limit=22.5 2023-11-21 17:43:30,683 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11350, loss[loss=0.0588, simple_loss=0.07221, pruned_loss=0.0128, audio_tagging_loss=0.009895, over 14544.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09523, pruned_loss=0.01618, audio_tagging_loss=0.009616, over 3050424.62 frames. ], batch size: 54, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:43:31,985 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239800 2023-11-21 17:43:55,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1598780.0, ans=0.125 2023-11-21 17:44:01,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1598780.0, ans=0.125 2023-11-21 17:44:22,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1598913.3333333333, ans=0.125 2023-11-21 17:44:27,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1598913.3333333333, ans=0.0 2023-11-21 17:44:33,777 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11400, loss[loss=0.07698, simple_loss=0.1009, pruned_loss=0.01704, audio_tagging_loss=0.009499, over 15843.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09496, pruned_loss=0.01609, audio_tagging_loss=0.009495, over 3055029.37 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:44:35,084 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239850 2023-11-21 17:44:52,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1599046.6666666667, ans=0.2 2023-11-21 17:44:59,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1599113.3333333333, ans=0.2 2023-11-21 17:45:11,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.35 vs. limit=15.0 2023-11-21 17:45:12,697 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.120e+01 8.500e+01 9.528e+01 1.320e+02, threshold=1.700e+02, percent-clipped=0.0 2023-11-21 17:45:29,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1599246.6666666667, ans=0.2 2023-11-21 17:45:37,182 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11450, loss[loss=0.07107, simple_loss=0.09539, pruned_loss=0.01456, audio_tagging_loss=0.008806, over 16184.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09602, pruned_loss=0.01643, audio_tagging_loss=0.009329, over 3048334.24 frames. ], batch size: 61, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:45:38,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239900 2023-11-21 17:45:56,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1599380.0, ans=0.125 2023-11-21 17:46:02,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1599446.6666666667, ans=0.125 2023-11-21 17:46:34,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1599580.0, ans=0.0 2023-11-21 17:46:37,571 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.98 vs. limit=15.0 2023-11-21 17:46:40,479 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11500, loss[loss=0.09347, simple_loss=0.1249, pruned_loss=0.0218, audio_tagging_loss=0.009216, over 15707.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09673, pruned_loss=0.01646, audio_tagging_loss=0.009262, over 3053662.38 frames. ], batch size: 61, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:46:41,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 239950 2023-11-21 17:46:44,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1599646.6666666667, ans=0.125 2023-11-21 17:47:10,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1599780.0, ans=0.125 2023-11-21 17:47:16,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.32 vs. limit=12.0 2023-11-21 17:47:19,134 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 7.879e+01 8.671e+01 9.205e+01 1.101e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-21 17:47:43,793 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11550, loss[loss=0.08574, simple_loss=0.1206, pruned_loss=0.01756, audio_tagging_loss=0.007858, over 15183.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09643, pruned_loss=0.01637, audio_tagging_loss=0.00941, over 3050140.93 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0 2023-11-21 17:47:44,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.73 vs. limit=15.0 2023-11-21 17:47:45,115 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240000 2023-11-21 17:48:13,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1600113.3333333333, ans=0.125 2023-11-21 17:48:15,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1600113.3333333333, ans=0.0 2023-11-21 17:48:24,240 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 17:48:27,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.53 vs. limit=15.0 2023-11-21 17:48:37,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1600246.6666666667, ans=0.125 2023-11-21 17:48:50,301 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11600, loss[loss=0.05946, simple_loss=0.07174, pruned_loss=0.01364, audio_tagging_loss=0.009949, over 16900.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09634, pruned_loss=0.01654, audio_tagging_loss=0.009384, over 3052330.73 frames. ], batch size: 68, lr: 3.38e-03, grad_scale: 32.0 2023-11-21 17:48:51,692 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240050 2023-11-21 17:49:01,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1600313.3333333333, ans=0.125 2023-11-21 17:49:14,155 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0 2023-11-21 17:49:14,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1600446.6666666667, ans=0.0 2023-11-21 17:49:16,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1600446.6666666667, ans=0.125 2023-11-21 17:49:29,609 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.173e+01 8.910e+01 9.625e+01 1.220e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-21 17:49:54,074 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11650, loss[loss=0.08156, simple_loss=0.1033, pruned_loss=0.01901, audio_tagging_loss=0.0109, over 15510.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.09604, pruned_loss=0.01642, audio_tagging_loss=0.009367, over 3052006.38 frames. ], batch size: 58, lr: 3.38e-03, grad_scale: 32.0 2023-11-21 17:49:55,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240100 2023-11-21 17:50:16,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1600713.3333333333, ans=0.0 2023-11-21 17:50:33,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1600846.6666666667, ans=0.2 2023-11-21 17:50:37,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.54 vs. limit=12.0 2023-11-21 17:50:50,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1600913.3333333333, ans=0.0 2023-11-21 17:50:57,979 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11700, loss[loss=0.08083, simple_loss=0.1031, pruned_loss=0.01854, audio_tagging_loss=0.01074, over 14994.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09581, pruned_loss=0.01635, audio_tagging_loss=0.009483, over 3049218.94 frames. ], batch size: 57, lr: 3.38e-03, grad_scale: 16.0 2023-11-21 17:50:59,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240150 2023-11-21 17:50:59,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1600980.0, ans=0.125 2023-11-21 17:50:59,999 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.01 vs. limit=22.5 2023-11-21 17:51:10,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1601046.6666666667, ans=0.125 2023-11-21 17:51:22,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1601113.3333333333, ans=0.125 2023-11-21 17:51:28,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1601113.3333333333, ans=0.125 2023-11-21 17:51:38,207 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.589e+01 8.116e+01 8.836e+01 9.594e+01 1.189e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 17:52:00,597 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11750, loss[loss=0.08989, simple_loss=0.116, pruned_loss=0.02334, audio_tagging_loss=0.008571, over 16070.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09639, pruned_loss=0.01659, audio_tagging_loss=0.009531, over 3050368.75 frames. ], batch size: 58, lr: 3.38e-03, grad_scale: 16.0 2023-11-21 17:52:01,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240200 2023-11-21 17:52:10,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1601313.3333333333, ans=0.125 2023-11-21 17:52:28,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1601446.6666666667, ans=0.2 2023-11-21 17:52:44,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1601513.3333333333, ans=0.2 2023-11-21 17:52:58,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.08 vs. limit=22.5 2023-11-21 17:53:03,893 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11800, loss[loss=0.06654, simple_loss=0.08175, pruned_loss=0.01482, audio_tagging_loss=0.01084, over 14486.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09575, pruned_loss=0.01638, audio_tagging_loss=0.009604, over 3048322.95 frames. ], batch size: 55, lr: 3.38e-03, grad_scale: 16.0 2023-11-21 17:53:05,215 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240250 2023-11-21 17:53:06,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1601646.6666666667, ans=0.125 2023-11-21 17:53:21,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1601713.3333333333, ans=0.125 2023-11-21 17:53:24,145 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.34 vs. limit=10.0 2023-11-21 17:53:38,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1601780.0, ans=0.0 2023-11-21 17:53:41,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1601846.6666666667, ans=0.0 2023-11-21 17:53:44,066 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.430e+01 8.055e+01 8.902e+01 9.602e+01 1.321e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-21 17:53:51,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1601846.6666666667, ans=0.1 2023-11-21 17:54:07,644 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11850, loss[loss=0.07112, simple_loss=0.09442, pruned_loss=0.01487, audio_tagging_loss=0.009046, over 14970.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09549, pruned_loss=0.01629, audio_tagging_loss=0.009691, over 3053098.60 frames. ], batch size: 56, lr: 3.38e-03, grad_scale: 16.0 2023-11-21 17:54:08,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240300 2023-11-21 17:54:30,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1602046.6666666667, ans=0.125 2023-11-21 17:54:37,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1602113.3333333333, ans=0.0 2023-11-21 17:54:48,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1602180.0, ans=0.0 2023-11-21 17:55:09,219 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.33 vs. limit=15.0 2023-11-21 17:55:11,003 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11900, loss[loss=0.07642, simple_loss=0.101, pruned_loss=0.01975, audio_tagging_loss=0.006168, over 15941.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09539, pruned_loss=0.01618, audio_tagging_loss=0.009729, over 3057163.62 frames. ], batch size: 59, lr: 3.38e-03, grad_scale: 16.0 2023-11-21 17:55:12,289 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240350 2023-11-21 17:55:29,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1602380.0, ans=0.125 2023-11-21 17:55:31,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1602380.0, ans=0.0 2023-11-21 17:55:35,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1602446.6666666667, ans=0.1 2023-11-21 17:55:37,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1602446.6666666667, ans=0.0 2023-11-21 17:55:39,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1602446.6666666667, ans=0.125 2023-11-21 17:55:44,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1602446.6666666667, ans=0.2 2023-11-21 17:55:52,201 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 7.826e+01 8.536e+01 9.227e+01 1.162e+02, threshold=1.707e+02, percent-clipped=0.0 2023-11-21 17:56:07,534 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=15.0 2023-11-21 17:56:14,814 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 11950, loss[loss=0.07093, simple_loss=0.09214, pruned_loss=0.01555, audio_tagging_loss=0.009311, over 15823.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09523, pruned_loss=0.01614, audio_tagging_loss=0.009831, over 3055959.73 frames. ], batch size: 57, lr: 3.38e-03, grad_scale: 16.0 2023-11-21 17:56:16,855 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240400 2023-11-21 17:56:19,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1602646.6666666667, ans=0.2 2023-11-21 17:56:29,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1602713.3333333333, ans=0.125 2023-11-21 17:56:33,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1602713.3333333333, ans=0.0 2023-11-21 17:56:38,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1602713.3333333333, ans=0.0 2023-11-21 17:56:51,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.78 vs. limit=15.0 2023-11-21 17:56:52,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1602846.6666666667, ans=0.125 2023-11-21 17:57:16,246 INFO [train_asr.py:1221] (3/4) Epoch 20, batch 12000, loss[loss=0.07933, simple_loss=0.1019, pruned_loss=0.01868, audio_tagging_loss=0.009719, over 15102.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09484, pruned_loss=0.01609, audio_tagging_loss=0.009914, over 3048398.95 frames. ], batch size: 57, lr: 3.38e-03, grad_scale: 32.0 2023-11-21 17:57:16,246 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 17:57:57,972 INFO [train_asr.py:1253] (3/4) Epoch 20, validation: loss=0.06015, simple_loss=0.05214, pruned_loss=0.005246, audio_tagging_loss=0.02884, over 4681554.00 frames. 2023-11-21 17:57:57,973 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 17:57:59,202 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240450 2023-11-21 17:58:05,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1602980.0, ans=0.1 2023-11-21 17:58:20,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1603113.3333333333, ans=0.1 2023-11-21 17:58:58,660 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 0, loss[loss=0.09891, simple_loss=0.12, pruned_loss=0.01866, audio_tagging_loss=0.02025, over 14801.00 frames. ], tot_loss[loss=0.09891, simple_loss=0.12, pruned_loss=0.01866, audio_tagging_loss=0.02025, over 14801.00 frames. ], batch size: 55, lr: 3.30e-03, grad_scale: 32.0 2023-11-21 17:58:58,661 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 17:59:34,257 INFO [train_asr.py:1253] (3/4) Epoch 21, validation: loss=0.05942, simple_loss=0.05208, pruned_loss=0.00519, audio_tagging_loss=0.02819, over 4681554.00 frames. 2023-11-21 17:59:34,258 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 17:59:34,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1603133.3333333333, ans=0.125 2023-11-21 17:59:36,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.62 vs. limit=15.0 2023-11-21 17:59:46,187 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.163e+01 8.736e+01 1.018e+02 1.443e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 17:59:57,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1603200.0, ans=0.1 2023-11-21 18:00:03,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1603266.6666666667, ans=0.5 2023-11-21 18:00:10,980 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240500 2023-11-21 18:00:15,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1603333.3333333333, ans=0.1 2023-11-21 18:00:16,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=1603333.3333333333, ans=10.0 2023-11-21 18:00:27,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.96 vs. limit=15.0 2023-11-21 18:00:28,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1603400.0, ans=0.125 2023-11-21 18:00:30,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1603400.0, ans=0.1 2023-11-21 18:00:39,134 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 50, loss[loss=0.07965, simple_loss=0.07906, pruned_loss=0.02057, audio_tagging_loss=0.01955, over 14883.00 frames. ], tot_loss[loss=0.08243, simple_loss=0.0967, pruned_loss=0.01576, audio_tagging_loss=0.01832, over 692466.27 frames. ], batch size: 58, lr: 3.30e-03, grad_scale: 32.0 2023-11-21 18:00:41,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1603466.6666666667, ans=0.0 2023-11-21 18:00:49,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1603466.6666666667, ans=0.1 2023-11-21 18:01:01,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1603533.3333333333, ans=0.125 2023-11-21 18:01:06,769 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:01:15,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240550 2023-11-21 18:01:43,928 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 100, loss[loss=0.08027, simple_loss=0.1014, pruned_loss=0.01513, audio_tagging_loss=0.01442, over 15454.00 frames. ], tot_loss[loss=0.08309, simple_loss=0.09798, pruned_loss=0.01662, audio_tagging_loss=0.01748, over 1216830.38 frames. ], batch size: 57, lr: 3.30e-03, grad_scale: 32.0 2023-11-21 18:01:54,764 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.709e+01 8.663e+01 9.262e+01 1.008e+02 1.245e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-21 18:02:02,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1603866.6666666667, ans=0.1 2023-11-21 18:02:09,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1603933.3333333333, ans=0.0 2023-11-21 18:02:11,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1603933.3333333333, ans=0.1 2023-11-21 18:02:19,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240600 2023-11-21 18:02:26,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1604000.0, ans=0.5 2023-11-21 18:02:33,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1604000.0, ans=0.125 2023-11-21 18:02:47,887 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 150, loss[loss=0.09617, simple_loss=0.1112, pruned_loss=0.02893, audio_tagging_loss=0.01164, over 15768.00 frames. ], tot_loss[loss=0.08009, simple_loss=0.09638, pruned_loss=0.01606, audio_tagging_loss=0.01584, over 1615434.06 frames. ], batch size: 60, lr: 3.30e-03, grad_scale: 16.0 2023-11-21 18:02:55,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1604133.3333333333, ans=0.125 2023-11-21 18:02:55,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1604133.3333333333, ans=0.0 2023-11-21 18:02:57,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1604133.3333333333, ans=0.125 2023-11-21 18:03:14,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1604266.6666666667, ans=0.125 2023-11-21 18:03:14,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.28 vs. limit=22.5 2023-11-21 18:03:17,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1604266.6666666667, ans=0.0 2023-11-21 18:03:21,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1604266.6666666667, ans=0.125 2023-11-21 18:03:24,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240650 2023-11-21 18:03:32,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1604333.3333333333, ans=0.0 2023-11-21 18:03:51,994 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 200, loss[loss=0.09567, simple_loss=0.134, pruned_loss=0.01945, audio_tagging_loss=0.009235, over 14881.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.0968, pruned_loss=0.01643, audio_tagging_loss=0.01379, over 1930568.46 frames. ], batch size: 53, lr: 3.30e-03, grad_scale: 16.0 2023-11-21 18:03:59,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=1604466.6666666667, ans=0.2 2023-11-21 18:04:03,960 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.168e+01 8.831e+01 9.644e+01 1.295e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-21 18:04:07,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1604533.3333333333, ans=0.125 2023-11-21 18:04:27,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240700 2023-11-21 18:04:29,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1604666.6666666667, ans=0.0 2023-11-21 18:04:30,290 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0 2023-11-21 18:04:44,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1604733.3333333333, ans=0.125 2023-11-21 18:04:55,609 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 250, loss[loss=0.07594, simple_loss=0.1033, pruned_loss=0.01676, audio_tagging_loss=0.007508, over 14855.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09808, pruned_loss=0.01673, audio_tagging_loss=0.01251, over 2180781.86 frames. ], batch size: 54, lr: 3.30e-03, grad_scale: 8.0 2023-11-21 18:05:00,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.94 vs. limit=10.0 2023-11-21 18:05:05,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1604800.0, ans=0.125 2023-11-21 18:05:19,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1604933.3333333333, ans=0.125 2023-11-21 18:05:31,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240750 2023-11-21 18:05:55,139 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=12.0 2023-11-21 18:05:55,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.56 vs. limit=15.0 2023-11-21 18:05:59,423 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 300, loss[loss=0.08853, simple_loss=0.1166, pruned_loss=0.01944, audio_tagging_loss=0.01079, over 16043.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.0974, pruned_loss=0.01669, audio_tagging_loss=0.01166, over 2372960.89 frames. ], batch size: 59, lr: 3.30e-03, grad_scale: 8.0 2023-11-21 18:06:10,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1605133.3333333333, ans=0.0 2023-11-21 18:06:14,090 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.194e+01 8.148e+01 8.758e+01 9.398e+01 1.354e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-21 18:06:14,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1605200.0, ans=0.125 2023-11-21 18:06:21,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1605200.0, ans=0.125 2023-11-21 18:06:35,642 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240800 2023-11-21 18:06:45,685 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0 2023-11-21 18:06:46,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1605333.3333333333, ans=0.2 2023-11-21 18:06:47,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1605333.3333333333, ans=0.125 2023-11-21 18:07:04,338 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 350, loss[loss=0.08412, simple_loss=0.1141, pruned_loss=0.01941, audio_tagging_loss=0.007642, over 15952.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.09736, pruned_loss=0.01673, audio_tagging_loss=0.01103, over 2525122.09 frames. ], batch size: 58, lr: 3.30e-03, grad_scale: 8.0 2023-11-21 18:07:10,055 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.32 vs. limit=15.0 2023-11-21 18:07:27,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1605533.3333333333, ans=0.125 2023-11-21 18:07:28,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1605600.0, ans=0.125 2023-11-21 18:07:29,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1605600.0, ans=0.125 2023-11-21 18:07:39,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240850 2023-11-21 18:07:49,557 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.23 vs. limit=15.0 2023-11-21 18:07:52,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1605666.6666666667, ans=0.0 2023-11-21 18:07:58,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1605733.3333333333, ans=0.125 2023-11-21 18:08:08,090 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 400, loss[loss=0.06093, simple_loss=0.0805, pruned_loss=0.0109, audio_tagging_loss=0.009779, over 14424.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09779, pruned_loss=0.01655, audio_tagging_loss=0.01059, over 2647325.50 frames. ], batch size: 53, lr: 3.30e-03, grad_scale: 16.0 2023-11-21 18:08:10,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1605800.0, ans=0.0 2023-11-21 18:08:21,903 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:08:22,786 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.012e+01 8.744e+01 9.405e+01 1.302e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-21 18:08:41,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1605933.3333333333, ans=0.125 2023-11-21 18:08:44,717 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240900 2023-11-21 18:08:44,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1605933.3333333333, ans=0.0 2023-11-21 18:08:48,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1606000.0, ans=0.2 2023-11-21 18:08:53,997 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.39 vs. limit=6.0 2023-11-21 18:09:02,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1606066.6666666667, ans=0.125 2023-11-21 18:09:07,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1606066.6666666667, ans=0.05 2023-11-21 18:09:12,849 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 450, loss[loss=0.06445, simple_loss=0.09263, pruned_loss=0.009673, audio_tagging_loss=0.008466, over 14191.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09732, pruned_loss=0.01647, audio_tagging_loss=0.01025, over 2727953.58 frames. ], batch size: 51, lr: 3.30e-03, grad_scale: 16.0 2023-11-21 18:09:34,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1606200.0, ans=0.0 2023-11-21 18:09:40,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1606266.6666666667, ans=0.1 2023-11-21 18:09:40,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1606266.6666666667, ans=0.1 2023-11-21 18:09:42,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1606266.6666666667, ans=0.125 2023-11-21 18:09:49,217 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 240950 2023-11-21 18:09:51,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1606333.3333333333, ans=0.04949747468305833 2023-11-21 18:10:08,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1606400.0, ans=0.125 2023-11-21 18:10:17,047 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 500, loss[loss=0.07179, simple_loss=0.09121, pruned_loss=0.01695, audio_tagging_loss=0.009228, over 14437.00 frames. ], tot_loss[loss=0.07545, simple_loss=0.09754, pruned_loss=0.01668, audio_tagging_loss=0.01001, over 2794792.59 frames. ], batch size: 55, lr: 3.30e-03, grad_scale: 16.0 2023-11-21 18:10:31,507 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.140e+01 8.109e+01 8.628e+01 9.380e+01 1.335e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-21 18:10:43,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1606600.0, ans=0.1 2023-11-21 18:10:49,521 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:10:53,675 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241000 2023-11-21 18:10:55,659 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.28 vs. limit=22.5 2023-11-21 18:10:59,620 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.34 vs. limit=15.0 2023-11-21 18:11:05,228 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0 2023-11-21 18:11:12,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1606733.3333333333, ans=0.1 2023-11-21 18:11:22,076 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 550, loss[loss=0.05882, simple_loss=0.08368, pruned_loss=0.009812, audio_tagging_loss=0.007171, over 14685.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09695, pruned_loss=0.01649, audio_tagging_loss=0.009931, over 2853296.28 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:11:24,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1606800.0, ans=0.1 2023-11-21 18:11:42,541 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.43 vs. limit=15.0 2023-11-21 18:11:57,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.59 vs. limit=12.0 2023-11-21 18:11:58,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241050 2023-11-21 18:12:12,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.49 vs. limit=15.0 2023-11-21 18:12:21,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1607066.6666666667, ans=0.1 2023-11-21 18:12:22,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1607066.6666666667, ans=0.1 2023-11-21 18:12:25,552 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 600, loss[loss=0.07995, simple_loss=0.1098, pruned_loss=0.01439, audio_tagging_loss=0.01064, over 14651.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.09625, pruned_loss=0.01627, audio_tagging_loss=0.009908, over 2895983.98 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:12:39,222 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.61 vs. limit=15.0 2023-11-21 18:12:39,616 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.804e+01 8.139e+01 8.699e+01 9.445e+01 1.159e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 18:12:43,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.48 vs. limit=15.0 2023-11-21 18:13:01,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241100 2023-11-21 18:13:03,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1607333.3333333333, ans=0.0 2023-11-21 18:13:12,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1607333.3333333333, ans=0.0 2023-11-21 18:13:20,276 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.32 vs. limit=15.0 2023-11-21 18:13:25,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1607400.0, ans=0.125 2023-11-21 18:13:30,504 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 650, loss[loss=0.09905, simple_loss=0.1358, pruned_loss=0.02401, audio_tagging_loss=0.007166, over 15935.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09724, pruned_loss=0.01633, audio_tagging_loss=0.009754, over 2929465.46 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:13:47,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1607533.3333333333, ans=0.125 2023-11-21 18:13:49,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1607533.3333333333, ans=0.0 2023-11-21 18:13:57,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1607600.0, ans=0.125 2023-11-21 18:14:05,727 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241150 2023-11-21 18:14:20,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.93 vs. limit=22.5 2023-11-21 18:14:24,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1607733.3333333333, ans=0.2 2023-11-21 18:14:34,264 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 700, loss[loss=0.05947, simple_loss=0.08175, pruned_loss=0.008673, audio_tagging_loss=0.009922, over 13987.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09706, pruned_loss=0.0162, audio_tagging_loss=0.009683, over 2951488.71 frames. ], batch size: 53, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:14:38,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.51 vs. limit=15.0 2023-11-21 18:14:48,424 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.012e+01 8.519e+01 9.265e+01 1.175e+02, threshold=1.704e+02, percent-clipped=0.0 2023-11-21 18:15:10,794 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241200 2023-11-21 18:15:19,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1608000.0, ans=0.0 2023-11-21 18:15:22,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.87 vs. limit=15.0 2023-11-21 18:15:25,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1608066.6666666667, ans=0.1 2023-11-21 18:15:29,518 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.20 vs. limit=15.0 2023-11-21 18:15:39,234 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 750, loss[loss=0.09155, simple_loss=0.109, pruned_loss=0.02358, audio_tagging_loss=0.01346, over 14074.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09747, pruned_loss=0.01633, audio_tagging_loss=0.009733, over 2975424.03 frames. ], batch size: 52, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:15:48,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1608133.3333333333, ans=0.1 2023-11-21 18:16:05,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1608266.6666666667, ans=0.125 2023-11-21 18:16:14,938 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241250 2023-11-21 18:16:42,243 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 800, loss[loss=0.07589, simple_loss=0.09234, pruned_loss=0.01734, audio_tagging_loss=0.01239, over 13786.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09718, pruned_loss=0.0164, audio_tagging_loss=0.009731, over 2990326.20 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 32.0 2023-11-21 18:16:57,474 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.159e+01 8.981e+01 9.751e+01 1.353e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-21 18:17:09,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1608600.0, ans=0.0 2023-11-21 18:17:18,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241300 2023-11-21 18:17:21,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1608666.6666666667, ans=10.0 2023-11-21 18:17:34,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1608733.3333333333, ans=0.125 2023-11-21 18:17:47,504 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 850, loss[loss=0.06671, simple_loss=0.08318, pruned_loss=0.0166, audio_tagging_loss=0.00852, over 15116.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09759, pruned_loss=0.01663, audio_tagging_loss=0.009793, over 3003438.22 frames. ], batch size: 61, lr: 3.29e-03, grad_scale: 32.0 2023-11-21 18:17:55,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1608800.0, ans=0.125 2023-11-21 18:18:05,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1608866.6666666667, ans=0.125 2023-11-21 18:18:06,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1608866.6666666667, ans=0.1 2023-11-21 18:18:17,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1608933.3333333333, ans=0.035 2023-11-21 18:18:18,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1608933.3333333333, ans=0.04949747468305833 2023-11-21 18:18:23,356 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241350 2023-11-21 18:18:28,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1609000.0, ans=0.125 2023-11-21 18:18:43,444 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.12 vs. limit=10.0 2023-11-21 18:18:52,216 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 900, loss[loss=0.08491, simple_loss=0.1129, pruned_loss=0.02015, audio_tagging_loss=0.008336, over 15599.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.0978, pruned_loss=0.0168, audio_tagging_loss=0.009759, over 3026337.07 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 32.0 2023-11-21 18:18:59,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1609133.3333333333, ans=0.125 2023-11-21 18:19:05,368 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.217e+01 8.787e+01 9.484e+01 1.241e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-21 18:19:18,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1609266.6666666667, ans=0.0 2023-11-21 18:19:27,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.84 vs. limit=15.0 2023-11-21 18:19:27,873 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241400 2023-11-21 18:19:37,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1609333.3333333333, ans=0.125 2023-11-21 18:19:38,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1609333.3333333333, ans=0.07 2023-11-21 18:19:40,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.65 vs. limit=15.0 2023-11-21 18:19:52,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1609400.0, ans=0.09899494936611666 2023-11-21 18:19:55,358 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 950, loss[loss=0.07011, simple_loss=0.08331, pruned_loss=0.0178, audio_tagging_loss=0.01066, over 15468.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09768, pruned_loss=0.01688, audio_tagging_loss=0.009678, over 3026873.14 frames. ], batch size: 60, lr: 3.29e-03, grad_scale: 32.0 2023-11-21 18:20:31,538 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241450 2023-11-21 18:20:42,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1609666.6666666667, ans=0.0 2023-11-21 18:20:53,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1609733.3333333333, ans=0.125 2023-11-21 18:21:00,406 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1000, loss[loss=0.07243, simple_loss=0.09259, pruned_loss=0.01669, audio_tagging_loss=0.009441, over 14331.00 frames. ], tot_loss[loss=0.07507, simple_loss=0.09772, pruned_loss=0.01678, audio_tagging_loss=0.009427, over 3032560.74 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:21:15,644 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.205e+01 8.818e+01 9.430e+01 1.136e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-21 18:21:26,816 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 18:21:35,609 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241500 2023-11-21 18:21:39,075 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0 2023-11-21 18:22:01,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1610066.6666666667, ans=0.025 2023-11-21 18:22:04,694 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1050, loss[loss=0.07233, simple_loss=0.09748, pruned_loss=0.01347, audio_tagging_loss=0.01011, over 14462.00 frames. ], tot_loss[loss=0.07423, simple_loss=0.09648, pruned_loss=0.0166, audio_tagging_loss=0.009388, over 3033244.42 frames. ], batch size: 53, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:22:05,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1610133.3333333333, ans=0.0 2023-11-21 18:22:17,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1610200.0, ans=0.125 2023-11-21 18:22:35,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1610266.6666666667, ans=0.125 2023-11-21 18:22:40,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241550 2023-11-21 18:22:43,691 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.60 vs. limit=10.0 2023-11-21 18:22:48,768 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.62 vs. limit=6.0 2023-11-21 18:22:51,591 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.17 vs. limit=12.0 2023-11-21 18:23:08,015 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1100, loss[loss=0.06667, simple_loss=0.08798, pruned_loss=0.01453, audio_tagging_loss=0.008154, over 15275.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09518, pruned_loss=0.01632, audio_tagging_loss=0.009339, over 3038695.06 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:23:10,530 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 18:23:10,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1610466.6666666667, ans=0.125 2023-11-21 18:23:12,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1610466.6666666667, ans=0.2 2023-11-21 18:23:23,983 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.562e+01 7.887e+01 8.672e+01 9.418e+01 1.186e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-21 18:23:25,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.44 vs. limit=15.0 2023-11-21 18:23:32,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1610533.3333333333, ans=0.07 2023-11-21 18:23:34,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1610600.0, ans=0.125 2023-11-21 18:23:36,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1610600.0, ans=0.2 2023-11-21 18:23:39,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1610600.0, ans=0.2 2023-11-21 18:23:44,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241600 2023-11-21 18:24:08,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1610733.3333333333, ans=0.0 2023-11-21 18:24:09,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.01 vs. limit=15.0 2023-11-21 18:24:12,789 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1150, loss[loss=0.07888, simple_loss=0.1047, pruned_loss=0.0179, audio_tagging_loss=0.008615, over 14726.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09608, pruned_loss=0.01646, audio_tagging_loss=0.009248, over 3037699.88 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:24:41,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1610933.3333333333, ans=0.125 2023-11-21 18:24:48,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241650 2023-11-21 18:24:59,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1611000.0, ans=0.2 2023-11-21 18:25:00,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1611000.0, ans=0.05 2023-11-21 18:25:02,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1611066.6666666667, ans=0.5 2023-11-21 18:25:05,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.16 vs. limit=10.0 2023-11-21 18:25:17,314 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1200, loss[loss=0.07342, simple_loss=0.09819, pruned_loss=0.01618, audio_tagging_loss=0.008151, over 16448.00 frames. ], tot_loss[loss=0.07497, simple_loss=0.09807, pruned_loss=0.0168, audio_tagging_loss=0.009132, over 3049048.02 frames. ], batch size: 61, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:25:33,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.048e+01 8.609e+01 9.236e+01 1.274e+02, threshold=1.722e+02, percent-clipped=0.0 2023-11-21 18:25:52,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241700 2023-11-21 18:25:53,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1611333.3333333333, ans=0.1 2023-11-21 18:26:20,654 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1250, loss[loss=0.05172, simple_loss=0.06594, pruned_loss=0.007661, audio_tagging_loss=0.0111, over 15838.00 frames. ], tot_loss[loss=0.07391, simple_loss=0.09655, pruned_loss=0.01639, audio_tagging_loss=0.009244, over 3050668.61 frames. ], batch size: 62, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:26:20,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1611466.6666666667, ans=0.1 2023-11-21 18:26:23,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.72 vs. limit=15.0 2023-11-21 18:26:24,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1611466.6666666667, ans=0.2 2023-11-21 18:26:29,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1611466.6666666667, ans=0.125 2023-11-21 18:26:43,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1611533.3333333333, ans=0.125 2023-11-21 18:26:54,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=1611600.0, ans=0.02 2023-11-21 18:26:57,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241750 2023-11-21 18:27:08,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1611666.6666666667, ans=0.2 2023-11-21 18:27:25,084 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1300, loss[loss=0.05643, simple_loss=0.07643, pruned_loss=0.01131, audio_tagging_loss=0.00691, over 14854.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.0963, pruned_loss=0.01641, audio_tagging_loss=0.009298, over 3050630.66 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:27:41,457 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.105e+01 8.749e+01 9.378e+01 1.268e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 18:27:44,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1611866.6666666667, ans=0.09899494936611666 2023-11-21 18:27:50,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1611933.3333333333, ans=0.125 2023-11-21 18:27:55,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1611933.3333333333, ans=0.125 2023-11-21 18:28:00,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241800 2023-11-21 18:28:10,937 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.39 vs. limit=12.0 2023-11-21 18:28:28,417 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1350, loss[loss=0.07608, simple_loss=0.1106, pruned_loss=0.01363, audio_tagging_loss=0.007172, over 15106.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09616, pruned_loss=0.01632, audio_tagging_loss=0.009253, over 3046723.96 frames. ], batch size: 53, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:28:33,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=1612133.3333333333, ans=0.02 2023-11-21 18:28:37,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1612133.3333333333, ans=0.07 2023-11-21 18:28:39,561 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:28:42,314 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=12.0 2023-11-21 18:28:46,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1612200.0, ans=0.0 2023-11-21 18:29:05,071 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241850 2023-11-21 18:29:11,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.50 vs. limit=15.0 2023-11-21 18:29:14,747 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 18:29:15,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1612333.3333333333, ans=0.2 2023-11-21 18:29:15,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1612333.3333333333, ans=15.0 2023-11-21 18:29:31,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1612466.6666666667, ans=10.0 2023-11-21 18:29:32,995 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1400, loss[loss=0.07085, simple_loss=0.08833, pruned_loss=0.01534, audio_tagging_loss=0.01134, over 14646.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09656, pruned_loss=0.0165, audio_tagging_loss=0.009321, over 3052832.93 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:29:34,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1612466.6666666667, ans=0.0 2023-11-21 18:29:50,233 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.505e+01 9.180e+01 9.942e+01 1.333e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-21 18:29:56,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1612533.3333333333, ans=0.2 2023-11-21 18:30:09,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241900 2023-11-21 18:30:12,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1612666.6666666667, ans=10.0 2023-11-21 18:30:13,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1612666.6666666667, ans=0.1 2023-11-21 18:30:37,672 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1450, loss[loss=0.08556, simple_loss=0.1205, pruned_loss=0.01924, audio_tagging_loss=0.006066, over 14769.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09647, pruned_loss=0.01655, audio_tagging_loss=0.009428, over 3054148.62 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:30:46,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1612800.0, ans=0.125 2023-11-21 18:30:54,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1612866.6666666667, ans=0.125 2023-11-21 18:31:13,406 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 241950 2023-11-21 18:31:40,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1613133.3333333333, ans=0.95 2023-11-21 18:31:41,557 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1500, loss[loss=0.06536, simple_loss=0.07911, pruned_loss=0.01426, audio_tagging_loss=0.01154, over 13962.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09739, pruned_loss=0.0169, audio_tagging_loss=0.009448, over 3043604.21 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 8.0 2023-11-21 18:31:42,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1613133.3333333333, ans=0.2 2023-11-21 18:31:59,580 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.238e+01 7.980e+01 8.647e+01 9.591e+01 1.272e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-21 18:32:09,817 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:32:11,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1613266.6666666667, ans=0.125 2023-11-21 18:32:11,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1613266.6666666667, ans=0.125 2023-11-21 18:32:18,000 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242000 2023-11-21 18:32:28,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1613333.3333333333, ans=0.0 2023-11-21 18:32:46,303 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1550, loss[loss=0.07745, simple_loss=0.1003, pruned_loss=0.01777, audio_tagging_loss=0.00951, over 14668.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09733, pruned_loss=0.01693, audio_tagging_loss=0.009646, over 3043123.05 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 8.0 2023-11-21 18:32:52,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1613466.6666666667, ans=0.125 2023-11-21 18:32:52,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1613466.6666666667, ans=0.0 2023-11-21 18:32:58,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1613533.3333333333, ans=0.125 2023-11-21 18:33:18,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.64 vs. limit=10.0 2023-11-21 18:33:22,005 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242050 2023-11-21 18:33:42,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1613733.3333333333, ans=0.2 2023-11-21 18:33:50,194 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1600, loss[loss=0.06407, simple_loss=0.08341, pruned_loss=0.01036, audio_tagging_loss=0.012, over 15196.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09818, pruned_loss=0.01693, audio_tagging_loss=0.009639, over 3043981.94 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:34:07,795 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.028e+01 8.717e+01 9.586e+01 1.995e+02, threshold=1.743e+02, percent-clipped=1.0 2023-11-21 18:34:26,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242100 2023-11-21 18:34:54,149 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1650, loss[loss=0.05537, simple_loss=0.06119, pruned_loss=0.01465, audio_tagging_loss=0.01012, over 14488.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09717, pruned_loss=0.0168, audio_tagging_loss=0.009704, over 3044766.97 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:35:10,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0 2023-11-21 18:35:12,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1614200.0, ans=0.2 2023-11-21 18:35:31,027 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242150 2023-11-21 18:35:43,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1614333.3333333333, ans=0.1 2023-11-21 18:35:57,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1614466.6666666667, ans=0.0 2023-11-21 18:35:58,535 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1700, loss[loss=0.07077, simple_loss=0.09195, pruned_loss=0.01503, audio_tagging_loss=0.009768, over 15274.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09628, pruned_loss=0.01648, audio_tagging_loss=0.009813, over 3050870.73 frames. ], batch size: 59, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:36:08,136 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.14 vs. limit=15.0 2023-11-21 18:36:10,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1614533.3333333333, ans=0.1 2023-11-21 18:36:10,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.76 vs. limit=15.0 2023-11-21 18:36:16,621 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.384e+01 8.955e+01 9.684e+01 1.332e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-21 18:36:26,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1614600.0, ans=0.0 2023-11-21 18:36:33,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1614600.0, ans=0.125 2023-11-21 18:36:34,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242200 2023-11-21 18:36:35,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1614666.6666666667, ans=0.125 2023-11-21 18:36:49,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1614733.3333333333, ans=0.125 2023-11-21 18:36:52,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1614733.3333333333, ans=0.125 2023-11-21 18:37:02,898 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1750, loss[loss=0.05374, simple_loss=0.06847, pruned_loss=0.01022, audio_tagging_loss=0.009285, over 13874.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09611, pruned_loss=0.01642, audio_tagging_loss=0.009869, over 3052512.39 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:37:24,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1614866.6666666667, ans=0.125 2023-11-21 18:37:38,629 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242250 2023-11-21 18:37:38,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1614933.3333333333, ans=0.125 2023-11-21 18:37:44,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1615000.0, ans=0.2 2023-11-21 18:37:45,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1615000.0, ans=0.125 2023-11-21 18:37:57,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1615066.6666666667, ans=0.125 2023-11-21 18:38:07,259 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1800, loss[loss=0.09538, simple_loss=0.1287, pruned_loss=0.02397, audio_tagging_loss=0.007062, over 15192.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09686, pruned_loss=0.0167, audio_tagging_loss=0.009666, over 3056311.32 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:38:16,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1615133.3333333333, ans=0.125 2023-11-21 18:38:25,233 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.166e+01 8.773e+01 9.532e+01 1.071e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 18:38:29,855 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.33 vs. limit=15.0 2023-11-21 18:38:42,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1615266.6666666667, ans=0.125 2023-11-21 18:38:43,030 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.97 vs. limit=22.5 2023-11-21 18:38:43,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242300 2023-11-21 18:38:45,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1615333.3333333333, ans=0.04949747468305833 2023-11-21 18:38:45,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1615333.3333333333, ans=0.07 2023-11-21 18:39:02,814 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.17 vs. limit=10.0 2023-11-21 18:39:11,201 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1850, loss[loss=0.05542, simple_loss=0.06462, pruned_loss=0.01194, audio_tagging_loss=0.01118, over 14423.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09606, pruned_loss=0.01648, audio_tagging_loss=0.009539, over 3057099.47 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:39:22,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1615533.3333333333, ans=0.1 2023-11-21 18:39:31,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.36 vs. limit=15.0 2023-11-21 18:39:44,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1615600.0, ans=10.0 2023-11-21 18:39:47,080 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242350 2023-11-21 18:40:14,383 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1900, loss[loss=0.07117, simple_loss=0.0922, pruned_loss=0.01478, audio_tagging_loss=0.0103, over 16145.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09486, pruned_loss=0.0163, audio_tagging_loss=0.00957, over 3055526.65 frames. ], batch size: 62, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:40:19,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1615800.0, ans=0.125 2023-11-21 18:40:29,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1615866.6666666667, ans=0.125 2023-11-21 18:40:33,151 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.159e+01 8.678e+01 9.357e+01 1.521e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-21 18:40:39,827 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.33 vs. limit=12.0 2023-11-21 18:40:42,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1615933.3333333333, ans=0.125 2023-11-21 18:40:50,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242400 2023-11-21 18:41:16,070 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.60 vs. limit=12.0 2023-11-21 18:41:19,185 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 1950, loss[loss=0.0807, simple_loss=0.107, pruned_loss=0.02017, audio_tagging_loss=0.007015, over 14780.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.095, pruned_loss=0.01623, audio_tagging_loss=0.009589, over 3059770.68 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0 2023-11-21 18:41:33,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1616200.0, ans=0.125 2023-11-21 18:41:44,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1616266.6666666667, ans=0.07 2023-11-21 18:41:45,209 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.26 vs. limit=15.0 2023-11-21 18:41:54,103 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242450 2023-11-21 18:42:15,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1616400.0, ans=0.125 2023-11-21 18:42:18,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1616400.0, ans=0.2 2023-11-21 18:42:23,240 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2000, loss[loss=0.0842, simple_loss=0.1272, pruned_loss=0.01611, audio_tagging_loss=0.004485, over 15569.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.09423, pruned_loss=0.01615, audio_tagging_loss=0.009613, over 3057438.97 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 32.0 2023-11-21 18:42:40,262 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 7.872e+01 8.647e+01 9.317e+01 2.008e+02, threshold=1.729e+02, percent-clipped=1.0 2023-11-21 18:42:41,166 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.79 vs. limit=22.5 2023-11-21 18:42:55,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1616600.0, ans=0.125 2023-11-21 18:42:58,693 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242500 2023-11-21 18:43:00,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1616666.6666666667, ans=0.0 2023-11-21 18:43:26,312 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2050, loss[loss=0.0739, simple_loss=0.1048, pruned_loss=0.01405, audio_tagging_loss=0.00744, over 15801.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09484, pruned_loss=0.01611, audio_tagging_loss=0.009521, over 3053530.05 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 32.0 2023-11-21 18:43:41,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1616866.6666666667, ans=0.125 2023-11-21 18:44:03,201 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242550 2023-11-21 18:44:08,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1617000.0, ans=0.125 2023-11-21 18:44:21,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=1617066.6666666667, ans=15.0 2023-11-21 18:44:31,775 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2100, loss[loss=0.0739, simple_loss=0.0989, pruned_loss=0.01647, audio_tagging_loss=0.007983, over 14845.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09516, pruned_loss=0.01619, audio_tagging_loss=0.009505, over 3052330.37 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:44:35,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.55 vs. limit=15.0 2023-11-21 18:44:50,751 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 8.232e+01 8.682e+01 9.296e+01 1.860e+02, threshold=1.736e+02, percent-clipped=1.0 2023-11-21 18:44:57,978 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.52 vs. limit=15.0 2023-11-21 18:45:00,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.94 vs. limit=15.0 2023-11-21 18:45:02,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1617266.6666666667, ans=0.125 2023-11-21 18:45:06,931 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242600 2023-11-21 18:45:36,200 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2150, loss[loss=0.06969, simple_loss=0.09414, pruned_loss=0.01384, audio_tagging_loss=0.008777, over 15032.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09491, pruned_loss=0.01599, audio_tagging_loss=0.009448, over 3046402.13 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:45:53,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1617533.3333333333, ans=0.1 2023-11-21 18:46:12,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242650 2023-11-21 18:46:14,888 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 18:46:17,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1617666.6666666667, ans=0.125 2023-11-21 18:46:31,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1617733.3333333333, ans=0.0 2023-11-21 18:46:39,506 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2200, loss[loss=0.0693, simple_loss=0.08481, pruned_loss=0.01518, audio_tagging_loss=0.01172, over 15460.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09564, pruned_loss=0.01625, audio_tagging_loss=0.009457, over 3042462.31 frames. ], batch size: 60, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:46:41,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1617800.0, ans=0.0 2023-11-21 18:46:44,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.69 vs. limit=15.0 2023-11-21 18:46:58,053 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:46:58,820 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.003e+01 8.660e+01 9.579e+01 1.643e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 18:47:05,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1617933.3333333333, ans=0.0 2023-11-21 18:47:07,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1617933.3333333333, ans=0.1 2023-11-21 18:47:10,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1617933.3333333333, ans=0.0 2023-11-21 18:47:15,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242700 2023-11-21 18:47:25,860 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=2.514e-03 2023-11-21 18:47:29,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1618066.6666666667, ans=6.0 2023-11-21 18:47:43,670 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2250, loss[loss=0.06199, simple_loss=0.08058, pruned_loss=0.01079, audio_tagging_loss=0.01091, over 14961.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09628, pruned_loss=0.01626, audio_tagging_loss=0.009401, over 3041162.44 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:47:56,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1618200.0, ans=0.1 2023-11-21 18:48:10,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1618266.6666666667, ans=0.2 2023-11-21 18:48:18,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242750 2023-11-21 18:48:24,982 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:48:26,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1618333.3333333333, ans=0.125 2023-11-21 18:48:43,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1618400.0, ans=10.0 2023-11-21 18:48:46,727 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.64 vs. limit=15.0 2023-11-21 18:48:47,270 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2300, loss[loss=0.07594, simple_loss=0.1009, pruned_loss=0.01737, audio_tagging_loss=0.008099, over 13763.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.0952, pruned_loss=0.01604, audio_tagging_loss=0.009494, over 3047058.94 frames. ], batch size: 54, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:48:51,165 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.72 vs. limit=15.0 2023-11-21 18:48:51,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1618466.6666666667, ans=0.125 2023-11-21 18:48:56,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1618466.6666666667, ans=0.07 2023-11-21 18:49:06,239 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.387e+01 7.925e+01 8.512e+01 9.086e+01 1.194e+02, threshold=1.702e+02, percent-clipped=0.0 2023-11-21 18:49:22,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1618600.0, ans=0.05 2023-11-21 18:49:23,573 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242800 2023-11-21 18:49:28,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1618666.6666666667, ans=0.125 2023-11-21 18:49:38,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1618733.3333333333, ans=0.1 2023-11-21 18:49:40,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1618733.3333333333, ans=0.0 2023-11-21 18:49:44,460 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 18:49:51,765 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2350, loss[loss=0.07187, simple_loss=0.09002, pruned_loss=0.01847, audio_tagging_loss=0.008392, over 14826.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09483, pruned_loss=0.01593, audio_tagging_loss=0.009548, over 3043708.54 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:49:54,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1618800.0, ans=0.2 2023-11-21 18:50:08,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1618866.6666666667, ans=0.125 2023-11-21 18:50:18,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1618933.3333333333, ans=0.125 2023-11-21 18:50:24,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1618933.3333333333, ans=0.5 2023-11-21 18:50:28,241 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242850 2023-11-21 18:50:34,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1619000.0, ans=0.0 2023-11-21 18:50:42,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1619066.6666666667, ans=0.125 2023-11-21 18:50:43,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1619066.6666666667, ans=0.0 2023-11-21 18:50:44,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1619066.6666666667, ans=0.2 2023-11-21 18:50:56,094 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2400, loss[loss=0.08198, simple_loss=0.1106, pruned_loss=0.01838, audio_tagging_loss=0.008295, over 17032.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09468, pruned_loss=0.01587, audio_tagging_loss=0.009592, over 3044702.56 frames. ], batch size: 62, lr: 3.28e-03, grad_scale: 32.0 2023-11-21 18:50:59,316 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.51 vs. limit=10.0 2023-11-21 18:51:07,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1619200.0, ans=0.1 2023-11-21 18:51:15,447 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.343e+01 8.783e+01 9.778e+01 1.689e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-21 18:51:24,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1619266.6666666667, ans=0.125 2023-11-21 18:51:31,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242900 2023-11-21 18:51:33,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.82 vs. limit=15.0 2023-11-21 18:51:33,909 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 18:51:35,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1619333.3333333333, ans=0.125 2023-11-21 18:51:37,927 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.48 vs. limit=15.0 2023-11-21 18:51:49,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1619400.0, ans=0.125 2023-11-21 18:51:57,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1619400.0, ans=0.125 2023-11-21 18:51:59,517 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2450, loss[loss=0.07846, simple_loss=0.1005, pruned_loss=0.01726, audio_tagging_loss=0.01095, over 14985.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09513, pruned_loss=0.01597, audio_tagging_loss=0.00969, over 3044160.27 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 32.0 2023-11-21 18:52:13,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1619533.3333333333, ans=0.125 2023-11-21 18:52:23,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1619533.3333333333, ans=0.0 2023-11-21 18:52:34,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1619600.0, ans=0.0 2023-11-21 18:52:36,205 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 242950 2023-11-21 18:52:43,033 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.56 vs. limit=22.5 2023-11-21 18:52:43,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1619666.6666666667, ans=0.1 2023-11-21 18:52:53,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1619733.3333333333, ans=0.05 2023-11-21 18:52:53,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1619733.3333333333, ans=0.1 2023-11-21 18:52:54,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1619733.3333333333, ans=0.1 2023-11-21 18:53:03,999 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2500, loss[loss=0.07683, simple_loss=0.09356, pruned_loss=0.02001, audio_tagging_loss=0.01004, over 15593.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09499, pruned_loss=0.01591, audio_tagging_loss=0.009764, over 3045169.37 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 32.0 2023-11-21 18:53:07,264 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=22.5 2023-11-21 18:53:23,258 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.882e+01 8.197e+01 8.814e+01 9.365e+01 1.246e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-21 18:53:27,737 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.12 vs. limit=10.0 2023-11-21 18:53:39,832 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243000 2023-11-21 18:54:06,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1620133.3333333333, ans=0.2 2023-11-21 18:54:08,141 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2550, loss[loss=0.06261, simple_loss=0.07977, pruned_loss=0.01499, audio_tagging_loss=0.007731, over 14477.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09617, pruned_loss=0.01621, audio_tagging_loss=0.009593, over 3050303.97 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 32.0 2023-11-21 18:54:09,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1620133.3333333333, ans=0.1 2023-11-21 18:54:20,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1620200.0, ans=0.125 2023-11-21 18:54:44,370 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243050 2023-11-21 18:54:51,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1620333.3333333333, ans=0.1 2023-11-21 18:55:01,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1620400.0, ans=0.125 2023-11-21 18:55:04,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.12 vs. limit=15.0 2023-11-21 18:55:12,132 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2600, loss[loss=0.08145, simple_loss=0.1172, pruned_loss=0.01714, audio_tagging_loss=0.005731, over 14902.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09531, pruned_loss=0.01606, audio_tagging_loss=0.009549, over 3054017.79 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:55:25,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1620533.3333333333, ans=0.125 2023-11-21 18:55:32,962 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.728e+01 8.396e+01 8.905e+01 9.438e+01 1.443e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-21 18:55:48,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243100 2023-11-21 18:55:54,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1620666.6666666667, ans=0.125 2023-11-21 18:55:59,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1620666.6666666667, ans=0.1 2023-11-21 18:56:00,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.24 vs. limit=12.0 2023-11-21 18:56:02,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1620733.3333333333, ans=0.0 2023-11-21 18:56:06,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1620733.3333333333, ans=0.05 2023-11-21 18:56:16,579 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2650, loss[loss=0.07217, simple_loss=0.09151, pruned_loss=0.01866, audio_tagging_loss=0.007752, over 15805.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09479, pruned_loss=0.01589, audio_tagging_loss=0.009422, over 3053473.00 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 18:56:20,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1620800.0, ans=0.125 2023-11-21 18:56:51,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243150 2023-11-21 18:56:53,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1621000.0, ans=0.2 2023-11-21 18:56:55,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1621000.0, ans=0.125 2023-11-21 18:56:56,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1621000.0, ans=0.1 2023-11-21 18:57:01,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1621000.0, ans=0.0 2023-11-21 18:57:05,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1621000.0, ans=0.0 2023-11-21 18:57:19,600 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.07 vs. limit=15.0 2023-11-21 18:57:19,994 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2700, loss[loss=0.1029, simple_loss=0.1357, pruned_loss=0.02765, audio_tagging_loss=0.007436, over 16237.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09518, pruned_loss=0.01596, audio_tagging_loss=0.009366, over 3050155.41 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 18:57:26,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1621133.3333333333, ans=0.0 2023-11-21 18:57:41,531 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 7.974e+01 8.822e+01 9.240e+01 1.303e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-21 18:57:56,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243200 2023-11-21 18:58:10,524 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.42 vs. limit=15.0 2023-11-21 18:58:13,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1621400.0, ans=0.125 2023-11-21 18:58:24,260 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2750, loss[loss=0.08381, simple_loss=0.1079, pruned_loss=0.01909, audio_tagging_loss=0.01074, over 14429.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.09604, pruned_loss=0.01611, audio_tagging_loss=0.009303, over 3047752.18 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 18:58:46,591 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.70 vs. limit=10.0 2023-11-21 18:58:48,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1621600.0, ans=0.0 2023-11-21 18:59:00,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243250 2023-11-21 18:59:17,897 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 18:59:28,135 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2800, loss[loss=0.06619, simple_loss=0.09147, pruned_loss=0.01227, audio_tagging_loss=0.008185, over 15817.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09548, pruned_loss=0.0161, audio_tagging_loss=0.009306, over 3048950.78 frames. ], batch size: 62, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 18:59:45,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1621866.6666666667, ans=0.125 2023-11-21 18:59:50,356 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 7.876e+01 8.639e+01 9.327e+01 1.140e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-21 18:59:53,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1621933.3333333333, ans=0.0 2023-11-21 18:59:57,552 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.29 vs. limit=6.0 2023-11-21 19:00:01,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1621933.3333333333, ans=0.0 2023-11-21 19:00:03,993 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243300 2023-11-21 19:00:07,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1622000.0, ans=0.125 2023-11-21 19:00:13,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1622000.0, ans=0.125 2023-11-21 19:00:15,547 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 19:00:31,561 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2850, loss[loss=0.06962, simple_loss=0.0891, pruned_loss=0.01496, audio_tagging_loss=0.01011, over 14978.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09504, pruned_loss=0.01613, audio_tagging_loss=0.009301, over 3044986.24 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:00:33,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1622133.3333333333, ans=0.1 2023-11-21 19:00:42,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1622133.3333333333, ans=0.95 2023-11-21 19:01:06,479 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243350 2023-11-21 19:01:06,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1622266.6666666667, ans=0.125 2023-11-21 19:01:06,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1622266.6666666667, ans=0.1 2023-11-21 19:01:20,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1622400.0, ans=0.0 2023-11-21 19:01:26,098 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.57 vs. limit=22.5 2023-11-21 19:01:34,251 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2900, loss[loss=0.09431, simple_loss=0.129, pruned_loss=0.02177, audio_tagging_loss=0.008041, over 15463.00 frames. ], tot_loss[loss=0.07414, simple_loss=0.09637, pruned_loss=0.01658, audio_tagging_loss=0.009377, over 3049452.12 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:01:51,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1622533.3333333333, ans=0.125 2023-11-21 19:01:55,209 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.905e+01 8.105e+01 8.663e+01 9.336e+01 1.251e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-21 19:02:09,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243400 2023-11-21 19:02:23,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1622666.6666666667, ans=0.125 2023-11-21 19:02:37,971 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 2950, loss[loss=0.06757, simple_loss=0.08849, pruned_loss=0.01438, audio_tagging_loss=0.008944, over 15103.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09591, pruned_loss=0.01636, audio_tagging_loss=0.009394, over 3054198.73 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 19:02:38,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1622800.0, ans=0.125 2023-11-21 19:02:39,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1622800.0, ans=0.1 2023-11-21 19:02:58,391 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.742e-03 2023-11-21 19:03:13,813 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243450 2023-11-21 19:03:25,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1623000.0, ans=0.125 2023-11-21 19:03:25,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1623000.0, ans=0.125 2023-11-21 19:03:37,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1623066.6666666667, ans=0.125 2023-11-21 19:03:39,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1623066.6666666667, ans=0.125 2023-11-21 19:03:41,271 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3000, loss[loss=0.0581, simple_loss=0.06984, pruned_loss=0.0104, audio_tagging_loss=0.01278, over 15236.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09591, pruned_loss=0.01638, audio_tagging_loss=0.009402, over 3057969.79 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 19:03:41,272 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 19:04:24,759 INFO [train_asr.py:1253] (3/4) Epoch 21, validation: loss=0.0594, simple_loss=0.05205, pruned_loss=0.005197, audio_tagging_loss=0.02817, over 4681554.00 frames. 2023-11-21 19:04:24,760 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 19:04:47,417 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.111e+01 8.701e+01 9.542e+01 1.276e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 19:04:50,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1623266.6666666667, ans=0.125 2023-11-21 19:04:58,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1623266.6666666667, ans=0.0 2023-11-21 19:05:00,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243500 2023-11-21 19:05:28,937 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3050, loss[loss=0.07544, simple_loss=0.09062, pruned_loss=0.019, audio_tagging_loss=0.01113, over 14320.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.09536, pruned_loss=0.01618, audio_tagging_loss=0.009566, over 3051504.84 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 19:06:05,509 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 19:06:05,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243550 2023-11-21 19:06:07,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1623666.6666666667, ans=0.1 2023-11-21 19:06:32,832 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3100, loss[loss=0.0807, simple_loss=0.1071, pruned_loss=0.02015, audio_tagging_loss=0.007004, over 16218.00 frames. ], tot_loss[loss=0.07391, simple_loss=0.09608, pruned_loss=0.01635, audio_tagging_loss=0.009519, over 3052956.35 frames. ], batch size: 60, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 19:06:56,497 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.034e+01 8.558e+01 9.488e+01 1.458e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-21 19:07:09,222 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243600 2023-11-21 19:07:14,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1624000.0, ans=0.125 2023-11-21 19:07:15,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1624000.0, ans=0.1 2023-11-21 19:07:15,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1624000.0, ans=0.0 2023-11-21 19:07:25,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1624066.6666666667, ans=0.0 2023-11-21 19:07:31,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.44 vs. limit=15.0 2023-11-21 19:07:38,436 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3150, loss[loss=0.06416, simple_loss=0.08692, pruned_loss=0.01233, audio_tagging_loss=0.008365, over 14790.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09674, pruned_loss=0.01632, audio_tagging_loss=0.009468, over 3040955.93 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 8.0 2023-11-21 19:08:01,451 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 19:08:01,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1624200.0, ans=0.0 2023-11-21 19:08:13,667 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243650 2023-11-21 19:08:29,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.41 vs. limit=10.0 2023-11-21 19:08:41,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1624400.0, ans=0.0 2023-11-21 19:08:42,946 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3200, loss[loss=0.06894, simple_loss=0.08747, pruned_loss=0.01305, audio_tagging_loss=0.01216, over 15053.00 frames. ], tot_loss[loss=0.07478, simple_loss=0.09728, pruned_loss=0.01653, audio_tagging_loss=0.009612, over 3038855.44 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:08:58,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.53 vs. limit=22.5 2023-11-21 19:09:04,552 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.386e+01 8.002e+01 8.673e+01 9.467e+01 1.702e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 19:09:11,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1624600.0, ans=0.125 2023-11-21 19:09:18,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243700 2023-11-21 19:09:27,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1624666.6666666667, ans=0.125 2023-11-21 19:09:45,217 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3250, loss[loss=0.06749, simple_loss=0.08386, pruned_loss=0.01527, audio_tagging_loss=0.01029, over 16119.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09619, pruned_loss=0.0162, audio_tagging_loss=0.009785, over 3041670.72 frames. ], batch size: 62, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:10:12,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1624933.3333333333, ans=0.1 2023-11-21 19:10:12,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1624933.3333333333, ans=0.04949747468305833 2023-11-21 19:10:20,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243750 2023-11-21 19:10:48,917 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3300, loss[loss=0.07409, simple_loss=0.0948, pruned_loss=0.01616, audio_tagging_loss=0.01053, over 15424.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.0964, pruned_loss=0.01638, audio_tagging_loss=0.009777, over 3044512.15 frames. ], batch size: 60, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:10:50,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1625133.3333333333, ans=0.1 2023-11-21 19:10:53,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1625133.3333333333, ans=0.05 2023-11-21 19:10:59,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1625133.3333333333, ans=10.0 2023-11-21 19:11:12,003 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 8.264e+01 8.839e+01 9.527e+01 1.267e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 19:11:13,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1625266.6666666667, ans=0.1 2023-11-21 19:11:24,427 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243800 2023-11-21 19:11:29,172 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.19 vs. limit=22.5 2023-11-21 19:11:53,923 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3350, loss[loss=0.1004, simple_loss=0.1376, pruned_loss=0.02552, audio_tagging_loss=0.006098, over 16303.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.0959, pruned_loss=0.01621, audio_tagging_loss=0.009713, over 3047541.76 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:12:01,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1625466.6666666667, ans=0.0 2023-11-21 19:12:10,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.81 vs. limit=15.0 2023-11-21 19:12:29,508 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243850 2023-11-21 19:12:57,966 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3400, loss[loss=0.05896, simple_loss=0.07174, pruned_loss=0.009295, audio_tagging_loss=0.01379, over 15361.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.09548, pruned_loss=0.01605, audio_tagging_loss=0.009649, over 3046878.15 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:13:10,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1625866.6666666667, ans=0.07 2023-11-21 19:13:17,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1625866.6666666667, ans=0.2 2023-11-21 19:13:19,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1625866.6666666667, ans=0.0 2023-11-21 19:13:20,637 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.826e+01 8.123e+01 8.713e+01 9.440e+01 3.289e+02, threshold=1.743e+02, percent-clipped=1.0 2023-11-21 19:13:34,172 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243900 2023-11-21 19:13:50,743 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.39 vs. limit=12.0 2023-11-21 19:13:53,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1626066.6666666667, ans=0.5 2023-11-21 19:14:01,812 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3450, loss[loss=0.05075, simple_loss=0.06678, pruned_loss=0.007869, audio_tagging_loss=0.009492, over 17059.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09576, pruned_loss=0.01609, audio_tagging_loss=0.009502, over 3043160.76 frames. ], batch size: 66, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:14:23,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=1626200.0, ans=0.2 2023-11-21 19:14:37,952 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 243950 2023-11-21 19:14:43,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1626333.3333333333, ans=0.125 2023-11-21 19:14:57,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1626400.0, ans=0.125 2023-11-21 19:15:01,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1626400.0, ans=0.0 2023-11-21 19:15:06,970 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3500, loss[loss=0.08229, simple_loss=0.1052, pruned_loss=0.02143, audio_tagging_loss=0.008246, over 16079.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09615, pruned_loss=0.01626, audio_tagging_loss=0.009325, over 3049015.71 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 16.0 2023-11-21 19:15:21,501 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.50 vs. limit=10.0 2023-11-21 19:15:29,261 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.309e+01 8.089e+01 8.726e+01 9.622e+01 1.634e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-21 19:15:33,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1626600.0, ans=0.2 2023-11-21 19:15:35,049 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.36 vs. limit=12.0 2023-11-21 19:15:37,851 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 19:15:42,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244000 2023-11-21 19:16:02,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1626733.3333333333, ans=0.125 2023-11-21 19:16:02,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.13 vs. limit=12.0 2023-11-21 19:16:13,957 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3550, loss[loss=0.04591, simple_loss=0.04926, pruned_loss=0.007212, audio_tagging_loss=0.01407, over 15110.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09641, pruned_loss=0.01636, audio_tagging_loss=0.009288, over 3037954.68 frames. ], batch size: 59, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:16:19,508 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.67 vs. limit=15.0 2023-11-21 19:16:31,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1626866.6666666667, ans=0.1 2023-11-21 19:16:32,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1626866.6666666667, ans=0.125 2023-11-21 19:16:32,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1626866.6666666667, ans=0.125 2023-11-21 19:16:45,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1626933.3333333333, ans=0.125 2023-11-21 19:16:51,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244050 2023-11-21 19:16:58,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1627000.0, ans=0.2 2023-11-21 19:17:02,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1627000.0, ans=0.2 2023-11-21 19:17:04,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1627066.6666666667, ans=0.125 2023-11-21 19:17:18,062 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3600, loss[loss=0.06543, simple_loss=0.07268, pruned_loss=0.0176, audio_tagging_loss=0.01149, over 14717.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09618, pruned_loss=0.01619, audio_tagging_loss=0.009375, over 3045261.40 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:17:41,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0 2023-11-21 19:17:41,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.77 vs. limit=15.0 2023-11-21 19:17:42,708 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.963e+01 8.176e+01 8.799e+01 9.565e+01 1.406e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-21 19:17:49,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.79 vs. limit=10.0 2023-11-21 19:17:53,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244100 2023-11-21 19:18:02,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1627333.3333333333, ans=0.0 2023-11-21 19:18:21,753 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3650, loss[loss=0.06764, simple_loss=0.0929, pruned_loss=0.01182, audio_tagging_loss=0.009375, over 15447.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09513, pruned_loss=0.01605, audio_tagging_loss=0.009418, over 3043835.86 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:18:28,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1627466.6666666667, ans=0.1 2023-11-21 19:18:56,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1627600.0, ans=0.1 2023-11-21 19:18:57,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244150 2023-11-21 19:19:26,331 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3700, loss[loss=0.06603, simple_loss=0.06846, pruned_loss=0.01832, audio_tagging_loss=0.01348, over 15303.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09512, pruned_loss=0.01624, audio_tagging_loss=0.009562, over 3042950.56 frames. ], batch size: 61, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:19:33,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0 2023-11-21 19:19:50,827 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.276e+01 8.879e+01 9.943e+01 1.926e+02, threshold=1.776e+02, percent-clipped=1.0 2023-11-21 19:19:53,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1627933.3333333333, ans=0.0 2023-11-21 19:19:55,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1627933.3333333333, ans=0.05 2023-11-21 19:20:02,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244200 2023-11-21 19:20:13,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1628000.0, ans=0.0 2023-11-21 19:20:17,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.33 vs. limit=15.0 2023-11-21 19:20:27,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1628066.6666666667, ans=0.0 2023-11-21 19:20:30,385 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3750, loss[loss=0.07374, simple_loss=0.09843, pruned_loss=0.01728, audio_tagging_loss=0.007249, over 16368.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.0952, pruned_loss=0.01622, audio_tagging_loss=0.009543, over 3048660.84 frames. ], batch size: 62, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:20:34,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1628133.3333333333, ans=0.125 2023-11-21 19:20:40,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1628133.3333333333, ans=0.0 2023-11-21 19:20:45,724 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.13 vs. limit=10.0 2023-11-21 19:20:50,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1628200.0, ans=15.0 2023-11-21 19:21:00,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.70 vs. limit=12.0 2023-11-21 19:21:06,994 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244250 2023-11-21 19:21:14,299 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 19:21:34,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1628466.6666666667, ans=0.125 2023-11-21 19:21:35,244 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3800, loss[loss=0.09318, simple_loss=0.1181, pruned_loss=0.02512, audio_tagging_loss=0.008987, over 15366.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.0958, pruned_loss=0.0164, audio_tagging_loss=0.009496, over 3045520.35 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:21:49,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=15.0 2023-11-21 19:21:59,386 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.245e+01 8.842e+01 9.701e+01 1.541e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 19:22:10,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244300 2023-11-21 19:22:13,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.01 vs. limit=15.0 2023-11-21 19:22:22,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.39 vs. limit=12.0 2023-11-21 19:22:24,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1628666.6666666667, ans=0.0 2023-11-21 19:22:27,116 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.70 vs. limit=22.5 2023-11-21 19:22:30,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.42 vs. limit=22.5 2023-11-21 19:22:39,765 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3850, loss[loss=0.07676, simple_loss=0.1043, pruned_loss=0.01772, audio_tagging_loss=0.006895, over 14480.00 frames. ], tot_loss[loss=0.07369, simple_loss=0.09592, pruned_loss=0.01623, audio_tagging_loss=0.0095, over 3048594.70 frames. ], batch size: 55, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:22:46,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1628800.0, ans=0.0 2023-11-21 19:22:51,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1628866.6666666667, ans=0.0 2023-11-21 19:23:13,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1628933.3333333333, ans=0.125 2023-11-21 19:23:15,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244350 2023-11-21 19:23:15,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1628933.3333333333, ans=0.0 2023-11-21 19:23:25,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1629000.0, ans=0.2 2023-11-21 19:23:36,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1629066.6666666667, ans=0.1 2023-11-21 19:23:43,366 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3900, loss[loss=0.0598, simple_loss=0.07489, pruned_loss=0.01117, audio_tagging_loss=0.01118, over 15046.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09591, pruned_loss=0.01637, audio_tagging_loss=0.009447, over 3052265.64 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:23:46,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1629133.3333333333, ans=0.07 2023-11-21 19:23:49,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1629133.3333333333, ans=0.125 2023-11-21 19:24:08,372 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.034e+01 8.736e+01 9.358e+01 1.610e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 19:24:14,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1629266.6666666667, ans=0.125 2023-11-21 19:24:16,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1629266.6666666667, ans=0.1 2023-11-21 19:24:20,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244400 2023-11-21 19:24:21,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1629333.3333333333, ans=0.0 2023-11-21 19:24:48,576 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 3950, loss[loss=0.07672, simple_loss=0.09041, pruned_loss=0.01793, audio_tagging_loss=0.01359, over 15837.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09574, pruned_loss=0.01636, audio_tagging_loss=0.009607, over 3050340.59 frames. ], batch size: 61, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:25:08,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1629533.3333333333, ans=0.2 2023-11-21 19:25:23,559 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244450 2023-11-21 19:25:28,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1629666.6666666667, ans=0.1 2023-11-21 19:25:29,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1629666.6666666667, ans=0.0 2023-11-21 19:25:37,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1629666.6666666667, ans=0.0 2023-11-21 19:25:38,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1629733.3333333333, ans=0.0 2023-11-21 19:25:40,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=12.0 2023-11-21 19:25:49,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1629733.3333333333, ans=0.125 2023-11-21 19:25:50,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1629733.3333333333, ans=0.125 2023-11-21 19:25:52,440 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4000, loss[loss=0.08911, simple_loss=0.1188, pruned_loss=0.02032, audio_tagging_loss=0.009374, over 14876.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09623, pruned_loss=0.01648, audio_tagging_loss=0.009676, over 3052987.18 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:25:56,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1629800.0, ans=0.125 2023-11-21 19:26:16,052 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.271e+01 8.886e+01 9.884e+01 1.294e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-21 19:26:21,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.95 vs. limit=12.0 2023-11-21 19:26:22,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1629933.3333333333, ans=0.1 2023-11-21 19:26:28,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244500 2023-11-21 19:26:42,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1630066.6666666667, ans=0.1 2023-11-21 19:26:56,265 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4050, loss[loss=0.06821, simple_loss=0.103, pruned_loss=0.009707, audio_tagging_loss=0.007014, over 16026.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09621, pruned_loss=0.01639, audio_tagging_loss=0.009677, over 3054613.66 frames. ], batch size: 61, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:26:56,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1630133.3333333333, ans=0.125 2023-11-21 19:26:58,668 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 19:27:04,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1630133.3333333333, ans=0.2 2023-11-21 19:27:10,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.59 vs. limit=15.0 2023-11-21 19:27:15,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1630200.0, ans=0.2 2023-11-21 19:27:22,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1630266.6666666667, ans=0.2 2023-11-21 19:27:25,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.47 vs. limit=15.0 2023-11-21 19:27:32,328 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244550 2023-11-21 19:27:36,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1630333.3333333333, ans=0.125 2023-11-21 19:28:00,860 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4100, loss[loss=0.0665, simple_loss=0.07995, pruned_loss=0.0173, audio_tagging_loss=0.009218, over 14419.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.09647, pruned_loss=0.01647, audio_tagging_loss=0.009725, over 3055605.31 frames. ], batch size: 55, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:28:19,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.74 vs. limit=15.0 2023-11-21 19:28:21,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1630533.3333333333, ans=0.0 2023-11-21 19:28:25,962 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.257e+01 8.384e+01 8.914e+01 9.573e+01 1.296e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-21 19:28:34,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.39 vs. limit=15.0 2023-11-21 19:28:35,071 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.72 vs. limit=12.0 2023-11-21 19:28:36,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244600 2023-11-21 19:28:39,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1630666.6666666667, ans=0.0 2023-11-21 19:29:06,013 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4150, loss[loss=0.08943, simple_loss=0.1195, pruned_loss=0.0232, audio_tagging_loss=0.006495, over 16504.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09609, pruned_loss=0.01636, audio_tagging_loss=0.009531, over 3052328.06 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:29:11,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1630800.0, ans=0.125 2023-11-21 19:29:26,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1630866.6666666667, ans=0.0 2023-11-21 19:29:40,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.57 vs. limit=15.0 2023-11-21 19:29:42,538 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244650 2023-11-21 19:29:53,003 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 19:29:56,050 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.11 vs. limit=15.0 2023-11-21 19:29:57,540 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.11 vs. limit=10.0 2023-11-21 19:29:58,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1631066.6666666667, ans=0.125 2023-11-21 19:30:09,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1631133.3333333333, ans=0.1 2023-11-21 19:30:10,071 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4200, loss[loss=0.08949, simple_loss=0.127, pruned_loss=0.01725, audio_tagging_loss=0.008751, over 16289.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09782, pruned_loss=0.01651, audio_tagging_loss=0.009385, over 3056731.54 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:30:34,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.19 vs. limit=15.0 2023-11-21 19:30:36,271 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 7.875e+01 8.455e+01 9.386e+01 1.251e+02, threshold=1.691e+02, percent-clipped=0.0 2023-11-21 19:30:46,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244700 2023-11-21 19:30:52,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1631333.3333333333, ans=0.0 2023-11-21 19:30:59,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1631333.3333333333, ans=0.015 2023-11-21 19:31:02,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1631400.0, ans=0.2 2023-11-21 19:31:15,166 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4250, loss[loss=0.0642, simple_loss=0.07456, pruned_loss=0.0161, audio_tagging_loss=0.01082, over 15562.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09712, pruned_loss=0.01646, audio_tagging_loss=0.009365, over 3055326.70 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:31:23,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1631466.6666666667, ans=0.2 2023-11-21 19:31:25,557 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.66 vs. limit=22.5 2023-11-21 19:31:50,282 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244750 2023-11-21 19:31:51,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1631666.6666666667, ans=0.2 2023-11-21 19:31:58,954 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.01 vs. limit=15.0 2023-11-21 19:32:06,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1631733.3333333333, ans=0.5 2023-11-21 19:32:11,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1631733.3333333333, ans=0.125 2023-11-21 19:32:19,209 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4300, loss[loss=0.09703, simple_loss=0.1315, pruned_loss=0.0241, audio_tagging_loss=0.007173, over 15194.00 frames. ], tot_loss[loss=0.07433, simple_loss=0.0973, pruned_loss=0.01641, audio_tagging_loss=0.009266, over 3053940.22 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:32:32,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1631866.6666666667, ans=0.05 2023-11-21 19:32:38,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.43 vs. limit=5.0 2023-11-21 19:32:43,222 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.395e+01 8.866e+01 9.565e+01 1.376e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-21 19:32:55,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244800 2023-11-21 19:33:10,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1632066.6666666667, ans=0.0 2023-11-21 19:33:15,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1632066.6666666667, ans=0.05 2023-11-21 19:33:22,514 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4350, loss[loss=0.06304, simple_loss=0.07706, pruned_loss=0.01104, audio_tagging_loss=0.01347, over 14819.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.0968, pruned_loss=0.01627, audio_tagging_loss=0.009219, over 3048035.41 frames. ], batch size: 59, lr: 3.27e-03, grad_scale: 16.0 2023-11-21 19:33:29,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1632133.3333333333, ans=0.0 2023-11-21 19:33:33,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1632200.0, ans=0.125 2023-11-21 19:33:36,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.20 vs. limit=22.5 2023-11-21 19:33:53,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1632266.6666666667, ans=0.0 2023-11-21 19:33:58,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244850 2023-11-21 19:34:02,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1632333.3333333333, ans=0.0 2023-11-21 19:34:11,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1632333.3333333333, ans=0.07 2023-11-21 19:34:20,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1632400.0, ans=0.125 2023-11-21 19:34:20,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.45 vs. limit=15.0 2023-11-21 19:34:27,310 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4400, loss[loss=0.07946, simple_loss=0.1102, pruned_loss=0.01731, audio_tagging_loss=0.007036, over 15090.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09707, pruned_loss=0.01626, audio_tagging_loss=0.009249, over 3052894.87 frames. ], batch size: 55, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:34:30,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.64 vs. limit=15.0 2023-11-21 19:34:41,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1632533.3333333333, ans=0.1 2023-11-21 19:34:52,769 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.144e+01 8.724e+01 9.531e+01 1.294e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-21 19:35:00,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1632600.0, ans=0.125 2023-11-21 19:35:02,714 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244900 2023-11-21 19:35:05,746 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.07 vs. limit=10.0 2023-11-21 19:35:10,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1632666.6666666667, ans=0.125 2023-11-21 19:35:19,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1632733.3333333333, ans=0.125 2023-11-21 19:35:24,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1632733.3333333333, ans=0.1 2023-11-21 19:35:32,214 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4450, loss[loss=0.07827, simple_loss=0.09873, pruned_loss=0.0207, audio_tagging_loss=0.00821, over 13798.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09651, pruned_loss=0.01618, audio_tagging_loss=0.009293, over 3048047.24 frames. ], batch size: 53, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:35:36,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1632800.0, ans=0.125 2023-11-21 19:35:44,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1632866.6666666667, ans=0.0 2023-11-21 19:35:52,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1632866.6666666667, ans=0.2 2023-11-21 19:35:53,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.40 vs. limit=15.0 2023-11-21 19:36:05,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.75 vs. limit=22.5 2023-11-21 19:36:07,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 244950 2023-11-21 19:36:14,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1633000.0, ans=0.0 2023-11-21 19:36:35,222 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4500, loss[loss=0.08356, simple_loss=0.1065, pruned_loss=0.02333, audio_tagging_loss=0.006981, over 14880.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.09676, pruned_loss=0.01628, audio_tagging_loss=0.009268, over 3049338.49 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:36:42,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1633133.3333333333, ans=0.125 2023-11-21 19:36:51,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1633200.0, ans=0.125 2023-11-21 19:36:54,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1633200.0, ans=0.125 2023-11-21 19:36:59,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1633200.0, ans=0.125 2023-11-21 19:37:00,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1633266.6666666667, ans=0.0 2023-11-21 19:37:01,318 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.155e+01 8.993e+01 9.811e+01 1.324e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-21 19:37:05,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1633266.6666666667, ans=0.0 2023-11-21 19:37:10,239 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.21 vs. limit=22.5 2023-11-21 19:37:11,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245000 2023-11-21 19:37:39,068 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4550, loss[loss=0.0801, simple_loss=0.09968, pruned_loss=0.01909, audio_tagging_loss=0.01117, over 14984.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09668, pruned_loss=0.01631, audio_tagging_loss=0.009299, over 3044372.32 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:37:42,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1633466.6666666667, ans=0.125 2023-11-21 19:37:51,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1633466.6666666667, ans=0.2 2023-11-21 19:37:57,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1633533.3333333333, ans=0.0 2023-11-21 19:38:16,117 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245050 2023-11-21 19:38:28,284 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 19:38:32,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1633733.3333333333, ans=0.2 2023-11-21 19:38:34,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1633733.3333333333, ans=0.0 2023-11-21 19:38:44,453 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4600, loss[loss=0.07881, simple_loss=0.1148, pruned_loss=0.01384, audio_tagging_loss=0.007544, over 14969.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09543, pruned_loss=0.01621, audio_tagging_loss=0.009454, over 3044811.34 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:38:53,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1633800.0, ans=0.125 2023-11-21 19:39:09,275 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.725e+01 8.114e+01 8.737e+01 9.348e+01 1.187e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 19:39:14,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1633933.3333333333, ans=0.125 2023-11-21 19:39:19,879 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245100 2023-11-21 19:39:29,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1634000.0, ans=0.2 2023-11-21 19:39:36,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.66 vs. limit=15.0 2023-11-21 19:39:39,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1634066.6666666667, ans=0.125 2023-11-21 19:39:42,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1634066.6666666667, ans=10.0 2023-11-21 19:39:47,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1634133.3333333333, ans=0.125 2023-11-21 19:39:48,074 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4650, loss[loss=0.08587, simple_loss=0.1069, pruned_loss=0.01909, audio_tagging_loss=0.01334, over 14978.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09582, pruned_loss=0.01634, audio_tagging_loss=0.009599, over 3042960.50 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:39:53,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1634133.3333333333, ans=0.0 2023-11-21 19:40:02,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1634200.0, ans=0.1 2023-11-21 19:40:08,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1634200.0, ans=0.125 2023-11-21 19:40:21,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1634266.6666666667, ans=0.0 2023-11-21 19:40:23,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245150 2023-11-21 19:40:29,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1634333.3333333333, ans=0.2 2023-11-21 19:40:31,563 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.68 vs. limit=15.0 2023-11-21 19:40:49,117 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0 2023-11-21 19:40:50,952 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4700, loss[loss=0.07268, simple_loss=0.09373, pruned_loss=0.01469, audio_tagging_loss=0.01112, over 15300.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09616, pruned_loss=0.01648, audio_tagging_loss=0.009645, over 3047400.43 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:41:17,148 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.203e+01 8.093e+01 8.785e+01 9.721e+01 1.198e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-21 19:41:17,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1634600.0, ans=0.1 2023-11-21 19:41:21,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1634600.0, ans=0.0 2023-11-21 19:41:27,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245200 2023-11-21 19:41:36,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.80 vs. limit=15.0 2023-11-21 19:41:55,452 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4750, loss[loss=0.05949, simple_loss=0.07514, pruned_loss=0.01192, audio_tagging_loss=0.01, over 13886.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09492, pruned_loss=0.0161, audio_tagging_loss=0.009672, over 3042752.49 frames. ], batch size: 54, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:42:02,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1634800.0, ans=0.0 2023-11-21 19:42:10,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1634866.6666666667, ans=0.125 2023-11-21 19:42:18,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1634866.6666666667, ans=0.125 2023-11-21 19:42:28,067 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 19:42:30,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245250 2023-11-21 19:42:36,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1635000.0, ans=0.1 2023-11-21 19:42:58,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1635133.3333333333, ans=0.025 2023-11-21 19:42:59,123 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4800, loss[loss=0.05947, simple_loss=0.07397, pruned_loss=0.01109, audio_tagging_loss=0.0114, over 15335.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09438, pruned_loss=0.0161, audio_tagging_loss=0.009883, over 3044774.46 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:43:25,997 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.114e+01 8.825e+01 9.753e+01 1.179e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-21 19:43:34,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245300 2023-11-21 19:43:35,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1635266.6666666667, ans=0.0 2023-11-21 19:43:50,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1635400.0, ans=0.2 2023-11-21 19:44:02,532 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4850, loss[loss=0.07692, simple_loss=0.09654, pruned_loss=0.01967, audio_tagging_loss=0.008977, over 17134.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09506, pruned_loss=0.01631, audio_tagging_loss=0.009856, over 3053272.84 frames. ], batch size: 65, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:44:05,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1635466.6666666667, ans=0.125 2023-11-21 19:44:06,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.03 vs. limit=15.0 2023-11-21 19:44:06,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1635466.6666666667, ans=0.125 2023-11-21 19:44:26,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1635533.3333333333, ans=0.125 2023-11-21 19:44:34,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1635600.0, ans=0.2 2023-11-21 19:44:36,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.75 vs. limit=15.0 2023-11-21 19:44:39,413 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245350 2023-11-21 19:44:50,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1635666.6666666667, ans=0.1 2023-11-21 19:45:07,161 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4900, loss[loss=0.09408, simple_loss=0.1222, pruned_loss=0.02383, audio_tagging_loss=0.009148, over 15165.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09592, pruned_loss=0.01625, audio_tagging_loss=0.009864, over 3053051.69 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:45:09,139 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.89 vs. limit=10.0 2023-11-21 19:45:34,466 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.513e+01 7.987e+01 8.445e+01 9.023e+01 1.315e+02, threshold=1.689e+02, percent-clipped=0.0 2023-11-21 19:45:37,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1635933.3333333333, ans=0.125 2023-11-21 19:45:43,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245400 2023-11-21 19:45:49,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1636000.0, ans=0.125 2023-11-21 19:45:52,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1636000.0, ans=0.0 2023-11-21 19:45:59,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1636066.6666666667, ans=0.1 2023-11-21 19:46:03,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1636066.6666666667, ans=0.05 2023-11-21 19:46:08,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1636066.6666666667, ans=0.2 2023-11-21 19:46:12,630 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 4950, loss[loss=0.08274, simple_loss=0.09889, pruned_loss=0.02376, audio_tagging_loss=0.009536, over 14863.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09549, pruned_loss=0.01626, audio_tagging_loss=0.009616, over 3043591.74 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:46:21,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1636133.3333333333, ans=0.0 2023-11-21 19:46:37,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1636266.6666666667, ans=0.035 2023-11-21 19:46:42,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1636266.6666666667, ans=0.125 2023-11-21 19:46:46,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1636266.6666666667, ans=0.125 2023-11-21 19:46:48,302 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245450 2023-11-21 19:47:08,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1636400.0, ans=0.125 2023-11-21 19:47:14,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1636400.0, ans=0.2 2023-11-21 19:47:14,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1636400.0, ans=0.0 2023-11-21 19:47:16,840 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5000, loss[loss=0.06013, simple_loss=0.0728, pruned_loss=0.01241, audio_tagging_loss=0.01132, over 15279.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09468, pruned_loss=0.01618, audio_tagging_loss=0.009587, over 3040991.61 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 32.0 2023-11-21 19:47:25,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1636466.6666666667, ans=0.125 2023-11-21 19:47:44,388 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.789e+01 8.147e+01 8.751e+01 9.572e+01 1.278e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-21 19:47:47,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1636600.0, ans=0.04949747468305833 2023-11-21 19:47:48,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1636600.0, ans=0.125 2023-11-21 19:47:53,487 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245500 2023-11-21 19:48:12,136 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.36 vs. limit=15.0 2023-11-21 19:48:21,026 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5050, loss[loss=0.07174, simple_loss=0.08388, pruned_loss=0.01838, audio_tagging_loss=0.01142, over 14811.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09531, pruned_loss=0.01629, audio_tagging_loss=0.009553, over 3042525.47 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:48:50,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1636933.3333333333, ans=0.125 2023-11-21 19:48:56,568 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245550 2023-11-21 19:48:56,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1636933.3333333333, ans=0.2 2023-11-21 19:49:25,080 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5100, loss[loss=0.08522, simple_loss=0.1107, pruned_loss=0.02057, audio_tagging_loss=0.009305, over 14540.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.095, pruned_loss=0.01622, audio_tagging_loss=0.009556, over 3037082.73 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:49:36,126 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.41 vs. limit=10.0 2023-11-21 19:49:51,103 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.84 vs. limit=15.0 2023-11-21 19:49:51,612 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.776e+01 7.985e+01 8.659e+01 9.540e+01 1.416e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-21 19:50:00,790 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245600 2023-11-21 19:50:29,306 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5150, loss[loss=0.06792, simple_loss=0.08777, pruned_loss=0.0168, audio_tagging_loss=0.007241, over 14724.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09435, pruned_loss=0.01605, audio_tagging_loss=0.009469, over 3038627.20 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 19:50:44,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1637533.3333333333, ans=0.0 2023-11-21 19:50:46,908 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.60 vs. limit=22.5 2023-11-21 19:51:05,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245650 2023-11-21 19:51:17,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.59 vs. limit=6.0 2023-11-21 19:51:31,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1637733.3333333333, ans=0.1 2023-11-21 19:51:33,901 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5200, loss[loss=0.1033, simple_loss=0.1345, pruned_loss=0.02747, audio_tagging_loss=0.008569, over 15166.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09536, pruned_loss=0.01609, audio_tagging_loss=0.009315, over 3047269.82 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:52:01,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 7.936e+01 8.716e+01 9.499e+01 2.064e+02, threshold=1.743e+02, percent-clipped=1.0 2023-11-21 19:52:05,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1637933.3333333333, ans=0.2 2023-11-21 19:52:09,266 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245700 2023-11-21 19:52:10,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1638000.0, ans=0.125 2023-11-21 19:52:30,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1638066.6666666667, ans=0.2 2023-11-21 19:52:38,056 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5250, loss[loss=0.06805, simple_loss=0.08731, pruned_loss=0.01286, audio_tagging_loss=0.01154, over 15346.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09622, pruned_loss=0.01642, audio_tagging_loss=0.009334, over 3047749.70 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:52:55,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1638200.0, ans=0.125 2023-11-21 19:53:06,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1638266.6666666667, ans=0.125 2023-11-21 19:53:07,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1638266.6666666667, ans=0.025 2023-11-21 19:53:14,343 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245750 2023-11-21 19:53:22,387 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.80 vs. limit=15.0 2023-11-21 19:53:33,313 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 19:53:33,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1638400.0, ans=0.125 2023-11-21 19:53:41,696 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5300, loss[loss=0.07923, simple_loss=0.09194, pruned_loss=0.02043, audio_tagging_loss=0.01282, over 15129.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.09578, pruned_loss=0.01619, audio_tagging_loss=0.009348, over 3044842.25 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:53:44,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1638466.6666666667, ans=0.1 2023-11-21 19:53:51,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1638466.6666666667, ans=0.2 2023-11-21 19:53:55,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1638533.3333333333, ans=0.125 2023-11-21 19:54:03,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1638533.3333333333, ans=0.1 2023-11-21 19:54:10,763 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.198e+01 8.861e+01 9.238e+01 1.215e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-21 19:54:18,873 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245800 2023-11-21 19:54:21,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1638666.6666666667, ans=0.125 2023-11-21 19:54:30,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1638666.6666666667, ans=0.125 2023-11-21 19:54:47,322 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5350, loss[loss=0.06875, simple_loss=0.09374, pruned_loss=0.01416, audio_tagging_loss=0.007721, over 15002.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09601, pruned_loss=0.0161, audio_tagging_loss=0.009355, over 3045118.75 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:54:50,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1638800.0, ans=0.125 2023-11-21 19:54:58,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1638800.0, ans=0.5 2023-11-21 19:55:00,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.29 vs. limit=15.0 2023-11-21 19:55:19,863 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.20 vs. limit=6.0 2023-11-21 19:55:23,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245850 2023-11-21 19:55:35,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1639000.0, ans=0.1 2023-11-21 19:55:41,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1639066.6666666667, ans=0.0 2023-11-21 19:55:52,159 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5400, loss[loss=0.08361, simple_loss=0.1193, pruned_loss=0.01623, audio_tagging_loss=0.007721, over 15992.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09672, pruned_loss=0.01611, audio_tagging_loss=0.009407, over 3046599.01 frames. ], batch size: 60, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:55:54,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.38 vs. limit=22.5 2023-11-21 19:56:03,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1639200.0, ans=0.5 2023-11-21 19:56:04,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1639200.0, ans=0.5 2023-11-21 19:56:06,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1639200.0, ans=0.125 2023-11-21 19:56:19,861 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.098e+01 8.835e+01 9.334e+01 1.205e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 19:56:25,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1639266.6666666667, ans=0.125 2023-11-21 19:56:27,446 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 19:56:28,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245900 2023-11-21 19:56:55,970 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5450, loss[loss=0.07393, simple_loss=0.09875, pruned_loss=0.0125, audio_tagging_loss=0.01206, over 15977.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09681, pruned_loss=0.01619, audio_tagging_loss=0.00946, over 3043506.18 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:57:01,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1639466.6666666667, ans=0.125 2023-11-21 19:57:32,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 245950 2023-11-21 19:57:49,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1639733.3333333333, ans=0.125 2023-11-21 19:57:49,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1639733.3333333333, ans=0.2 2023-11-21 19:58:00,377 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5500, loss[loss=0.0688, simple_loss=0.08819, pruned_loss=0.01397, audio_tagging_loss=0.01074, over 15164.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09632, pruned_loss=0.01605, audio_tagging_loss=0.009458, over 3043216.32 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:58:10,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1639800.0, ans=0.1 2023-11-21 19:58:28,914 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.083e+01 8.649e+01 9.254e+01 1.187e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-21 19:58:36,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246000 2023-11-21 19:58:56,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1640066.6666666667, ans=0.125 2023-11-21 19:59:04,518 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5550, loss[loss=0.07106, simple_loss=0.08938, pruned_loss=0.01487, audio_tagging_loss=0.0115, over 16781.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09617, pruned_loss=0.01618, audio_tagging_loss=0.0096, over 3046744.29 frames. ], batch size: 64, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 19:59:06,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1640133.3333333333, ans=0.2 2023-11-21 19:59:10,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1640133.3333333333, ans=0.0 2023-11-21 19:59:11,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1640133.3333333333, ans=0.1 2023-11-21 19:59:40,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246050 2023-11-21 19:59:40,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1640266.6666666667, ans=0.1 2023-11-21 19:59:57,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1640400.0, ans=0.2 2023-11-21 19:59:59,370 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.33 vs. limit=15.0 2023-11-21 20:00:05,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.49 vs. limit=15.0 2023-11-21 20:00:08,714 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5600, loss[loss=0.06954, simple_loss=0.09035, pruned_loss=0.01427, audio_tagging_loss=0.01009, over 14599.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09561, pruned_loss=0.01612, audio_tagging_loss=0.009736, over 3046313.07 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 20:00:16,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1640466.6666666667, ans=0.125 2023-11-21 20:00:33,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1640600.0, ans=0.0 2023-11-21 20:00:37,207 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.033e+01 8.931e+01 9.623e+01 1.146e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-21 20:00:40,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1640600.0, ans=0.125 2023-11-21 20:00:44,777 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246100 2023-11-21 20:00:46,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1640666.6666666667, ans=0.0 2023-11-21 20:00:53,816 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 20:00:59,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.04 vs. limit=22.5 2023-11-21 20:01:00,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1640733.3333333333, ans=0.125 2023-11-21 20:01:02,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1640733.3333333333, ans=0.0 2023-11-21 20:01:11,918 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5650, loss[loss=0.1049, simple_loss=0.1365, pruned_loss=0.03003, audio_tagging_loss=0.006632, over 14649.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09588, pruned_loss=0.01618, audio_tagging_loss=0.009737, over 3045431.05 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 20:01:23,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1640800.0, ans=0.125 2023-11-21 20:01:40,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1640933.3333333333, ans=0.09899494936611666 2023-11-21 20:01:48,537 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246150 2023-11-21 20:02:02,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1641066.6666666667, ans=0.125 2023-11-21 20:02:09,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1641066.6666666667, ans=0.2 2023-11-21 20:02:15,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1641133.3333333333, ans=0.1 2023-11-21 20:02:16,869 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5700, loss[loss=0.08054, simple_loss=0.09625, pruned_loss=0.02066, audio_tagging_loss=0.01175, over 15574.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09594, pruned_loss=0.01629, audio_tagging_loss=0.009754, over 3045366.13 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 20:02:37,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1641200.0, ans=0.125 2023-11-21 20:02:39,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1641200.0, ans=0.125 2023-11-21 20:02:44,511 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.226e+01 8.873e+01 9.610e+01 1.173e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-21 20:02:52,762 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246200 2023-11-21 20:02:54,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1641333.3333333333, ans=0.125 2023-11-21 20:03:02,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1641333.3333333333, ans=0.125 2023-11-21 20:03:06,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1641333.3333333333, ans=0.0 2023-11-21 20:03:19,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1641400.0, ans=0.125 2023-11-21 20:03:21,456 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.04 vs. limit=15.0 2023-11-21 20:03:21,880 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5750, loss[loss=0.05456, simple_loss=0.06999, pruned_loss=0.01258, audio_tagging_loss=0.006983, over 14731.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09523, pruned_loss=0.01619, audio_tagging_loss=0.009669, over 3044534.92 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 20:03:27,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0 2023-11-21 20:03:58,220 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246250 2023-11-21 20:04:06,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1641666.6666666667, ans=0.0 2023-11-21 20:04:14,011 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.41 vs. limit=12.0 2023-11-21 20:04:25,622 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5800, loss[loss=0.05724, simple_loss=0.07919, pruned_loss=0.009299, audio_tagging_loss=0.00835, over 15234.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09474, pruned_loss=0.01598, audio_tagging_loss=0.009573, over 3042582.13 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:04:39,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1641866.6666666667, ans=0.0 2023-11-21 20:04:50,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.19 vs. limit=22.5 2023-11-21 20:04:54,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1641933.3333333333, ans=0.07 2023-11-21 20:04:54,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1641933.3333333333, ans=0.125 2023-11-21 20:04:55,494 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.128e+01 8.477e+01 9.291e+01 1.218e+02, threshold=1.695e+02, percent-clipped=0.0 2023-11-21 20:04:58,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1641933.3333333333, ans=0.125 2023-11-21 20:05:01,748 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246300 2023-11-21 20:05:03,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1642000.0, ans=0.0 2023-11-21 20:05:09,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.69 vs. limit=15.0 2023-11-21 20:05:15,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1642066.6666666667, ans=22.5 2023-11-21 20:05:29,852 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5850, loss[loss=0.06003, simple_loss=0.0792, pruned_loss=0.008927, audio_tagging_loss=0.0115, over 15523.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09378, pruned_loss=0.01571, audio_tagging_loss=0.009484, over 3042441.26 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:05:49,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1642200.0, ans=0.0 2023-11-21 20:06:05,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246350 2023-11-21 20:06:21,591 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.75 vs. limit=15.0 2023-11-21 20:06:24,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.65 vs. limit=15.0 2023-11-21 20:06:34,585 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5900, loss[loss=0.07516, simple_loss=0.1007, pruned_loss=0.01542, audio_tagging_loss=0.009403, over 15772.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.0939, pruned_loss=0.0157, audio_tagging_loss=0.009482, over 3042392.99 frames. ], batch size: 59, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:07:03,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1642600.0, ans=0.125 2023-11-21 20:07:04,002 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.116e+01 8.807e+01 9.373e+01 1.126e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-21 20:07:04,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1642600.0, ans=0.1 2023-11-21 20:07:10,122 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246400 2023-11-21 20:07:37,654 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 5950, loss[loss=0.09651, simple_loss=0.1217, pruned_loss=0.02137, audio_tagging_loss=0.0143, over 15993.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09463, pruned_loss=0.01578, audio_tagging_loss=0.009532, over 3044594.65 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:07:39,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1642800.0, ans=0.125 2023-11-21 20:07:57,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.47 vs. limit=15.0 2023-11-21 20:08:10,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.93 vs. limit=15.0 2023-11-21 20:08:14,383 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246450 2023-11-21 20:08:19,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1643000.0, ans=0.05 2023-11-21 20:08:42,284 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6000, loss[loss=0.07063, simple_loss=0.09017, pruned_loss=0.01319, audio_tagging_loss=0.01236, over 15211.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09575, pruned_loss=0.01607, audio_tagging_loss=0.009385, over 3043517.80 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0 2023-11-21 20:08:42,284 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 20:09:23,090 INFO [train_asr.py:1253] (3/4) Epoch 21, validation: loss=0.05951, simple_loss=0.05205, pruned_loss=0.005242, audio_tagging_loss=0.02825, over 4681554.00 frames. 2023-11-21 20:09:23,091 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 20:09:32,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.80 vs. limit=15.0 2023-11-21 20:09:34,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1643200.0, ans=0.125 2023-11-21 20:09:48,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.38 vs. limit=22.5 2023-11-21 20:09:48,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1643266.6666666667, ans=0.125 2023-11-21 20:09:52,395 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 7.949e+01 8.551e+01 9.154e+01 1.132e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-21 20:09:58,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246500 2023-11-21 20:10:09,451 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 20:10:24,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1643400.0, ans=0.2 2023-11-21 20:10:25,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1643466.6666666667, ans=0.1 2023-11-21 20:10:26,632 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6050, loss[loss=0.05361, simple_loss=0.06876, pruned_loss=0.01094, audio_tagging_loss=0.008285, over 14259.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09628, pruned_loss=0.01605, audio_tagging_loss=0.009287, over 3045648.93 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:10:29,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1643466.6666666667, ans=0.125 2023-11-21 20:10:32,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1643466.6666666667, ans=0.125 2023-11-21 20:10:44,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1643533.3333333333, ans=0.125 2023-11-21 20:11:02,835 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246550 2023-11-21 20:11:05,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1643666.6666666667, ans=0.125 2023-11-21 20:11:06,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1643666.6666666667, ans=0.125 2023-11-21 20:11:09,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1643666.6666666667, ans=0.125 2023-11-21 20:11:10,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1643666.6666666667, ans=0.125 2023-11-21 20:11:30,730 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6100, loss[loss=0.07861, simple_loss=0.09975, pruned_loss=0.01883, audio_tagging_loss=0.009895, over 15872.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09536, pruned_loss=0.01608, audio_tagging_loss=0.009457, over 3046197.80 frames. ], batch size: 62, lr: 3.26e-03, grad_scale: 8.0 2023-11-21 20:11:47,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1643866.6666666667, ans=0.125 2023-11-21 20:11:58,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1643933.3333333333, ans=0.0 2023-11-21 20:12:03,544 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 7.866e+01 8.426e+01 9.427e+01 1.141e+02, threshold=1.685e+02, percent-clipped=0.0 2023-11-21 20:12:06,079 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246600 2023-11-21 20:12:13,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1644000.0, ans=0.0 2023-11-21 20:12:17,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1644000.0, ans=0.125 2023-11-21 20:12:18,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1644000.0, ans=0.1 2023-11-21 20:12:23,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1644066.6666666667, ans=0.0 2023-11-21 20:12:34,404 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6150, loss[loss=0.09617, simple_loss=0.1254, pruned_loss=0.0242, audio_tagging_loss=0.00928, over 16147.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09523, pruned_loss=0.01594, audio_tagging_loss=0.009507, over 3039989.56 frames. ], batch size: 61, lr: 3.26e-03, grad_scale: 8.0 2023-11-21 20:13:00,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1644266.6666666667, ans=0.125 2023-11-21 20:13:07,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1644266.6666666667, ans=0.1 2023-11-21 20:13:09,396 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.71 vs. limit=15.0 2023-11-21 20:13:10,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246650 2023-11-21 20:13:12,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1644333.3333333333, ans=0.125 2023-11-21 20:13:37,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1644466.6666666667, ans=0.0 2023-11-21 20:13:37,956 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6200, loss[loss=0.08881, simple_loss=0.1176, pruned_loss=0.01966, audio_tagging_loss=0.01034, over 15954.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09476, pruned_loss=0.01592, audio_tagging_loss=0.009591, over 3043107.30 frames. ], batch size: 59, lr: 3.26e-03, grad_scale: 8.0 2023-11-21 20:13:56,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1644533.3333333333, ans=0.0 2023-11-21 20:14:05,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1644600.0, ans=0.025 2023-11-21 20:14:11,061 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.261e+01 8.724e+01 9.317e+01 1.600e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-21 20:14:13,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246700 2023-11-21 20:14:13,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1644600.0, ans=0.125 2023-11-21 20:14:22,515 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.55 vs. limit=15.0 2023-11-21 20:14:26,097 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.93 vs. limit=10.0 2023-11-21 20:14:28,470 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.34 vs. limit=15.0 2023-11-21 20:14:42,099 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6250, loss[loss=0.07385, simple_loss=0.09456, pruned_loss=0.01422, audio_tagging_loss=0.01235, over 15168.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09527, pruned_loss=0.01604, audio_tagging_loss=0.009624, over 3047702.22 frames. ], batch size: 59, lr: 3.26e-03, grad_scale: 8.0 2023-11-21 20:15:16,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1644933.3333333333, ans=0.125 2023-11-21 20:15:17,622 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246750 2023-11-21 20:15:24,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1645000.0, ans=0.2 2023-11-21 20:15:45,985 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6300, loss[loss=0.06783, simple_loss=0.09095, pruned_loss=0.01251, audio_tagging_loss=0.009846, over 15867.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09553, pruned_loss=0.01599, audio_tagging_loss=0.009661, over 3050226.67 frames. ], batch size: 59, lr: 3.26e-03, grad_scale: 8.0 2023-11-21 20:16:18,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=15.0 2023-11-21 20:16:19,053 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.125e+01 8.837e+01 9.702e+01 1.272e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 20:16:21,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246800 2023-11-21 20:16:22,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.25 vs. limit=15.0 2023-11-21 20:16:31,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1645333.3333333333, ans=0.0 2023-11-21 20:16:32,302 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.21 vs. limit=15.0 2023-11-21 20:16:42,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1645400.0, ans=0.125 2023-11-21 20:16:47,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1645400.0, ans=0.125 2023-11-21 20:16:50,195 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6350, loss[loss=0.05232, simple_loss=0.06321, pruned_loss=0.01014, audio_tagging_loss=0.01057, over 15647.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09529, pruned_loss=0.01596, audio_tagging_loss=0.00973, over 3044670.82 frames. ], batch size: 62, lr: 3.26e-03, grad_scale: 8.0 2023-11-21 20:17:06,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1645533.3333333333, ans=0.0 2023-11-21 20:17:18,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1645600.0, ans=0.1 2023-11-21 20:17:26,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246850 2023-11-21 20:17:54,730 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.69 vs. limit=15.0 2023-11-21 20:17:55,208 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6400, loss[loss=0.04499, simple_loss=0.04726, pruned_loss=0.008623, audio_tagging_loss=0.01274, over 12339.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09471, pruned_loss=0.01576, audio_tagging_loss=0.009879, over 3037243.25 frames. ], batch size: 52, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:17:59,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1645800.0, ans=0.125 2023-11-21 20:18:08,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1645866.6666666667, ans=0.2 2023-11-21 20:18:13,519 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.88 vs. limit=22.5 2023-11-21 20:18:21,311 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.23 vs. limit=15.0 2023-11-21 20:18:27,614 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.810e+01 7.956e+01 8.478e+01 9.124e+01 1.124e+02, threshold=1.696e+02, percent-clipped=0.0 2023-11-21 20:18:30,197 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246900 2023-11-21 20:18:40,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1646000.0, ans=0.0 2023-11-21 20:18:58,485 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6450, loss[loss=0.08356, simple_loss=0.1112, pruned_loss=0.01729, audio_tagging_loss=0.01064, over 15757.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09428, pruned_loss=0.01563, audio_tagging_loss=0.009989, over 3046087.05 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:19:09,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1646133.3333333333, ans=0.125 2023-11-21 20:19:14,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1646200.0, ans=0.125 2023-11-21 20:19:34,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 246950 2023-11-21 20:19:52,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1646400.0, ans=0.0 2023-11-21 20:20:02,062 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6500, loss[loss=0.0939, simple_loss=0.1237, pruned_loss=0.02404, audio_tagging_loss=0.007992, over 14423.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.0959, pruned_loss=0.016, audio_tagging_loss=0.009877, over 3052088.09 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 16.0 2023-11-21 20:20:16,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1646533.3333333333, ans=0.0 2023-11-21 20:20:30,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1646600.0, ans=0.125 2023-11-21 20:20:35,579 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.211e+01 8.118e+01 8.877e+01 9.588e+01 1.151e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-21 20:20:38,755 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247000 2023-11-21 20:20:44,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1646666.6666666667, ans=0.0 2023-11-21 20:20:46,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1646666.6666666667, ans=0.1 2023-11-21 20:20:47,020 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.23 vs. limit=15.0 2023-11-21 20:20:51,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1646666.6666666667, ans=0.0 2023-11-21 20:21:07,055 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6550, loss[loss=0.07859, simple_loss=0.109, pruned_loss=0.01392, audio_tagging_loss=0.01016, over 15120.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.0961, pruned_loss=0.01601, audio_tagging_loss=0.009795, over 3052350.07 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:21:10,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1646800.0, ans=0.0 2023-11-21 20:21:36,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1646933.3333333333, ans=0.125 2023-11-21 20:21:42,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1646933.3333333333, ans=0.0 2023-11-21 20:21:43,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247050 2023-11-21 20:22:08,096 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:22:09,768 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.77 vs. limit=15.0 2023-11-21 20:22:11,495 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6600, loss[loss=0.09181, simple_loss=0.1187, pruned_loss=0.02311, audio_tagging_loss=0.009358, over 14532.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09602, pruned_loss=0.01596, audio_tagging_loss=0.009591, over 3046588.03 frames. ], batch size: 53, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:22:36,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1647266.6666666667, ans=0.0 2023-11-21 20:22:39,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1647266.6666666667, ans=0.125 2023-11-21 20:22:44,401 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.213e+01 8.717e+01 9.748e+01 1.869e+02, threshold=1.743e+02, percent-clipped=1.0 2023-11-21 20:22:47,541 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247100 2023-11-21 20:23:10,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1647400.0, ans=0.125 2023-11-21 20:23:15,533 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6650, loss[loss=0.06701, simple_loss=0.08291, pruned_loss=0.01411, audio_tagging_loss=0.01144, over 14881.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09527, pruned_loss=0.0159, audio_tagging_loss=0.009561, over 3041219.68 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:23:42,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1647600.0, ans=0.125 2023-11-21 20:23:43,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1647600.0, ans=0.125 2023-11-21 20:23:50,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247150 2023-11-21 20:24:00,284 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.84 vs. limit=15.0 2023-11-21 20:24:03,931 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.22 vs. limit=15.0 2023-11-21 20:24:08,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1647733.3333333333, ans=0.125 2023-11-21 20:24:11,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1647733.3333333333, ans=0.5 2023-11-21 20:24:17,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1647800.0, ans=0.125 2023-11-21 20:24:18,050 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6700, loss[loss=0.06848, simple_loss=0.08664, pruned_loss=0.01418, audio_tagging_loss=0.01098, over 15031.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09417, pruned_loss=0.01578, audio_tagging_loss=0.009501, over 3036767.34 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:24:21,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1647800.0, ans=0.125 2023-11-21 20:24:39,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.82 vs. limit=6.0 2023-11-21 20:24:51,830 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.102e+01 8.678e+01 9.415e+01 1.201e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-21 20:24:52,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1647933.3333333333, ans=0.2 2023-11-21 20:24:54,346 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247200 2023-11-21 20:24:54,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0 2023-11-21 20:24:58,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1648000.0, ans=0.0 2023-11-21 20:25:14,844 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.44 vs. limit=10.0 2023-11-21 20:25:22,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1648133.3333333333, ans=0.125 2023-11-21 20:25:23,152 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6750, loss[loss=0.06949, simple_loss=0.09519, pruned_loss=0.01409, audio_tagging_loss=0.007802, over 15322.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09448, pruned_loss=0.01579, audio_tagging_loss=0.009487, over 3034841.30 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:25:24,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1648133.3333333333, ans=0.04949747468305833 2023-11-21 20:25:54,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1648266.6666666667, ans=0.025 2023-11-21 20:25:55,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1648266.6666666667, ans=0.2 2023-11-21 20:25:58,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247250 2023-11-21 20:26:01,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1648333.3333333333, ans=0.1 2023-11-21 20:26:01,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1648333.3333333333, ans=0.125 2023-11-21 20:26:26,350 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6800, loss[loss=0.07111, simple_loss=0.09937, pruned_loss=0.01313, audio_tagging_loss=0.008294, over 14367.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09488, pruned_loss=0.01588, audio_tagging_loss=0.009438, over 3028952.84 frames. ], batch size: 54, lr: 3.25e-03, grad_scale: 32.0 2023-11-21 20:26:59,646 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 7.919e+01 8.607e+01 9.295e+01 1.340e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-21 20:27:02,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247300 2023-11-21 20:27:05,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.75 vs. limit=15.0 2023-11-21 20:27:12,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1648666.6666666667, ans=0.125 2023-11-21 20:27:29,483 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6850, loss[loss=0.06769, simple_loss=0.09179, pruned_loss=0.01504, audio_tagging_loss=0.006761, over 15869.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09549, pruned_loss=0.01598, audio_tagging_loss=0.009371, over 3031678.41 frames. ], batch size: 61, lr: 3.25e-03, grad_scale: 32.0 2023-11-21 20:27:32,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1648800.0, ans=0.125 2023-11-21 20:27:34,535 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.00 vs. limit=22.5 2023-11-21 20:27:49,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1648866.6666666667, ans=0.1 2023-11-21 20:28:03,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1648933.3333333333, ans=0.125 2023-11-21 20:28:06,033 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247350 2023-11-21 20:28:29,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1649066.6666666667, ans=0.04949747468305833 2023-11-21 20:28:33,852 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6900, loss[loss=0.0824, simple_loss=0.1146, pruned_loss=0.01735, audio_tagging_loss=0.007761, over 14265.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09545, pruned_loss=0.01593, audio_tagging_loss=0.009375, over 3036272.81 frames. ], batch size: 53, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:28:38,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1649133.3333333333, ans=0.2 2023-11-21 20:28:39,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1649133.3333333333, ans=0.125 2023-11-21 20:28:44,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1649133.3333333333, ans=0.1 2023-11-21 20:28:49,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1649200.0, ans=0.125 2023-11-21 20:29:04,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0 2023-11-21 20:29:08,794 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.897e+01 8.008e+01 8.614e+01 9.356e+01 1.357e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 20:29:08,938 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247400 2023-11-21 20:29:20,266 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:29:20,759 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0 2023-11-21 20:29:23,618 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 20:29:37,984 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 6950, loss[loss=0.09822, simple_loss=0.1254, pruned_loss=0.0275, audio_tagging_loss=0.008036, over 15412.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09599, pruned_loss=0.01606, audio_tagging_loss=0.009403, over 3039530.34 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0 2023-11-21 20:29:49,696 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.30 vs. limit=15.0 2023-11-21 20:29:49,710 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.18 vs. limit=12.0 2023-11-21 20:29:54,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.51 vs. limit=15.0 2023-11-21 20:30:13,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247450 2023-11-21 20:30:26,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1649666.6666666667, ans=10.0 2023-11-21 20:30:29,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.00 vs. limit=22.5 2023-11-21 20:30:32,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.95 vs. limit=12.0 2023-11-21 20:30:41,554 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7000, loss[loss=0.07328, simple_loss=0.09433, pruned_loss=0.01658, audio_tagging_loss=0.009531, over 15925.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09519, pruned_loss=0.01598, audio_tagging_loss=0.009482, over 3036201.00 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 8.0 2023-11-21 20:31:08,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1649933.3333333333, ans=0.1 2023-11-21 20:31:10,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1649933.3333333333, ans=0.125 2023-11-21 20:31:17,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1649933.3333333333, ans=0.1 2023-11-21 20:31:18,710 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 7.928e+01 8.493e+01 9.155e+01 1.182e+02, threshold=1.699e+02, percent-clipped=0.0 2023-11-21 20:31:18,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247500 2023-11-21 20:31:22,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1650000.0, ans=0.125 2023-11-21 20:31:46,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1650133.3333333333, ans=0.2 2023-11-21 20:31:47,110 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7050, loss[loss=0.07102, simple_loss=0.09622, pruned_loss=0.01431, audio_tagging_loss=0.008599, over 15980.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09517, pruned_loss=0.01598, audio_tagging_loss=0.009585, over 3035533.15 frames. ], batch size: 63, lr: 3.25e-03, grad_scale: 8.0 2023-11-21 20:32:11,960 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.19 vs. limit=15.0 2023-11-21 20:32:22,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247550 2023-11-21 20:32:22,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1650266.6666666667, ans=0.0 2023-11-21 20:32:28,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.87 vs. limit=15.0 2023-11-21 20:32:36,152 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:32:51,798 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7100, loss[loss=0.08143, simple_loss=0.1167, pruned_loss=0.01622, audio_tagging_loss=0.006876, over 15335.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09561, pruned_loss=0.01604, audio_tagging_loss=0.009643, over 3040126.60 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0 2023-11-21 20:33:02,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.40 vs. limit=22.5 2023-11-21 20:33:27,103 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.093e+01 8.713e+01 9.346e+01 1.337e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 20:33:27,239 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247600 2023-11-21 20:33:32,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1650666.6666666667, ans=0.0 2023-11-21 20:33:34,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1650666.6666666667, ans=0.0 2023-11-21 20:33:48,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1650733.3333333333, ans=0.125 2023-11-21 20:33:51,147 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.80 vs. limit=12.0 2023-11-21 20:33:55,489 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7150, loss[loss=0.06236, simple_loss=0.08252, pruned_loss=0.01247, audio_tagging_loss=0.008624, over 15161.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.0955, pruned_loss=0.01603, audio_tagging_loss=0.009663, over 3033951.78 frames. ], batch size: 61, lr: 3.25e-03, grad_scale: 8.0 2023-11-21 20:33:59,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1650800.0, ans=0.0 2023-11-21 20:34:15,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1650866.6666666667, ans=0.05 2023-11-21 20:34:25,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.63 vs. limit=10.0 2023-11-21 20:34:26,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1650933.3333333333, ans=0.0 2023-11-21 20:34:32,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247650 2023-11-21 20:34:32,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1650933.3333333333, ans=0.1 2023-11-21 20:34:33,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1651000.0, ans=0.2 2023-11-21 20:34:44,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1651000.0, ans=0.0 2023-11-21 20:34:45,954 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.21 vs. limit=15.0 2023-11-21 20:34:51,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.67 vs. limit=22.5 2023-11-21 20:35:00,016 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7200, loss[loss=0.07915, simple_loss=0.1009, pruned_loss=0.01979, audio_tagging_loss=0.008898, over 16166.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.09623, pruned_loss=0.01625, audio_tagging_loss=0.009666, over 3036181.27 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:35:13,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1651200.0, ans=0.2 2023-11-21 20:35:30,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1651266.6666666667, ans=0.2 2023-11-21 20:35:32,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.45 vs. limit=15.0 2023-11-21 20:35:35,365 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.484e+01 9.159e+01 1.010e+02 1.295e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-21 20:35:35,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247700 2023-11-21 20:35:43,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1651333.3333333333, ans=0.125 2023-11-21 20:36:03,887 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7250, loss[loss=0.0873, simple_loss=0.1246, pruned_loss=0.01602, audio_tagging_loss=0.008971, over 16680.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09626, pruned_loss=0.01604, audio_tagging_loss=0.009679, over 3043459.46 frames. ], batch size: 60, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:36:04,235 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:36:09,309 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.47 vs. limit=15.0 2023-11-21 20:36:11,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1651466.6666666667, ans=0.125 2023-11-21 20:36:15,768 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:36:20,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1651533.3333333333, ans=0.2 2023-11-21 20:36:39,742 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247750 2023-11-21 20:36:47,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1651666.6666666667, ans=0.05 2023-11-21 20:37:05,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1651733.3333333333, ans=0.1 2023-11-21 20:37:07,606 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7300, loss[loss=0.074, simple_loss=0.1051, pruned_loss=0.01507, audio_tagging_loss=0.00639, over 16247.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09679, pruned_loss=0.01614, audio_tagging_loss=0.009535, over 3039203.34 frames. ], batch size: 61, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:37:43,491 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.100e+01 8.051e+01 8.700e+01 9.385e+01 1.347e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 20:37:43,634 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247800 2023-11-21 20:37:49,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.23 vs. limit=12.0 2023-11-21 20:37:53,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1652000.0, ans=0.125 2023-11-21 20:38:03,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1652066.6666666667, ans=0.125 2023-11-21 20:38:04,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1652066.6666666667, ans=0.1 2023-11-21 20:38:09,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1652066.6666666667, ans=0.04949747468305833 2023-11-21 20:38:10,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1652066.6666666667, ans=0.1 2023-11-21 20:38:12,562 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7350, loss[loss=0.06138, simple_loss=0.08463, pruned_loss=0.01049, audio_tagging_loss=0.008574, over 14868.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09661, pruned_loss=0.01613, audio_tagging_loss=0.009409, over 3039774.89 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:38:40,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1652266.6666666667, ans=0.125 2023-11-21 20:38:42,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1652266.6666666667, ans=0.2 2023-11-21 20:38:44,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1652266.6666666667, ans=0.125 2023-11-21 20:38:44,632 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:38:47,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247850 2023-11-21 20:38:52,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.64 vs. limit=15.0 2023-11-21 20:38:53,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1652333.3333333333, ans=0.125 2023-11-21 20:38:56,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1652333.3333333333, ans=0.2 2023-11-21 20:39:16,011 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7400, loss[loss=0.06231, simple_loss=0.07793, pruned_loss=0.009278, audio_tagging_loss=0.01407, over 13930.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09517, pruned_loss=0.01581, audio_tagging_loss=0.009431, over 3042630.30 frames. ], batch size: 53, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:39:41,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1652600.0, ans=0.0 2023-11-21 20:39:51,552 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.103e+01 8.809e+01 9.626e+01 1.321e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-21 20:39:51,710 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247900 2023-11-21 20:40:05,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1652733.3333333333, ans=0.125 2023-11-21 20:40:19,647 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7450, loss[loss=0.05113, simple_loss=0.06414, pruned_loss=0.008212, audio_tagging_loss=0.01085, over 15002.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09547, pruned_loss=0.01588, audio_tagging_loss=0.009359, over 3035745.25 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:40:30,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1652866.6666666667, ans=0.09899494936611666 2023-11-21 20:40:55,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 247950 2023-11-21 20:41:07,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1653000.0, ans=0.2 2023-11-21 20:41:13,027 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:41:22,575 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7500, loss[loss=0.06135, simple_loss=0.08203, pruned_loss=0.01152, audio_tagging_loss=0.008813, over 15019.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09545, pruned_loss=0.0159, audio_tagging_loss=0.00932, over 3037519.28 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:41:24,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1653133.3333333333, ans=15.0 2023-11-21 20:41:33,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1653133.3333333333, ans=0.0 2023-11-21 20:41:39,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1653200.0, ans=0.0 2023-11-21 20:41:44,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1653200.0, ans=0.125 2023-11-21 20:41:44,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1653200.0, ans=0.0 2023-11-21 20:41:57,958 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.210e+01 8.790e+01 9.485e+01 1.319e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 20:41:58,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248000 2023-11-21 20:42:13,331 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:42:13,892 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.01 vs. limit=15.0 2023-11-21 20:42:29,575 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7550, loss[loss=0.0735, simple_loss=0.1075, pruned_loss=0.0112, audio_tagging_loss=0.008529, over 14154.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09502, pruned_loss=0.01589, audio_tagging_loss=0.009251, over 3039239.23 frames. ], batch size: 54, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:42:33,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1653466.6666666667, ans=0.0 2023-11-21 20:42:51,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1653533.3333333333, ans=0.125 2023-11-21 20:42:55,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1653600.0, ans=0.0 2023-11-21 20:43:00,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1653600.0, ans=0.07 2023-11-21 20:43:05,357 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248050 2023-11-21 20:43:10,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1653666.6666666667, ans=0.125 2023-11-21 20:43:32,798 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7600, loss[loss=0.08247, simple_loss=0.1034, pruned_loss=0.02118, audio_tagging_loss=0.009597, over 16889.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09524, pruned_loss=0.01596, audio_tagging_loss=0.009221, over 3041940.40 frames. ], batch size: 64, lr: 3.25e-03, grad_scale: 32.0 2023-11-21 20:44:08,216 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.151e+01 8.758e+01 9.560e+01 1.334e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-21 20:44:08,378 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248100 2023-11-21 20:44:36,143 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7650, loss[loss=0.07558, simple_loss=0.09594, pruned_loss=0.02102, audio_tagging_loss=0.006593, over 14379.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09536, pruned_loss=0.01588, audio_tagging_loss=0.009255, over 3047567.30 frames. ], batch size: 54, lr: 3.25e-03, grad_scale: 32.0 2023-11-21 20:44:44,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1654133.3333333333, ans=0.125 2023-11-21 20:44:56,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1654200.0, ans=0.125 2023-11-21 20:45:11,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248150 2023-11-21 20:45:37,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1654400.0, ans=0.125 2023-11-21 20:45:37,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1654400.0, ans=0.0 2023-11-21 20:45:40,003 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7700, loss[loss=0.07381, simple_loss=0.08748, pruned_loss=0.01907, audio_tagging_loss=0.011, over 14855.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09564, pruned_loss=0.01601, audio_tagging_loss=0.009316, over 3047500.46 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 32.0 2023-11-21 20:45:42,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1654466.6666666667, ans=0.0 2023-11-21 20:45:45,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1654466.6666666667, ans=0.125 2023-11-21 20:45:47,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1654466.6666666667, ans=0.125 2023-11-21 20:45:50,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1654466.6666666667, ans=0.125 2023-11-21 20:45:50,580 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:45:51,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1654533.3333333333, ans=15.0 2023-11-21 20:45:55,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.72 vs. limit=12.0 2023-11-21 20:45:59,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1654533.3333333333, ans=0.1 2023-11-21 20:46:15,471 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.008e+01 8.521e+01 9.342e+01 1.167e+02, threshold=1.704e+02, percent-clipped=0.0 2023-11-21 20:46:15,615 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248200 2023-11-21 20:46:43,450 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7750, loss[loss=0.05821, simple_loss=0.08377, pruned_loss=0.009147, audio_tagging_loss=0.007174, over 15355.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.0956, pruned_loss=0.016, audio_tagging_loss=0.009366, over 3045138.44 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 32.0 2023-11-21 20:46:43,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1654800.0, ans=0.125 2023-11-21 20:47:19,557 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248250 2023-11-21 20:47:25,551 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.39 vs. limit=15.0 2023-11-21 20:47:37,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1655066.6666666667, ans=0.125 2023-11-21 20:47:46,964 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7800, loss[loss=0.06445, simple_loss=0.08606, pruned_loss=0.00929, audio_tagging_loss=0.01212, over 16212.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09621, pruned_loss=0.016, audio_tagging_loss=0.009382, over 3051076.17 frames. ], batch size: 62, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:47:55,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1655133.3333333333, ans=0.0 2023-11-21 20:47:57,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1655133.3333333333, ans=0.0 2023-11-21 20:47:59,242 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.51 vs. limit=22.5 2023-11-21 20:48:23,354 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248300 2023-11-21 20:48:24,377 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.033e+01 8.572e+01 9.400e+01 1.167e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-21 20:48:27,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1655333.3333333333, ans=0.125 2023-11-21 20:48:29,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1655333.3333333333, ans=0.2 2023-11-21 20:48:35,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1655333.3333333333, ans=0.0 2023-11-21 20:48:51,006 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7850, loss[loss=0.08134, simple_loss=0.106, pruned_loss=0.01992, audio_tagging_loss=0.00845, over 15326.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09582, pruned_loss=0.01597, audio_tagging_loss=0.009491, over 3046470.19 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:49:03,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.38 vs. limit=22.5 2023-11-21 20:49:12,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.58 vs. limit=15.0 2023-11-21 20:49:25,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248350 2023-11-21 20:49:35,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1655666.6666666667, ans=0.0 2023-11-21 20:49:49,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1655733.3333333333, ans=0.1 2023-11-21 20:49:53,827 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7900, loss[loss=0.1033, simple_loss=0.1416, pruned_loss=0.02544, audio_tagging_loss=0.007024, over 15757.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09682, pruned_loss=0.01622, audio_tagging_loss=0.009563, over 3052214.35 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0 2023-11-21 20:50:06,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1655866.6666666667, ans=0.125 2023-11-21 20:50:07,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1655866.6666666667, ans=0.125 2023-11-21 20:50:18,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1655933.3333333333, ans=0.125 2023-11-21 20:50:29,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248400 2023-11-21 20:50:32,497 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.166e+01 8.800e+01 9.422e+01 1.274e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-21 20:50:57,270 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 7950, loss[loss=0.07538, simple_loss=0.09972, pruned_loss=0.01637, audio_tagging_loss=0.009156, over 15142.00 frames. ], tot_loss[loss=0.07411, simple_loss=0.09633, pruned_loss=0.0163, audio_tagging_loss=0.009648, over 3048355.87 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0 2023-11-21 20:51:04,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1656133.3333333333, ans=0.2 2023-11-21 20:51:13,942 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 20:51:33,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1656266.6666666667, ans=0.2 2023-11-21 20:51:34,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248450 2023-11-21 20:51:34,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1656266.6666666667, ans=0.125 2023-11-21 20:51:41,859 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.43 vs. limit=15.0 2023-11-21 20:51:51,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1656400.0, ans=0.125 2023-11-21 20:52:02,353 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8000, loss[loss=0.07514, simple_loss=0.1079, pruned_loss=0.01216, audio_tagging_loss=0.009017, over 15285.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.09576, pruned_loss=0.01626, audio_tagging_loss=0.00976, over 3037871.53 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0 2023-11-21 20:52:25,356 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.47 vs. limit=15.0 2023-11-21 20:52:37,017 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248500 2023-11-21 20:52:39,309 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.016e+01 8.965e+01 9.786e+01 1.299e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-21 20:52:49,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1656666.6666666667, ans=0.09899494936611666 2023-11-21 20:52:51,249 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0 2023-11-21 20:53:01,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1656733.3333333333, ans=0.5 2023-11-21 20:53:05,617 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8050, loss[loss=0.05892, simple_loss=0.07824, pruned_loss=0.01104, audio_tagging_loss=0.008756, over 14997.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09557, pruned_loss=0.01621, audio_tagging_loss=0.00974, over 3035184.70 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 20:53:20,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1656866.6666666667, ans=0.125 2023-11-21 20:53:25,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1656866.6666666667, ans=0.0 2023-11-21 20:53:41,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248550 2023-11-21 20:54:05,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1657066.6666666667, ans=0.125 2023-11-21 20:54:08,554 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8100, loss[loss=0.09108, simple_loss=0.1192, pruned_loss=0.02305, audio_tagging_loss=0.008409, over 14499.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09522, pruned_loss=0.01622, audio_tagging_loss=0.009752, over 3041307.65 frames. ], batch size: 52, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 20:54:10,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1657133.3333333333, ans=0.125 2023-11-21 20:54:44,701 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248600 2023-11-21 20:54:45,124 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.08 vs. limit=10.0 2023-11-21 20:54:47,358 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.022e+01 8.595e+01 9.139e+01 1.165e+02, threshold=1.719e+02, percent-clipped=0.0 2023-11-21 20:55:12,712 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8150, loss[loss=0.07236, simple_loss=0.0971, pruned_loss=0.01709, audio_tagging_loss=0.006722, over 15592.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09651, pruned_loss=0.01641, audio_tagging_loss=0.00953, over 3038468.90 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 20:55:13,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1657466.6666666667, ans=10.0 2023-11-21 20:55:15,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1657466.6666666667, ans=0.95 2023-11-21 20:55:45,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1657600.0, ans=0.125 2023-11-21 20:55:47,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248650 2023-11-21 20:56:16,224 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8200, loss[loss=0.07439, simple_loss=0.0999, pruned_loss=0.01687, audio_tagging_loss=0.007562, over 15010.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09597, pruned_loss=0.01635, audio_tagging_loss=0.009444, over 3039311.60 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 20:56:16,284 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 20:56:50,619 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248700 2023-11-21 20:56:52,870 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.429e+01 8.006e+01 8.771e+01 9.726e+01 1.237e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 20:57:05,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1658066.6666666667, ans=0.125 2023-11-21 20:57:12,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1658066.6666666667, ans=0.2 2023-11-21 20:57:18,339 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8250, loss[loss=0.0833, simple_loss=0.1161, pruned_loss=0.01712, audio_tagging_loss=0.008136, over 15856.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09475, pruned_loss=0.01612, audio_tagging_loss=0.009418, over 3036633.69 frames. ], batch size: 57, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 20:57:18,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1658133.3333333333, ans=0.125 2023-11-21 20:57:21,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1658133.3333333333, ans=0.125 2023-11-21 20:57:23,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.85 vs. limit=22.5 2023-11-21 20:57:25,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1658133.3333333333, ans=15.0 2023-11-21 20:57:27,566 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.14 vs. limit=15.0 2023-11-21 20:57:40,888 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.48 vs. limit=22.5 2023-11-21 20:57:43,190 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=22.5 2023-11-21 20:57:52,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1658266.6666666667, ans=0.0 2023-11-21 20:57:54,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248750 2023-11-21 20:58:22,525 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8300, loss[loss=0.08195, simple_loss=0.1105, pruned_loss=0.01638, audio_tagging_loss=0.0103, over 17128.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.0954, pruned_loss=0.01632, audio_tagging_loss=0.009469, over 3044746.90 frames. ], batch size: 62, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 20:58:43,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1658533.3333333333, ans=0.125 2023-11-21 20:58:46,095 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:58:52,556 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.98 vs. limit=10.0 2023-11-21 20:58:58,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248800 2023-11-21 20:58:59,639 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 20:59:00,790 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.249e+01 8.152e+01 8.713e+01 9.319e+01 1.326e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 20:59:13,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1658733.3333333333, ans=0.125 2023-11-21 20:59:21,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1658733.3333333333, ans=0.125 2023-11-21 20:59:26,926 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8350, loss[loss=0.09556, simple_loss=0.1296, pruned_loss=0.02591, audio_tagging_loss=0.004867, over 15999.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09572, pruned_loss=0.01634, audio_tagging_loss=0.009459, over 3046177.10 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 20:59:37,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1658800.0, ans=0.125 2023-11-21 20:59:56,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1658933.3333333333, ans=0.0 2023-11-21 21:00:02,722 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248850 2023-11-21 21:00:02,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1658933.3333333333, ans=0.125 2023-11-21 21:00:08,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1659000.0, ans=0.0 2023-11-21 21:00:14,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1659000.0, ans=0.125 2023-11-21 21:00:30,757 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8400, loss[loss=0.06295, simple_loss=0.07972, pruned_loss=0.009409, audio_tagging_loss=0.01368, over 16286.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09435, pruned_loss=0.016, audio_tagging_loss=0.00946, over 3046665.95 frames. ], batch size: 64, lr: 3.24e-03, grad_scale: 32.0 2023-11-21 21:01:06,701 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248900 2023-11-21 21:01:10,209 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.422e+01 7.990e+01 8.773e+01 9.433e+01 1.119e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 21:01:28,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1659400.0, ans=0.125 2023-11-21 21:01:29,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1659400.0, ans=0.2 2023-11-21 21:01:33,134 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8450, loss[loss=0.0575, simple_loss=0.07124, pruned_loss=0.01132, audio_tagging_loss=0.01056, over 15260.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09386, pruned_loss=0.01584, audio_tagging_loss=0.009396, over 3050802.64 frames. ], batch size: 57, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:01:33,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1659466.6666666667, ans=0.125 2023-11-21 21:01:46,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1659533.3333333333, ans=0.125 2023-11-21 21:01:46,345 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:01:54,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1659533.3333333333, ans=0.2 2023-11-21 21:01:58,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1659600.0, ans=0.1 2023-11-21 21:02:02,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1659600.0, ans=0.0 2023-11-21 21:02:03,426 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.90 vs. limit=15.0 2023-11-21 21:02:08,950 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 248950 2023-11-21 21:02:15,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1659666.6666666667, ans=0.2 2023-11-21 21:02:18,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1659666.6666666667, ans=0.0 2023-11-21 21:02:22,217 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.05 vs. limit=12.0 2023-11-21 21:02:24,999 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.69 vs. limit=15.0 2023-11-21 21:02:32,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1659733.3333333333, ans=0.125 2023-11-21 21:02:36,332 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8500, loss[loss=0.08618, simple_loss=0.1177, pruned_loss=0.01872, audio_tagging_loss=0.008615, over 15588.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09405, pruned_loss=0.01579, audio_tagging_loss=0.009425, over 3047691.32 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:02:57,649 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.56 vs. limit=15.0 2023-11-21 21:03:10,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1659933.3333333333, ans=0.125 2023-11-21 21:03:10,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1659933.3333333333, ans=0.0 2023-11-21 21:03:12,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249000 2023-11-21 21:03:15,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1660000.0, ans=0.0 2023-11-21 21:03:16,327 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.238e+01 7.932e+01 8.481e+01 9.270e+01 1.196e+02, threshold=1.696e+02, percent-clipped=0.0 2023-11-21 21:03:26,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1660066.6666666667, ans=0.125 2023-11-21 21:03:40,987 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8550, loss[loss=0.05428, simple_loss=0.06648, pruned_loss=0.01135, audio_tagging_loss=0.00969, over 15933.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09405, pruned_loss=0.01582, audio_tagging_loss=0.009505, over 3049170.82 frames. ], batch size: 62, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:03:46,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1660133.3333333333, ans=0.1 2023-11-21 21:03:54,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1660200.0, ans=0.5 2023-11-21 21:04:00,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1660200.0, ans=0.125 2023-11-21 21:04:09,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1660266.6666666667, ans=0.125 2023-11-21 21:04:16,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249050 2023-11-21 21:04:20,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1660333.3333333333, ans=0.1 2023-11-21 21:04:26,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1660333.3333333333, ans=0.2 2023-11-21 21:04:30,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1660400.0, ans=0.125 2023-11-21 21:04:30,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1660400.0, ans=0.125 2023-11-21 21:04:37,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1660400.0, ans=0.1 2023-11-21 21:04:43,480 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8600, loss[loss=0.08519, simple_loss=0.1126, pruned_loss=0.01743, audio_tagging_loss=0.01143, over 16135.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09435, pruned_loss=0.01594, audio_tagging_loss=0.009557, over 3047106.37 frames. ], batch size: 61, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:04:50,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1660466.6666666667, ans=0.0 2023-11-21 21:05:01,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1660533.3333333333, ans=0.125 2023-11-21 21:05:07,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.43 vs. limit=15.0 2023-11-21 21:05:20,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249100 2023-11-21 21:05:24,798 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.123e+01 8.590e+01 9.454e+01 1.253e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-21 21:05:38,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1660733.3333333333, ans=0.125 2023-11-21 21:05:47,725 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8650, loss[loss=0.07081, simple_loss=0.08281, pruned_loss=0.01941, audio_tagging_loss=0.009996, over 14190.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09371, pruned_loss=0.01587, audio_tagging_loss=0.009587, over 3033953.42 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:05:58,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1660800.0, ans=0.2 2023-11-21 21:06:15,030 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:06:23,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249150 2023-11-21 21:06:23,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1660933.3333333333, ans=0.0 2023-11-21 21:06:23,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1660933.3333333333, ans=0.125 2023-11-21 21:06:27,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0 2023-11-21 21:06:51,498 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8700, loss[loss=0.07886, simple_loss=0.107, pruned_loss=0.01737, audio_tagging_loss=0.007976, over 15217.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09444, pruned_loss=0.01596, audio_tagging_loss=0.009601, over 3040011.03 frames. ], batch size: 59, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:06:59,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1661133.3333333333, ans=0.0 2023-11-21 21:07:16,950 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.02 vs. limit=12.0 2023-11-21 21:07:27,363 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249200 2023-11-21 21:07:32,749 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.120e+01 8.963e+01 9.729e+01 1.356e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-21 21:07:55,270 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8750, loss[loss=0.06823, simple_loss=0.08703, pruned_loss=0.01602, audio_tagging_loss=0.008686, over 14509.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09473, pruned_loss=0.01618, audio_tagging_loss=0.009807, over 3047808.98 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:08:04,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1661466.6666666667, ans=0.2 2023-11-21 21:08:10,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.21 vs. limit=15.0 2023-11-21 21:08:31,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249250 2023-11-21 21:08:31,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=1661600.0, ans=15.0 2023-11-21 21:08:35,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1661666.6666666667, ans=0.0 2023-11-21 21:08:44,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1661666.6666666667, ans=0.125 2023-11-21 21:08:52,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1661733.3333333333, ans=0.0 2023-11-21 21:08:56,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1661733.3333333333, ans=0.125 2023-11-21 21:08:59,671 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8800, loss[loss=0.1065, simple_loss=0.1429, pruned_loss=0.02742, audio_tagging_loss=0.007668, over 15603.00 frames. ], tot_loss[loss=0.074, simple_loss=0.0955, pruned_loss=0.01637, audio_tagging_loss=0.009883, over 3047306.57 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:09:02,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1661800.0, ans=0.0 2023-11-21 21:09:35,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249300 2023-11-21 21:09:40,734 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.249e+01 8.864e+01 9.702e+01 1.215e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-21 21:09:41,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1662000.0, ans=0.2 2023-11-21 21:10:03,881 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8850, loss[loss=0.08535, simple_loss=0.1116, pruned_loss=0.01939, audio_tagging_loss=0.01013, over 16265.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.0959, pruned_loss=0.01631, audio_tagging_loss=0.009788, over 3048135.23 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:10:10,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1662133.3333333333, ans=0.125 2023-11-21 21:10:12,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1662133.3333333333, ans=0.0 2023-11-21 21:10:15,540 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:10:31,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1662266.6666666667, ans=0.1 2023-11-21 21:10:33,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1662266.6666666667, ans=0.1 2023-11-21 21:10:39,606 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249350 2023-11-21 21:11:07,403 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8900, loss[loss=0.07382, simple_loss=0.09696, pruned_loss=0.01458, audio_tagging_loss=0.01077, over 15522.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.0961, pruned_loss=0.01615, audio_tagging_loss=0.00959, over 3049757.80 frames. ], batch size: 57, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:11:13,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1662466.6666666667, ans=0.2 2023-11-21 21:11:43,062 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249400 2023-11-21 21:11:44,852 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.06 vs. limit=22.5 2023-11-21 21:11:49,419 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.502e+01 8.199e+01 8.684e+01 9.586e+01 1.559e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 21:11:52,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1662666.6666666667, ans=0.04949747468305833 2023-11-21 21:11:58,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1662733.3333333333, ans=0.125 2023-11-21 21:12:12,032 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 8950, loss[loss=0.08891, simple_loss=0.1205, pruned_loss=0.02276, audio_tagging_loss=0.005887, over 14912.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09656, pruned_loss=0.01616, audio_tagging_loss=0.009506, over 3045944.32 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:12:26,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1662866.6666666667, ans=0.0 2023-11-21 21:12:29,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1662866.6666666667, ans=0.2 2023-11-21 21:12:30,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1662866.6666666667, ans=22.5 2023-11-21 21:12:47,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249450 2023-11-21 21:12:56,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.05 vs. limit=22.5 2023-11-21 21:13:15,742 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9000, loss[loss=0.06989, simple_loss=0.08353, pruned_loss=0.01821, audio_tagging_loss=0.009912, over 14419.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09603, pruned_loss=0.01617, audio_tagging_loss=0.009435, over 3047612.13 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:13:15,743 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 21:13:42,308 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4508, 2.9878, 3.5783, 3.3474], device='cuda:3') 2023-11-21 21:13:56,849 INFO [train_asr.py:1253] (3/4) Epoch 21, validation: loss=0.06003, simple_loss=0.05194, pruned_loss=0.005168, audio_tagging_loss=0.0289, over 4681554.00 frames. 2023-11-21 21:13:56,850 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 21:14:23,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1663266.6666666667, ans=0.125 2023-11-21 21:14:32,763 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249500 2023-11-21 21:14:37,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1663333.3333333333, ans=0.125 2023-11-21 21:14:38,733 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 7.983e+01 8.777e+01 1.012e+02 1.174e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-21 21:14:42,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1663333.3333333333, ans=0.125 2023-11-21 21:14:49,305 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.71 vs. limit=6.0 2023-11-21 21:14:57,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1663400.0, ans=0.125 2023-11-21 21:15:01,135 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9050, loss[loss=0.0809, simple_loss=0.1004, pruned_loss=0.02136, audio_tagging_loss=0.009355, over 14145.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09542, pruned_loss=0.01616, audio_tagging_loss=0.009544, over 3043885.52 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:15:06,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1663466.6666666667, ans=0.125 2023-11-21 21:15:10,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1663466.6666666667, ans=0.125 2023-11-21 21:15:23,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1663533.3333333333, ans=0.125 2023-11-21 21:15:36,757 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249550 2023-11-21 21:16:04,953 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9100, loss[loss=0.08224, simple_loss=0.1009, pruned_loss=0.02154, audio_tagging_loss=0.01025, over 16687.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09516, pruned_loss=0.01609, audio_tagging_loss=0.009536, over 3048224.03 frames. ], batch size: 64, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:16:07,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1663800.0, ans=0.0 2023-11-21 21:16:12,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1663800.0, ans=0.125 2023-11-21 21:16:15,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.79 vs. limit=12.0 2023-11-21 21:16:40,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249600 2023-11-21 21:16:41,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1663933.3333333333, ans=0.0 2023-11-21 21:16:47,972 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.138e+01 8.810e+01 9.417e+01 1.142e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-21 21:17:08,719 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9150, loss[loss=0.06288, simple_loss=0.07519, pruned_loss=0.01282, audio_tagging_loss=0.01246, over 14622.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09573, pruned_loss=0.0162, audio_tagging_loss=0.009456, over 3043683.80 frames. ], batch size: 57, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:17:15,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1664133.3333333333, ans=0.125 2023-11-21 21:17:29,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1664200.0, ans=0.125 2023-11-21 21:17:33,408 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.13 vs. limit=15.0 2023-11-21 21:17:45,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249650 2023-11-21 21:17:55,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1664333.3333333333, ans=0.0 2023-11-21 21:18:04,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.27 vs. limit=15.0 2023-11-21 21:18:14,538 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9200, loss[loss=0.05938, simple_loss=0.07957, pruned_loss=0.01146, audio_tagging_loss=0.008128, over 15060.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09529, pruned_loss=0.01622, audio_tagging_loss=0.009402, over 3037212.60 frames. ], batch size: 59, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:18:30,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1664533.3333333333, ans=0.125 2023-11-21 21:18:30,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn2.whiten.whitening_limit, batch_count=1664533.3333333333, ans=22.5 2023-11-21 21:18:50,556 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249700 2023-11-21 21:18:53,157 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:18:56,914 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.394e+01 8.125e+01 8.694e+01 9.340e+01 1.162e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 21:19:13,719 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.572e-03 2023-11-21 21:19:19,568 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9250, loss[loss=0.07546, simple_loss=0.0951, pruned_loss=0.0193, audio_tagging_loss=0.008619, over 15027.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09438, pruned_loss=0.01589, audio_tagging_loss=0.009417, over 3046520.17 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:19:28,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1664800.0, ans=0.125 2023-11-21 21:19:46,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1664933.3333333333, ans=0.0 2023-11-21 21:19:49,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1664933.3333333333, ans=0.0 2023-11-21 21:19:55,954 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249750 2023-11-21 21:20:23,917 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9300, loss[loss=0.06337, simple_loss=0.07987, pruned_loss=0.01412, audio_tagging_loss=0.009313, over 14431.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09379, pruned_loss=0.01574, audio_tagging_loss=0.009445, over 3048104.20 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:20:25,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.12 vs. limit=15.0 2023-11-21 21:20:32,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1665133.3333333333, ans=0.0 2023-11-21 21:20:37,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1665200.0, ans=0.0 2023-11-21 21:20:46,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1665200.0, ans=0.125 2023-11-21 21:20:47,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1665200.0, ans=0.0 2023-11-21 21:21:00,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249800 2023-11-21 21:21:02,154 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.26 vs. limit=15.0 2023-11-21 21:21:06,630 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.134e+01 8.648e+01 9.186e+01 1.314e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-21 21:21:08,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1665333.3333333333, ans=0.0 2023-11-21 21:21:29,037 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9350, loss[loss=0.09064, simple_loss=0.1173, pruned_loss=0.02244, audio_tagging_loss=0.009582, over 14861.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09414, pruned_loss=0.01575, audio_tagging_loss=0.009419, over 3048610.91 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:21:46,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1665533.3333333333, ans=0.1 2023-11-21 21:21:52,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1665533.3333333333, ans=0.125 2023-11-21 21:22:04,360 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249850 2023-11-21 21:22:05,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1665666.6666666667, ans=0.0 2023-11-21 21:22:16,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1665666.6666666667, ans=0.1 2023-11-21 21:22:27,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1665733.3333333333, ans=0.125 2023-11-21 21:22:27,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1665733.3333333333, ans=0.125 2023-11-21 21:22:31,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1665733.3333333333, ans=0.125 2023-11-21 21:22:33,837 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9400, loss[loss=0.09093, simple_loss=0.117, pruned_loss=0.02157, audio_tagging_loss=0.01085, over 14111.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09357, pruned_loss=0.01588, audio_tagging_loss=0.009557, over 3047504.06 frames. ], batch size: 52, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:23:03,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1665933.3333333333, ans=0.0 2023-11-21 21:23:09,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249900 2023-11-21 21:23:16,397 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.499e+01 7.941e+01 8.716e+01 9.449e+01 2.230e+02, threshold=1.743e+02, percent-clipped=1.0 2023-11-21 21:23:35,600 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:23:38,008 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9450, loss[loss=0.06561, simple_loss=0.08538, pruned_loss=0.0112, audio_tagging_loss=0.01172, over 15369.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09394, pruned_loss=0.01582, audio_tagging_loss=0.009602, over 3053780.76 frames. ], batch size: 59, lr: 3.24e-03, grad_scale: 16.0 2023-11-21 21:23:44,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1666133.3333333333, ans=0.5 2023-11-21 21:23:46,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1666133.3333333333, ans=0.04949747468305833 2023-11-21 21:23:58,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1666200.0, ans=0.035 2023-11-21 21:24:05,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1666266.6666666667, ans=0.2 2023-11-21 21:24:08,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1666266.6666666667, ans=0.1 2023-11-21 21:24:14,521 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 249950 2023-11-21 21:24:20,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1666333.3333333333, ans=0.0 2023-11-21 21:24:42,631 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9500, loss[loss=0.07192, simple_loss=0.09258, pruned_loss=0.01583, audio_tagging_loss=0.009806, over 15195.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09393, pruned_loss=0.0158, audio_tagging_loss=0.009652, over 3050737.23 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:24:59,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1666533.3333333333, ans=0.0 2023-11-21 21:25:05,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1666533.3333333333, ans=0.125 2023-11-21 21:25:05,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1666533.3333333333, ans=0.2 2023-11-21 21:25:06,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1666533.3333333333, ans=0.1 2023-11-21 21:25:08,101 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.31 vs. limit=15.0 2023-11-21 21:25:08,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1666600.0, ans=0.0 2023-11-21 21:25:11,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1666600.0, ans=0.0 2023-11-21 21:25:12,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1666600.0, ans=0.2 2023-11-21 21:25:18,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250000 2023-11-21 21:25:26,604 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.269e+01 8.539e+01 8.975e+01 9.633e+01 1.238e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-21 21:25:31,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1666666.6666666667, ans=0.5 2023-11-21 21:25:39,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1666733.3333333333, ans=0.2 2023-11-21 21:25:48,001 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9550, loss[loss=0.05341, simple_loss=0.05179, pruned_loss=0.009756, audio_tagging_loss=0.01776, over 14208.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09367, pruned_loss=0.0158, audio_tagging_loss=0.009769, over 3049429.90 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 8.0 2023-11-21 21:25:57,958 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.23 vs. limit=15.0 2023-11-21 21:26:06,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1666866.6666666667, ans=0.0 2023-11-21 21:26:17,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1666933.3333333333, ans=0.1 2023-11-21 21:26:19,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1666933.3333333333, ans=0.125 2023-11-21 21:26:24,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250050 2023-11-21 21:26:53,399 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9600, loss[loss=0.05513, simple_loss=0.06369, pruned_loss=0.0127, audio_tagging_loss=0.01059, over 14560.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.094, pruned_loss=0.01585, audio_tagging_loss=0.009816, over 3046013.16 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:27:10,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1667200.0, ans=0.125 2023-11-21 21:27:15,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1667200.0, ans=0.125 2023-11-21 21:27:19,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1667266.6666666667, ans=0.0 2023-11-21 21:27:30,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250100 2023-11-21 21:27:38,184 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 8.305e+01 8.780e+01 9.351e+01 2.082e+02, threshold=1.756e+02, percent-clipped=1.0 2023-11-21 21:27:42,524 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.09 vs. limit=22.5 2023-11-21 21:27:58,568 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9650, loss[loss=0.09473, simple_loss=0.1232, pruned_loss=0.02465, audio_tagging_loss=0.008499, over 15011.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09338, pruned_loss=0.01552, audio_tagging_loss=0.009742, over 3046220.05 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:28:18,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1667533.3333333333, ans=0.125 2023-11-21 21:28:34,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250150 2023-11-21 21:28:49,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1667733.3333333333, ans=0.0 2023-11-21 21:29:01,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0 2023-11-21 21:29:03,962 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9700, loss[loss=0.06733, simple_loss=0.08469, pruned_loss=0.01487, audio_tagging_loss=0.01012, over 14139.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09362, pruned_loss=0.01543, audio_tagging_loss=0.009621, over 3040235.19 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:29:05,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1667800.0, ans=0.0 2023-11-21 21:29:08,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1667800.0, ans=0.125 2023-11-21 21:29:10,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1667800.0, ans=0.125 2023-11-21 21:29:40,200 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250200 2023-11-21 21:29:47,787 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.250e+01 8.691e+01 9.260e+01 1.203e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-21 21:30:07,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1668133.3333333333, ans=0.0 2023-11-21 21:30:09,301 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9750, loss[loss=0.0528, simple_loss=0.07022, pruned_loss=0.012, audio_tagging_loss=0.005683, over 15087.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.094, pruned_loss=0.01572, audio_tagging_loss=0.009464, over 3034776.77 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:30:10,944 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.026e-02 2023-11-21 21:30:12,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1668133.3333333333, ans=0.2 2023-11-21 21:30:46,077 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250250 2023-11-21 21:31:00,870 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.10 vs. limit=15.0 2023-11-21 21:31:04,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1668400.0, ans=0.125 2023-11-21 21:31:09,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1668400.0, ans=0.2 2023-11-21 21:31:13,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1668466.6666666667, ans=0.0 2023-11-21 21:31:13,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.75 vs. limit=15.0 2023-11-21 21:31:14,282 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9800, loss[loss=0.07652, simple_loss=0.1047, pruned_loss=0.01622, audio_tagging_loss=0.007932, over 14436.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09405, pruned_loss=0.01566, audio_tagging_loss=0.00937, over 3036619.55 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:31:27,910 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.99 vs. limit=10.0 2023-11-21 21:31:35,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1668533.3333333333, ans=0.0 2023-11-21 21:31:50,378 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250300 2023-11-21 21:31:54,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.20 vs. limit=12.0 2023-11-21 21:31:57,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1668666.6666666667, ans=0.125 2023-11-21 21:31:58,396 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 7.987e+01 8.487e+01 9.363e+01 1.387e+02, threshold=1.697e+02, percent-clipped=0.0 2023-11-21 21:31:59,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.35 vs. limit=22.5 2023-11-21 21:32:04,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.83 vs. limit=22.5 2023-11-21 21:32:11,908 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:32:19,231 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9850, loss[loss=0.0742, simple_loss=0.09474, pruned_loss=0.01422, audio_tagging_loss=0.01261, over 15406.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.0955, pruned_loss=0.01606, audio_tagging_loss=0.009171, over 3041704.72 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:32:20,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1668800.0, ans=0.125 2023-11-21 21:32:23,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1668800.0, ans=0.0 2023-11-21 21:32:55,321 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250350 2023-11-21 21:33:00,891 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:33:03,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1669000.0, ans=0.125 2023-11-21 21:33:12,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.07 vs. limit=15.0 2023-11-21 21:33:14,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1669066.6666666667, ans=0.0 2023-11-21 21:33:23,700 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9900, loss[loss=0.06994, simple_loss=0.09284, pruned_loss=0.01403, audio_tagging_loss=0.009493, over 14832.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09533, pruned_loss=0.016, audio_tagging_loss=0.009199, over 3034213.85 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:33:38,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.24 vs. limit=15.0 2023-11-21 21:34:00,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250400 2023-11-21 21:34:00,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1669266.6666666667, ans=0.125 2023-11-21 21:34:02,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1669333.3333333333, ans=0.125 2023-11-21 21:34:08,494 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.181e+01 8.919e+01 9.368e+01 2.171e+02, threshold=1.784e+02, percent-clipped=1.0 2023-11-21 21:34:17,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1669400.0, ans=0.0 2023-11-21 21:34:22,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1669400.0, ans=0.0 2023-11-21 21:34:28,649 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 9950, loss[loss=0.077, simple_loss=0.1012, pruned_loss=0.01827, audio_tagging_loss=0.008149, over 15700.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09533, pruned_loss=0.016, audio_tagging_loss=0.009317, over 3041991.74 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:34:38,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1669466.6666666667, ans=0.125 2023-11-21 21:34:56,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1669600.0, ans=0.125 2023-11-21 21:35:04,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250450 2023-11-21 21:35:24,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1669733.3333333333, ans=0.125 2023-11-21 21:35:27,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1669733.3333333333, ans=0.1 2023-11-21 21:35:29,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1669733.3333333333, ans=0.125 2023-11-21 21:35:32,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1669800.0, ans=0.125 2023-11-21 21:35:33,063 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10000, loss[loss=0.07904, simple_loss=0.1072, pruned_loss=0.01749, audio_tagging_loss=0.007948, over 16686.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09453, pruned_loss=0.01574, audio_tagging_loss=0.009325, over 3047844.56 frames. ], batch size: 60, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:36:09,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250500 2023-11-21 21:36:10,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1670000.0, ans=0.125 2023-11-21 21:36:16,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1670000.0, ans=0.0 2023-11-21 21:36:17,285 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.484e+01 8.244e+01 8.958e+01 9.849e+01 1.581e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-21 21:36:37,531 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10050, loss[loss=0.06729, simple_loss=0.08762, pruned_loss=0.01338, audio_tagging_loss=0.0101, over 15100.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09507, pruned_loss=0.01587, audio_tagging_loss=0.009433, over 3050057.29 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:37:13,267 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250550 2023-11-21 21:37:35,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1670400.0, ans=0.0 2023-11-21 21:37:40,401 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10100, loss[loss=0.05846, simple_loss=0.06819, pruned_loss=0.01085, audio_tagging_loss=0.01351, over 15261.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09493, pruned_loss=0.01583, audio_tagging_loss=0.009506, over 3054245.08 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:37:53,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1670533.3333333333, ans=0.0 2023-11-21 21:38:09,967 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0 2023-11-21 21:38:16,201 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.33 vs. limit=12.0 2023-11-21 21:38:16,915 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250600 2023-11-21 21:38:17,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.86 vs. limit=15.0 2023-11-21 21:38:25,702 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.286e+01 8.086e+01 8.639e+01 9.260e+01 1.282e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-21 21:38:31,771 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:38:35,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1670733.3333333333, ans=0.0 2023-11-21 21:38:39,444 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.82 vs. limit=12.0 2023-11-21 21:38:45,758 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10150, loss[loss=0.04726, simple_loss=0.05751, pruned_loss=0.007849, audio_tagging_loss=0.01065, over 14424.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.09413, pruned_loss=0.01576, audio_tagging_loss=0.00968, over 3050136.58 frames. ], batch size: 54, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:38:46,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1670800.0, ans=0.0 2023-11-21 21:39:14,468 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:39:20,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250650 2023-11-21 21:39:30,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1671000.0, ans=0.0 2023-11-21 21:39:32,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1671000.0, ans=0.2 2023-11-21 21:39:48,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1671066.6666666667, ans=0.2 2023-11-21 21:39:49,974 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10200, loss[loss=0.07222, simple_loss=0.1048, pruned_loss=0.01126, audio_tagging_loss=0.00854, over 14920.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.0947, pruned_loss=0.01577, audio_tagging_loss=0.009651, over 3052867.48 frames. ], batch size: 53, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:39:56,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1671133.3333333333, ans=0.0 2023-11-21 21:40:04,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1671200.0, ans=0.0 2023-11-21 21:40:12,565 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:40:19,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1671266.6666666667, ans=0.125 2023-11-21 21:40:22,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1671266.6666666667, ans=0.125 2023-11-21 21:40:26,533 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250700 2023-11-21 21:40:35,580 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.067e+01 8.565e+01 9.348e+01 1.300e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-21 21:40:38,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1671333.3333333333, ans=0.125 2023-11-21 21:40:42,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1671400.0, ans=0.125 2023-11-21 21:40:43,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1671400.0, ans=0.125 2023-11-21 21:40:54,051 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10250, loss[loss=0.06971, simple_loss=0.09149, pruned_loss=0.01413, audio_tagging_loss=0.009832, over 15054.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09571, pruned_loss=0.01604, audio_tagging_loss=0.009701, over 3056063.84 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:40:54,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1671466.6666666667, ans=0.1 2023-11-21 21:41:02,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1671466.6666666667, ans=0.125 2023-11-21 21:41:15,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1671533.3333333333, ans=0.0 2023-11-21 21:41:22,433 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.30 vs. limit=12.0 2023-11-21 21:41:30,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250750 2023-11-21 21:41:58,077 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10300, loss[loss=0.09486, simple_loss=0.1284, pruned_loss=0.02386, audio_tagging_loss=0.006798, over 16071.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09387, pruned_loss=0.01583, audio_tagging_loss=0.009959, over 3057173.33 frames. ], batch size: 60, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:41:58,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1671800.0, ans=0.125 2023-11-21 21:42:10,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1671866.6666666667, ans=0.2 2023-11-21 21:42:16,961 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.79 vs. limit=5.0 2023-11-21 21:42:29,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1671933.3333333333, ans=0.07 2023-11-21 21:42:33,100 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250800 2023-11-21 21:42:42,615 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.124e+01 8.842e+01 9.369e+01 1.260e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 21:42:49,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1672066.6666666667, ans=0.5 2023-11-21 21:42:51,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.40 vs. limit=22.5 2023-11-21 21:42:59,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1672066.6666666667, ans=0.2 2023-11-21 21:43:03,274 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10350, loss[loss=0.08362, simple_loss=0.1108, pruned_loss=0.01784, audio_tagging_loss=0.01038, over 14942.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.0957, pruned_loss=0.01613, audio_tagging_loss=0.009894, over 3056903.53 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:43:07,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1672133.3333333333, ans=0.2 2023-11-21 21:43:20,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1672200.0, ans=0.125 2023-11-21 21:43:26,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.07 vs. limit=15.0 2023-11-21 21:43:29,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1672266.6666666667, ans=0.1 2023-11-21 21:43:39,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250850 2023-11-21 21:43:43,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1672333.3333333333, ans=0.125 2023-11-21 21:43:45,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1672333.3333333333, ans=0.0 2023-11-21 21:44:07,045 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10400, loss[loss=0.06408, simple_loss=0.08175, pruned_loss=0.01234, audio_tagging_loss=0.01086, over 14468.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09577, pruned_loss=0.01617, audio_tagging_loss=0.01001, over 3057893.56 frames. ], batch size: 53, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:44:11,197 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:44:13,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1672466.6666666667, ans=0.0 2023-11-21 21:44:17,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1672466.6666666667, ans=0.0 2023-11-21 21:44:20,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.24 vs. limit=15.0 2023-11-21 21:44:24,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.30 vs. limit=15.0 2023-11-21 21:44:42,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1672600.0, ans=0.1 2023-11-21 21:44:43,939 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250900 2023-11-21 21:44:44,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0 2023-11-21 21:44:46,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1672666.6666666667, ans=0.2 2023-11-21 21:44:47,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1672666.6666666667, ans=0.07 2023-11-21 21:44:51,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1672666.6666666667, ans=0.125 2023-11-21 21:44:52,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.219e+01 8.868e+01 9.725e+01 1.469e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-21 21:44:57,066 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.76 vs. limit=15.0 2023-11-21 21:45:02,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1672733.3333333333, ans=0.0 2023-11-21 21:45:08,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1672733.3333333333, ans=0.0 2023-11-21 21:45:09,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1672733.3333333333, ans=0.125 2023-11-21 21:45:12,301 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10450, loss[loss=0.07633, simple_loss=0.104, pruned_loss=0.01463, audio_tagging_loss=0.009733, over 14856.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09534, pruned_loss=0.01626, audio_tagging_loss=0.009902, over 3052960.36 frames. ], batch size: 54, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:45:12,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1672800.0, ans=0.1 2023-11-21 21:45:17,913 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.53 vs. limit=10.0 2023-11-21 21:45:34,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1672866.6666666667, ans=0.125 2023-11-21 21:45:47,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1672933.3333333333, ans=0.1 2023-11-21 21:45:48,158 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 250950 2023-11-21 21:45:50,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1673000.0, ans=0.125 2023-11-21 21:45:55,533 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.68 vs. limit=12.0 2023-11-21 21:46:17,690 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10500, loss[loss=0.04901, simple_loss=0.06296, pruned_loss=0.009845, audio_tagging_loss=0.007689, over 14479.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09449, pruned_loss=0.01603, audio_tagging_loss=0.009772, over 3057195.74 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:46:23,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1673133.3333333333, ans=0.125 2023-11-21 21:46:28,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1673133.3333333333, ans=0.07 2023-11-21 21:46:45,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1673266.6666666667, ans=0.0 2023-11-21 21:46:54,139 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251000 2023-11-21 21:46:55,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1673333.3333333333, ans=0.1 2023-11-21 21:47:05,263 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.294e+01 8.913e+01 9.437e+01 1.275e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-21 21:47:06,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1673333.3333333333, ans=0.0 2023-11-21 21:47:16,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1673400.0, ans=0.2 2023-11-21 21:47:17,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1673400.0, ans=0.125 2023-11-21 21:47:23,406 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10550, loss[loss=0.09761, simple_loss=0.1324, pruned_loss=0.02386, audio_tagging_loss=0.007549, over 15650.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09435, pruned_loss=0.01585, audio_tagging_loss=0.009697, over 3058704.78 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:47:51,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.20 vs. limit=15.0 2023-11-21 21:47:58,662 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.53 vs. limit=15.0 2023-11-21 21:48:00,766 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251050 2023-11-21 21:48:07,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1673666.6666666667, ans=0.0 2023-11-21 21:48:10,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1673666.6666666667, ans=0.125 2023-11-21 21:48:13,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1673666.6666666667, ans=0.125 2023-11-21 21:48:23,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1673733.3333333333, ans=0.0 2023-11-21 21:48:27,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1673800.0, ans=0.2 2023-11-21 21:48:28,702 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10600, loss[loss=0.09156, simple_loss=0.1122, pruned_loss=0.02752, audio_tagging_loss=0.007922, over 16588.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09537, pruned_loss=0.01594, audio_tagging_loss=0.009405, over 3057071.03 frames. ], batch size: 62, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:48:34,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1673800.0, ans=0.0 2023-11-21 21:48:40,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1673866.6666666667, ans=0.2 2023-11-21 21:48:49,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1673866.6666666667, ans=0.2 2023-11-21 21:49:05,115 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251100 2023-11-21 21:49:14,678 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.69 vs. limit=15.0 2023-11-21 21:49:15,227 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.229e+01 8.713e+01 9.313e+01 1.246e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-21 21:49:18,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1674000.0, ans=0.125 2023-11-21 21:49:21,016 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:49:24,134 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.22 vs. limit=15.0 2023-11-21 21:49:33,274 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10650, loss[loss=0.06158, simple_loss=0.07051, pruned_loss=0.01304, audio_tagging_loss=0.01328, over 14655.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09483, pruned_loss=0.01571, audio_tagging_loss=0.009396, over 3054667.61 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:49:45,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1674200.0, ans=0.125 2023-11-21 21:49:52,450 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.53 vs. limit=15.0 2023-11-21 21:50:00,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1674266.6666666667, ans=0.1 2023-11-21 21:50:09,595 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251150 2023-11-21 21:50:10,027 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.73 vs. limit=15.0 2023-11-21 21:50:18,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0 2023-11-21 21:50:32,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1674400.0, ans=0.1 2023-11-21 21:50:38,635 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10700, loss[loss=0.07227, simple_loss=0.08996, pruned_loss=0.0185, audio_tagging_loss=0.008788, over 15146.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09488, pruned_loss=0.01562, audio_tagging_loss=0.00934, over 3054855.08 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:51:14,763 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251200 2023-11-21 21:51:22,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1674666.6666666667, ans=0.125 2023-11-21 21:51:25,543 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 8.099e+01 8.796e+01 9.421e+01 3.240e+02, threshold=1.759e+02, percent-clipped=1.0 2023-11-21 21:51:43,416 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10750, loss[loss=0.05027, simple_loss=0.06922, pruned_loss=0.008, audio_tagging_loss=0.007661, over 14507.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.0955, pruned_loss=0.01573, audio_tagging_loss=0.009213, over 3062482.99 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:51:51,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1674800.0, ans=0.0 2023-11-21 21:51:51,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1674800.0, ans=0.125 2023-11-21 21:52:17,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.96 vs. limit=15.0 2023-11-21 21:52:19,728 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251250 2023-11-21 21:52:44,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1675066.6666666667, ans=0.125 2023-11-21 21:52:47,589 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10800, loss[loss=0.06971, simple_loss=0.08862, pruned_loss=0.01817, audio_tagging_loss=0.007228, over 15064.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09632, pruned_loss=0.01596, audio_tagging_loss=0.009186, over 3057072.84 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:52:51,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1675133.3333333333, ans=0.95 2023-11-21 21:52:53,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1675133.3333333333, ans=0.125 2023-11-21 21:53:23,709 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251300 2023-11-21 21:53:23,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1675266.6666666667, ans=0.125 2023-11-21 21:53:34,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.266e+01 8.735e+01 9.337e+01 1.123e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-21 21:53:34,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.90 vs. limit=15.0 2023-11-21 21:53:39,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0 2023-11-21 21:53:52,954 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10850, loss[loss=0.08349, simple_loss=0.1149, pruned_loss=0.01972, audio_tagging_loss=0.006343, over 16087.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09655, pruned_loss=0.01613, audio_tagging_loss=0.009208, over 3053703.33 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:53:56,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1675466.6666666667, ans=0.1 2023-11-21 21:54:00,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1675466.6666666667, ans=0.125 2023-11-21 21:54:28,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251350 2023-11-21 21:54:52,146 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.90 vs. limit=22.5 2023-11-21 21:54:52,757 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:54:56,560 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10900, loss[loss=0.08768, simple_loss=0.1186, pruned_loss=0.02182, audio_tagging_loss=0.006552, over 16101.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09635, pruned_loss=0.01622, audio_tagging_loss=0.009308, over 3058886.63 frames. ], batch size: 60, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:55:27,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=22.5 2023-11-21 21:55:30,768 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.554e-03 2023-11-21 21:55:33,594 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251400 2023-11-21 21:55:41,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1676000.0, ans=0.125 2023-11-21 21:55:43,674 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.153e+01 8.721e+01 9.251e+01 1.356e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-21 21:56:02,094 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 10950, loss[loss=0.06159, simple_loss=0.07573, pruned_loss=0.01277, audio_tagging_loss=0.01096, over 14360.00 frames. ], tot_loss[loss=0.07285, simple_loss=0.09526, pruned_loss=0.01582, audio_tagging_loss=0.009402, over 3055986.65 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 32.0 2023-11-21 21:56:08,804 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:56:13,154 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0 2023-11-21 21:56:13,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1676200.0, ans=0.125 2023-11-21 21:56:26,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1676200.0, ans=0.125 2023-11-21 21:56:32,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1676266.6666666667, ans=0.0 2023-11-21 21:56:32,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1676266.6666666667, ans=0.125 2023-11-21 21:56:33,531 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.98 vs. limit=15.0 2023-11-21 21:56:37,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251450 2023-11-21 21:56:53,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1676400.0, ans=0.0 2023-11-21 21:57:06,331 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11000, loss[loss=0.0689, simple_loss=0.09274, pruned_loss=0.01276, audio_tagging_loss=0.009777, over 15573.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.0946, pruned_loss=0.01557, audio_tagging_loss=0.009418, over 3053963.96 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:57:09,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1676466.6666666667, ans=0.2 2023-11-21 21:57:09,077 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:57:12,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1676466.6666666667, ans=0.125 2023-11-21 21:57:13,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1676466.6666666667, ans=0.2 2023-11-21 21:57:16,913 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 21:57:24,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0 2023-11-21 21:57:26,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1676533.3333333333, ans=0.1 2023-11-21 21:57:33,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1676600.0, ans=0.025 2023-11-21 21:57:41,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251500 2023-11-21 21:57:45,045 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.09 vs. limit=15.0 2023-11-21 21:57:53,220 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.659e+01 8.207e+01 8.781e+01 9.534e+01 1.219e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-21 21:57:54,659 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 21:57:58,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.23 vs. limit=15.0 2023-11-21 21:58:00,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1676733.3333333333, ans=0.125 2023-11-21 21:58:09,749 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11050, loss[loss=0.08837, simple_loss=0.1195, pruned_loss=0.02054, audio_tagging_loss=0.008073, over 15364.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09446, pruned_loss=0.01561, audio_tagging_loss=0.009509, over 3050814.89 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:58:17,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.46 vs. limit=10.0 2023-11-21 21:58:29,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1676866.6666666667, ans=0.125 2023-11-21 21:58:44,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.58 vs. limit=22.5 2023-11-21 21:58:45,507 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251550 2023-11-21 21:58:51,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.77 vs. limit=15.0 2023-11-21 21:58:53,390 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.64 vs. limit=22.5 2023-11-21 21:58:58,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1677000.0, ans=10.0 2023-11-21 21:59:07,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.84 vs. limit=15.0 2023-11-21 21:59:14,242 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11100, loss[loss=0.05989, simple_loss=0.07539, pruned_loss=0.01106, audio_tagging_loss=0.01114, over 14837.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09483, pruned_loss=0.01565, audio_tagging_loss=0.009652, over 3051177.86 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0 2023-11-21 21:59:31,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1677200.0, ans=15.0 2023-11-21 21:59:37,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1677200.0, ans=0.125 2023-11-21 21:59:39,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1677266.6666666667, ans=0.0 2023-11-21 21:59:49,376 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.29 vs. limit=15.0 2023-11-21 21:59:50,056 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251600 2023-11-21 21:59:53,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1677333.3333333333, ans=0.125 2023-11-21 22:00:02,561 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.524e+01 9.104e+01 1.000e+02 2.938e+02, threshold=1.821e+02, percent-clipped=1.0 2023-11-21 22:00:09,446 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.85 vs. limit=15.0 2023-11-21 22:00:18,950 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11150, loss[loss=0.05537, simple_loss=0.06867, pruned_loss=0.008587, audio_tagging_loss=0.01245, over 14638.00 frames. ], tot_loss[loss=0.07312, simple_loss=0.09508, pruned_loss=0.01572, audio_tagging_loss=0.009864, over 3047689.15 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 16.0 2023-11-21 22:00:41,361 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.15 vs. limit=6.0 2023-11-21 22:00:44,203 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.30 vs. limit=10.0 2023-11-21 22:00:54,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1677600.0, ans=0.125 2023-11-21 22:00:55,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251650 2023-11-21 22:01:22,746 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11200, loss[loss=0.07887, simple_loss=0.1012, pruned_loss=0.01741, audio_tagging_loss=0.01084, over 15127.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09436, pruned_loss=0.01574, audio_tagging_loss=0.009904, over 3049441.23 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:01:32,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1677800.0, ans=0.0 2023-11-21 22:01:39,696 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 22:01:40,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.89 vs. limit=15.0 2023-11-21 22:01:59,503 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251700 2023-11-21 22:02:10,495 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 7.989e+01 8.616e+01 9.280e+01 1.646e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-21 22:02:28,505 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11250, loss[loss=0.08883, simple_loss=0.1145, pruned_loss=0.01958, audio_tagging_loss=0.012, over 15946.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09438, pruned_loss=0.01574, audio_tagging_loss=0.009879, over 3048328.40 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:02:35,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.47 vs. limit=15.0 2023-11-21 22:02:53,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1678266.6666666667, ans=0.0 2023-11-21 22:03:03,366 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251750 2023-11-21 22:03:09,645 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.62 vs. limit=12.0 2023-11-21 22:03:11,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1678333.3333333333, ans=0.125 2023-11-21 22:03:15,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1678333.3333333333, ans=0.2 2023-11-21 22:03:18,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1678400.0, ans=0.1 2023-11-21 22:03:29,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1678400.0, ans=0.0 2023-11-21 22:03:31,991 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11300, loss[loss=0.07278, simple_loss=0.09203, pruned_loss=0.0196, audio_tagging_loss=0.007158, over 14627.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09528, pruned_loss=0.01603, audio_tagging_loss=0.009753, over 3040696.70 frames. ], batch size: 54, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:03:39,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1678466.6666666667, ans=0.0 2023-11-21 22:03:41,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.96 vs. limit=15.0 2023-11-21 22:03:52,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1678533.3333333333, ans=0.0 2023-11-21 22:03:59,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1678600.0, ans=0.1 2023-11-21 22:04:01,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1678600.0, ans=0.0 2023-11-21 22:04:07,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1678600.0, ans=0.2 2023-11-21 22:04:08,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251800 2023-11-21 22:04:11,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=1678666.6666666667, ans=0.1 2023-11-21 22:04:19,971 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.032e+01 8.626e+01 9.245e+01 1.282e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-21 22:04:24,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1678733.3333333333, ans=0.0 2023-11-21 22:04:36,721 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11350, loss[loss=0.06537, simple_loss=0.08389, pruned_loss=0.01131, audio_tagging_loss=0.01211, over 16026.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.0952, pruned_loss=0.0159, audio_tagging_loss=0.009604, over 3041228.35 frames. ], batch size: 60, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:05:10,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1678933.3333333333, ans=0.2 2023-11-21 22:05:12,695 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251850 2023-11-21 22:05:18,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1679000.0, ans=0.0 2023-11-21 22:05:24,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1679000.0, ans=0.125 2023-11-21 22:05:28,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1679066.6666666667, ans=0.1 2023-11-21 22:05:28,856 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.05 vs. limit=10.0 2023-11-21 22:05:40,481 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11400, loss[loss=0.07352, simple_loss=0.09744, pruned_loss=0.01609, audio_tagging_loss=0.008702, over 14824.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09538, pruned_loss=0.01587, audio_tagging_loss=0.009428, over 3037760.54 frames. ], batch size: 54, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:05:42,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1679133.3333333333, ans=0.1 2023-11-21 22:05:45,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1679133.3333333333, ans=0.07 2023-11-21 22:05:54,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1679200.0, ans=0.2 2023-11-21 22:06:01,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1679200.0, ans=0.125 2023-11-21 22:06:15,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251900 2023-11-21 22:06:27,613 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.296e+01 8.125e+01 8.630e+01 9.501e+01 1.337e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-21 22:06:28,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1679333.3333333333, ans=0.125 2023-11-21 22:06:44,321 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11450, loss[loss=0.05564, simple_loss=0.06915, pruned_loss=0.00973, audio_tagging_loss=0.01134, over 16341.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09542, pruned_loss=0.01582, audio_tagging_loss=0.009389, over 3041955.34 frames. ], batch size: 63, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:06:57,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1679533.3333333333, ans=0.125 2023-11-21 22:06:59,689 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 22:07:04,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1679533.3333333333, ans=0.125 2023-11-21 22:07:09,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1679600.0, ans=0.95 2023-11-21 22:07:21,344 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 251950 2023-11-21 22:07:26,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1679666.6666666667, ans=0.2 2023-11-21 22:07:30,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1679666.6666666667, ans=0.125 2023-11-21 22:07:47,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1679800.0, ans=0.0 2023-11-21 22:07:48,707 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11500, loss[loss=0.05673, simple_loss=0.07196, pruned_loss=0.008066, audio_tagging_loss=0.01268, over 15004.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09493, pruned_loss=0.01582, audio_tagging_loss=0.009412, over 3040371.98 frames. ], batch size: 55, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:08:01,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1679866.6666666667, ans=0.0 2023-11-21 22:08:05,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1679866.6666666667, ans=0.125 2023-11-21 22:08:25,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252000 2023-11-21 22:08:34,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1680000.0, ans=0.0 2023-11-21 22:08:39,022 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.783e+01 7.996e+01 8.554e+01 9.237e+01 1.174e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-21 22:08:42,215 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.26 vs. limit=15.0 2023-11-21 22:08:56,404 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11550, loss[loss=0.07824, simple_loss=0.1002, pruned_loss=0.01753, audio_tagging_loss=0.01058, over 15739.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09536, pruned_loss=0.01584, audio_tagging_loss=0.009403, over 3043710.55 frames. ], batch size: 57, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:08:57,788 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 22:09:08,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1680200.0, ans=0.125 2023-11-21 22:09:10,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1680200.0, ans=0.125 2023-11-21 22:09:20,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1680200.0, ans=0.07 2023-11-21 22:09:20,485 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.43 vs. limit=15.0 2023-11-21 22:09:24,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1680266.6666666667, ans=0.0 2023-11-21 22:09:27,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1680266.6666666667, ans=0.125 2023-11-21 22:09:31,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252050 2023-11-21 22:09:34,291 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 22:09:59,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1680466.6666666667, ans=0.125 2023-11-21 22:10:00,385 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11600, loss[loss=0.07214, simple_loss=0.09097, pruned_loss=0.01696, audio_tagging_loss=0.00969, over 14926.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09632, pruned_loss=0.01599, audio_tagging_loss=0.009363, over 3047105.97 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:10:12,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1680533.3333333333, ans=0.2 2023-11-21 22:10:18,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1680533.3333333333, ans=0.125 2023-11-21 22:10:32,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1680600.0, ans=0.125 2023-11-21 22:10:36,291 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252100 2023-11-21 22:10:47,819 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.780e+01 8.258e+01 8.947e+01 9.707e+01 1.482e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-21 22:10:55,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1680733.3333333333, ans=0.125 2023-11-21 22:11:04,539 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11650, loss[loss=0.07507, simple_loss=0.09659, pruned_loss=0.01591, audio_tagging_loss=0.01087, over 15591.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09632, pruned_loss=0.0161, audio_tagging_loss=0.009387, over 3042639.66 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:11:08,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.62 vs. limit=15.0 2023-11-21 22:11:25,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1680866.6666666667, ans=0.1 2023-11-21 22:11:40,801 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252150 2023-11-21 22:11:55,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1681066.6666666667, ans=0.125 2023-11-21 22:12:02,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1681066.6666666667, ans=0.125 2023-11-21 22:12:04,708 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 22:12:08,132 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11700, loss[loss=0.05982, simple_loss=0.06767, pruned_loss=0.01369, audio_tagging_loss=0.0123, over 15246.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09575, pruned_loss=0.01604, audio_tagging_loss=0.009401, over 3052881.36 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:12:45,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252200 2023-11-21 22:12:45,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1681266.6666666667, ans=0.125 2023-11-21 22:12:48,659 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.40 vs. limit=15.0 2023-11-21 22:12:54,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1681333.3333333333, ans=0.125 2023-11-21 22:12:56,196 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.227e+01 8.797e+01 9.747e+01 1.315e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-21 22:13:11,743 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.57 vs. limit=15.0 2023-11-21 22:13:14,117 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11750, loss[loss=0.07343, simple_loss=0.108, pruned_loss=0.01352, audio_tagging_loss=0.005905, over 14671.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09631, pruned_loss=0.01629, audio_tagging_loss=0.009453, over 3047350.56 frames. ], batch size: 54, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:13:20,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1681466.6666666667, ans=0.0 2023-11-21 22:13:24,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1681466.6666666667, ans=0.125 2023-11-21 22:13:26,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1681533.3333333333, ans=0.0 2023-11-21 22:13:47,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1681600.0, ans=0.125 2023-11-21 22:13:50,099 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252250 2023-11-21 22:14:11,926 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.337e-02 2023-11-21 22:14:13,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1681733.3333333333, ans=0.0 2023-11-21 22:14:18,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1681800.0, ans=0.2 2023-11-21 22:14:19,021 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11800, loss[loss=0.08665, simple_loss=0.1145, pruned_loss=0.0208, audio_tagging_loss=0.008587, over 15218.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09635, pruned_loss=0.0163, audio_tagging_loss=0.00952, over 3048952.43 frames. ], batch size: 55, lr: 3.22e-03, grad_scale: 16.0 2023-11-21 22:14:30,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1681866.6666666667, ans=0.125 2023-11-21 22:14:36,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1681866.6666666667, ans=0.1 2023-11-21 22:14:43,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=1681933.3333333333, ans=8.0 2023-11-21 22:14:54,854 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252300 2023-11-21 22:15:02,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1682000.0, ans=0.1 2023-11-21 22:15:03,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1682000.0, ans=15.0 2023-11-21 22:15:07,670 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 7.961e+01 8.555e+01 9.322e+01 1.275e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-21 22:15:08,422 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.60 vs. limit=15.0 2023-11-21 22:15:10,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.09 vs. limit=15.0 2023-11-21 22:15:18,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1682066.6666666667, ans=0.04949747468305833 2023-11-21 22:15:22,858 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11850, loss[loss=0.06395, simple_loss=0.07389, pruned_loss=0.01751, audio_tagging_loss=0.00949, over 14573.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09602, pruned_loss=0.01618, audio_tagging_loss=0.009591, over 3043674.94 frames. ], batch size: 55, lr: 3.22e-03, grad_scale: 16.0 2023-11-21 22:15:25,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1682133.3333333333, ans=0.125 2023-11-21 22:15:43,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1682200.0, ans=0.125 2023-11-21 22:15:59,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1682266.6666666667, ans=0.125 2023-11-21 22:16:00,028 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252350 2023-11-21 22:16:09,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1682333.3333333333, ans=0.0 2023-11-21 22:16:10,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1682333.3333333333, ans=0.1 2023-11-21 22:16:12,137 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.13 vs. limit=15.0 2023-11-21 22:16:27,839 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11900, loss[loss=0.06922, simple_loss=0.08246, pruned_loss=0.01553, audio_tagging_loss=0.01247, over 14298.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09528, pruned_loss=0.01616, audio_tagging_loss=0.009738, over 3037786.76 frames. ], batch size: 55, lr: 3.22e-03, grad_scale: 16.0 2023-11-21 22:16:53,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.48 vs. limit=22.5 2023-11-21 22:17:04,122 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252400 2023-11-21 22:17:08,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.33 vs. limit=22.5 2023-11-21 22:17:17,497 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.240e+01 7.885e+01 8.457e+01 9.193e+01 1.226e+02, threshold=1.691e+02, percent-clipped=0.0 2023-11-21 22:17:33,582 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 11950, loss[loss=0.06928, simple_loss=0.08953, pruned_loss=0.01285, audio_tagging_loss=0.01167, over 15825.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09563, pruned_loss=0.01617, audio_tagging_loss=0.009822, over 3037380.74 frames. ], batch size: 60, lr: 3.22e-03, grad_scale: 16.0 2023-11-21 22:17:41,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1682800.0, ans=0.125 2023-11-21 22:17:41,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.64 vs. limit=22.5 2023-11-21 22:18:05,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1682933.3333333333, ans=0.125 2023-11-21 22:18:09,260 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252450 2023-11-21 22:18:17,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1683000.0, ans=0.07 2023-11-21 22:18:19,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1683000.0, ans=0.0 2023-11-21 22:18:36,449 INFO [train_asr.py:1221] (3/4) Epoch 21, batch 12000, loss[loss=0.08288, simple_loss=0.1116, pruned_loss=0.01738, audio_tagging_loss=0.009687, over 16495.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09546, pruned_loss=0.01624, audio_tagging_loss=0.009854, over 3036082.22 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 32.0 2023-11-21 22:18:36,450 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 22:18:55,924 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([3.9413, 2.6213, 4.4967, 2.2329], device='cuda:3') 2023-11-21 22:19:19,204 INFO [train_asr.py:1253] (3/4) Epoch 21, validation: loss=0.05938, simple_loss=0.05195, pruned_loss=0.005201, audio_tagging_loss=0.02821, over 4681554.00 frames. 2023-11-21 22:19:19,205 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 22:19:19,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1683133.3333333333, ans=0.0 2023-11-21 22:19:24,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1683133.3333333333, ans=0.1 2023-11-21 22:19:25,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.84 vs. limit=15.0 2023-11-21 22:19:27,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1683133.3333333333, ans=0.2 2023-11-21 22:20:22,680 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 0, loss[loss=0.1049, simple_loss=0.1214, pruned_loss=0.02469, audio_tagging_loss=0.01953, over 16005.00 frames. ], tot_loss[loss=0.1049, simple_loss=0.1214, pruned_loss=0.02469, audio_tagging_loss=0.01953, over 16005.00 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:20:22,684 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 22:20:54,540 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.5344, 2.8753, 3.7920, 3.5303], device='cuda:3') 2023-11-21 22:20:59,038 INFO [train_asr.py:1253] (3/4) Epoch 22, validation: loss=0.05904, simple_loss=0.0519, pruned_loss=0.0051, audio_tagging_loss=0.02799, over 4681554.00 frames. 2023-11-21 22:20:59,038 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 22:21:03,952 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252500 2023-11-21 22:21:16,300 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.182e+01 9.159e+01 9.846e+01 1.292e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-21 22:21:24,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1683426.6666666667, ans=0.0 2023-11-21 22:21:40,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1683493.3333333333, ans=0.125 2023-11-21 22:21:58,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1683560.0, ans=0.1 2023-11-21 22:22:02,756 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 50, loss[loss=0.08556, simple_loss=0.09403, pruned_loss=0.01878, audio_tagging_loss=0.01977, over 15558.00 frames. ], tot_loss[loss=0.08007, simple_loss=0.0901, pruned_loss=0.01561, audio_tagging_loss=0.0194, over 686936.19 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:22:07,692 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252550 2023-11-21 22:22:13,425 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.78 vs. limit=22.5 2023-11-21 22:22:30,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1683760.0, ans=0.1 2023-11-21 22:22:51,311 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.41 vs. limit=10.0 2023-11-21 22:22:52,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1683826.6666666667, ans=0.09899494936611666 2023-11-21 22:22:54,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.51 vs. limit=22.5 2023-11-21 22:23:05,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1683893.3333333333, ans=0.1 2023-11-21 22:23:06,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1683960.0, ans=0.125 2023-11-21 22:23:07,450 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 100, loss[loss=0.08895, simple_loss=0.1049, pruned_loss=0.02163, audio_tagging_loss=0.01487, over 14588.00 frames. ], tot_loss[loss=0.0801, simple_loss=0.09205, pruned_loss=0.01581, audio_tagging_loss=0.01827, over 1208896.11 frames. ], batch size: 53, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:23:08,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1683960.0, ans=0.07 2023-11-21 22:23:12,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252600 2023-11-21 22:23:16,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1683960.0, ans=0.125 2023-11-21 22:23:22,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1684026.6666666667, ans=0.07 2023-11-21 22:23:25,408 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.481e+01 8.773e+01 9.405e+01 1.016e+02 1.413e+02, threshold=1.881e+02, percent-clipped=0.0 2023-11-21 22:23:25,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1684026.6666666667, ans=0.0 2023-11-21 22:23:42,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1684093.3333333333, ans=0.0 2023-11-21 22:23:43,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1684093.3333333333, ans=0.125 2023-11-21 22:24:11,244 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 150, loss[loss=0.102, simple_loss=0.1245, pruned_loss=0.02755, audio_tagging_loss=0.0122, over 16498.00 frames. ], tot_loss[loss=0.07902, simple_loss=0.09391, pruned_loss=0.01592, audio_tagging_loss=0.01615, over 1622037.97 frames. ], batch size: 63, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:24:16,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252650 2023-11-21 22:24:18,426 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.70 vs. limit=22.5 2023-11-21 22:24:25,207 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.29 vs. limit=22.5 2023-11-21 22:24:26,197 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.98 vs. limit=6.0 2023-11-21 22:24:29,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1684360.0, ans=0.1 2023-11-21 22:25:00,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1684493.3333333333, ans=0.2 2023-11-21 22:25:08,215 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.016e-02 2023-11-21 22:25:15,846 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 200, loss[loss=0.08209, simple_loss=0.1177, pruned_loss=0.01617, audio_tagging_loss=0.007098, over 15138.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09297, pruned_loss=0.01573, audio_tagging_loss=0.01432, over 1934543.54 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:25:20,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252700 2023-11-21 22:25:30,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1684693.3333333333, ans=0.1 2023-11-21 22:25:34,372 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.318e+01 8.693e+01 9.716e+01 1.201e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 22:25:37,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1684693.3333333333, ans=0.125 2023-11-21 22:25:49,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1684760.0, ans=0.2 2023-11-21 22:25:57,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1684826.6666666667, ans=0.0 2023-11-21 22:26:02,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1684826.6666666667, ans=0.07 2023-11-21 22:26:19,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1684960.0, ans=0.125 2023-11-21 22:26:21,131 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 250, loss[loss=0.08104, simple_loss=0.1016, pruned_loss=0.02004, audio_tagging_loss=0.01019, over 13732.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.0926, pruned_loss=0.01562, audio_tagging_loss=0.0129, over 2177966.21 frames. ], batch size: 53, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:26:26,039 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252750 2023-11-21 22:26:37,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1685026.6666666667, ans=0.125 2023-11-21 22:26:43,434 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.68 vs. limit=15.0 2023-11-21 22:26:58,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1685160.0, ans=0.0 2023-11-21 22:27:12,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1685226.6666666667, ans=0.125 2023-11-21 22:27:22,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1685226.6666666667, ans=0.0 2023-11-21 22:27:24,799 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 300, loss[loss=0.08672, simple_loss=0.1294, pruned_loss=0.01631, audio_tagging_loss=0.005705, over 15895.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09344, pruned_loss=0.01564, audio_tagging_loss=0.01191, over 2368629.89 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:27:25,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1685293.3333333333, ans=0.0 2023-11-21 22:27:30,632 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252800 2023-11-21 22:27:30,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1685293.3333333333, ans=0.125 2023-11-21 22:27:36,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1685293.3333333333, ans=0.125 2023-11-21 22:27:43,810 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.171e+01 8.868e+01 9.880e+01 1.210e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-21 22:28:00,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1685426.6666666667, ans=0.125 2023-11-21 22:28:05,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.73 vs. limit=15.0 2023-11-21 22:28:06,030 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.34 vs. limit=10.0 2023-11-21 22:28:06,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1685493.3333333333, ans=0.125 2023-11-21 22:28:08,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1685493.3333333333, ans=0.04949747468305833 2023-11-21 22:28:09,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1685493.3333333333, ans=0.1 2023-11-21 22:28:19,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1685560.0, ans=0.0 2023-11-21 22:28:20,141 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0 2023-11-21 22:28:22,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1685560.0, ans=0.0 2023-11-21 22:28:30,575 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 350, loss[loss=0.06404, simple_loss=0.083, pruned_loss=0.01628, audio_tagging_loss=0.006261, over 13914.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09446, pruned_loss=0.01591, audio_tagging_loss=0.01121, over 2518278.67 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:28:36,350 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252850 2023-11-21 22:28:55,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1685693.3333333333, ans=0.1 2023-11-21 22:29:07,281 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.32 vs. limit=15.0 2023-11-21 22:29:25,963 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.74 vs. limit=15.0 2023-11-21 22:29:28,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1685893.3333333333, ans=0.125 2023-11-21 22:29:34,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1685893.3333333333, ans=0.125 2023-11-21 22:29:36,496 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 400, loss[loss=0.07586, simple_loss=0.1003, pruned_loss=0.01737, audio_tagging_loss=0.008339, over 14908.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.0942, pruned_loss=0.01592, audio_tagging_loss=0.01074, over 2630869.01 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:29:42,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252900 2023-11-21 22:29:49,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1686026.6666666667, ans=0.125 2023-11-21 22:29:55,042 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.919e+01 8.135e+01 8.839e+01 9.372e+01 1.171e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 22:30:00,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1686026.6666666667, ans=0.2 2023-11-21 22:30:19,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1686160.0, ans=0.125 2023-11-21 22:30:40,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1686226.6666666667, ans=0.5 2023-11-21 22:30:42,527 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 450, loss[loss=0.07452, simple_loss=0.1086, pruned_loss=0.01284, audio_tagging_loss=0.007361, over 15760.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09407, pruned_loss=0.01582, audio_tagging_loss=0.01046, over 2731292.13 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:30:48,346 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 252950 2023-11-21 22:31:32,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1686493.3333333333, ans=0.1 2023-11-21 22:31:43,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0 2023-11-21 22:31:44,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.70 vs. limit=15.0 2023-11-21 22:31:48,770 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 500, loss[loss=0.06992, simple_loss=0.09026, pruned_loss=0.01521, audio_tagging_loss=0.009582, over 16610.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09435, pruned_loss=0.01593, audio_tagging_loss=0.01024, over 2795499.71 frames. ], batch size: 62, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:31:52,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1686626.6666666667, ans=0.125 2023-11-21 22:31:53,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253000 2023-11-21 22:31:58,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1686626.6666666667, ans=0.0 2023-11-21 22:32:01,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1686693.3333333333, ans=0.0 2023-11-21 22:32:05,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1686693.3333333333, ans=0.0 2023-11-21 22:32:06,980 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.271e+01 8.753e+01 9.475e+01 1.190e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 22:32:39,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.03 vs. limit=10.0 2023-11-21 22:32:42,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1686893.3333333333, ans=0.125 2023-11-21 22:32:48,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1686893.3333333333, ans=0.0 2023-11-21 22:32:53,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1686960.0, ans=0.1 2023-11-21 22:32:54,279 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 550, loss[loss=0.08024, simple_loss=0.1023, pruned_loss=0.02008, audio_tagging_loss=0.009015, over 13987.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09477, pruned_loss=0.01602, audio_tagging_loss=0.01012, over 2851230.89 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:32:59,567 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 22:33:00,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253050 2023-11-21 22:33:30,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1687093.3333333333, ans=0.125 2023-11-21 22:33:31,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1687093.3333333333, ans=0.125 2023-11-21 22:34:00,169 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 600, loss[loss=0.0699, simple_loss=0.08712, pruned_loss=0.01787, audio_tagging_loss=0.00847, over 16271.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09454, pruned_loss=0.01579, audio_tagging_loss=0.009908, over 2898174.55 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:34:05,350 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253100 2023-11-21 22:34:17,438 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=22.5 2023-11-21 22:34:18,148 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.004e+01 8.758e+01 9.434e+01 1.366e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-21 22:34:24,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1687426.6666666667, ans=0.5 2023-11-21 22:34:29,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1687426.6666666667, ans=0.125 2023-11-21 22:34:30,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1687426.6666666667, ans=0.09899494936611666 2023-11-21 22:34:41,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.27 vs. limit=22.5 2023-11-21 22:34:42,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1687493.3333333333, ans=0.2 2023-11-21 22:34:50,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1687493.3333333333, ans=0.0 2023-11-21 22:35:05,435 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 650, loss[loss=0.06357, simple_loss=0.08135, pruned_loss=0.009792, audio_tagging_loss=0.01311, over 14451.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09381, pruned_loss=0.0157, audio_tagging_loss=0.009841, over 2928955.68 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:35:10,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253150 2023-11-21 22:35:38,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.97 vs. limit=10.0 2023-11-21 22:35:46,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1687826.6666666667, ans=0.125 2023-11-21 22:35:46,627 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.67 vs. limit=15.0 2023-11-21 22:35:58,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1687893.3333333333, ans=0.09899494936611666 2023-11-21 22:36:06,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1687893.3333333333, ans=0.125 2023-11-21 22:36:09,018 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 700, loss[loss=0.09101, simple_loss=0.118, pruned_loss=0.02482, audio_tagging_loss=0.007195, over 15322.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09513, pruned_loss=0.01589, audio_tagging_loss=0.009784, over 2956466.27 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 16.0 2023-11-21 22:36:11,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1687960.0, ans=0.0 2023-11-21 22:36:13,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253200 2023-11-21 22:36:25,615 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.32 vs. limit=22.5 2023-11-21 22:36:28,395 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.099e+01 8.868e+01 9.467e+01 1.457e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-21 22:36:42,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1688093.3333333333, ans=0.125 2023-11-21 22:37:13,498 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 750, loss[loss=0.08722, simple_loss=0.1061, pruned_loss=0.02528, audio_tagging_loss=0.008905, over 14869.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09516, pruned_loss=0.016, audio_tagging_loss=0.009818, over 2979326.57 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 16.0 2023-11-21 22:37:18,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253250 2023-11-21 22:37:21,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1688293.3333333333, ans=0.0 2023-11-21 22:37:23,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1688293.3333333333, ans=0.025 2023-11-21 22:37:34,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1688360.0, ans=0.09899494936611666 2023-11-21 22:37:47,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1688426.6666666667, ans=0.0 2023-11-21 22:37:47,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1688426.6666666667, ans=0.125 2023-11-21 22:37:51,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1688493.3333333333, ans=0.0 2023-11-21 22:38:02,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.43 vs. limit=15.0 2023-11-21 22:38:15,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1688560.0, ans=0.07 2023-11-21 22:38:17,551 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 800, loss[loss=0.07121, simple_loss=0.08544, pruned_loss=0.01692, audio_tagging_loss=0.01158, over 14930.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09408, pruned_loss=0.01599, audio_tagging_loss=0.009892, over 3002098.09 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:38:21,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1688626.6666666667, ans=0.0 2023-11-21 22:38:22,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253300 2023-11-21 22:38:36,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.171e+01 8.697e+01 9.274e+01 1.215e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-21 22:38:37,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1688693.3333333333, ans=0.125 2023-11-21 22:39:12,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.76 vs. limit=15.0 2023-11-21 22:39:20,962 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 850, loss[loss=0.05718, simple_loss=0.07274, pruned_loss=0.009467, audio_tagging_loss=0.01134, over 16583.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09461, pruned_loss=0.01602, audio_tagging_loss=0.009921, over 3014396.36 frames. ], batch size: 62, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:39:26,029 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253350 2023-11-21 22:39:47,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.21 vs. limit=15.0 2023-11-21 22:40:12,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1689226.6666666667, ans=0.125 2023-11-21 22:40:19,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1689226.6666666667, ans=0.1 2023-11-21 22:40:24,931 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 900, loss[loss=0.04449, simple_loss=0.05073, pruned_loss=0.007532, audio_tagging_loss=0.01159, over 15135.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09476, pruned_loss=0.01608, audio_tagging_loss=0.009966, over 3023476.83 frames. ], batch size: 59, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:40:30,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253400 2023-11-21 22:40:39,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1689360.0, ans=0.0 2023-11-21 22:40:44,872 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.175e+01 9.057e+01 9.659e+01 1.593e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-21 22:40:45,646 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.34 vs. limit=15.0 2023-11-21 22:40:46,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1689360.0, ans=0.125 2023-11-21 22:40:46,939 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.19 vs. limit=22.5 2023-11-21 22:41:01,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1689426.6666666667, ans=0.0 2023-11-21 22:41:30,295 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 950, loss[loss=0.06707, simple_loss=0.08033, pruned_loss=0.01994, audio_tagging_loss=0.006964, over 14803.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09506, pruned_loss=0.01596, audio_tagging_loss=0.00982, over 3025158.42 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:41:30,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.11 vs. limit=15.0 2023-11-21 22:41:35,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253450 2023-11-21 22:41:50,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1689693.3333333333, ans=0.0 2023-11-21 22:42:04,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1689760.0, ans=0.125 2023-11-21 22:42:07,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1689826.6666666667, ans=0.125 2023-11-21 22:42:18,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1689826.6666666667, ans=0.1 2023-11-21 22:42:32,968 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1000, loss[loss=0.06448, simple_loss=0.08707, pruned_loss=0.01338, audio_tagging_loss=0.007569, over 14738.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.0958, pruned_loss=0.01603, audio_tagging_loss=0.009542, over 3033488.96 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:42:37,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253500 2023-11-21 22:42:38,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.71 vs. limit=22.5 2023-11-21 22:42:44,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1690026.6666666667, ans=0.125 2023-11-21 22:42:51,812 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.522e+01 8.090e+01 8.690e+01 9.714e+01 1.265e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-21 22:42:54,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1690026.6666666667, ans=0.1 2023-11-21 22:42:59,990 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 22:43:01,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1690093.3333333333, ans=0.125 2023-11-21 22:43:37,564 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1050, loss[loss=0.1025, simple_loss=0.1394, pruned_loss=0.02674, audio_tagging_loss=0.006066, over 14787.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09578, pruned_loss=0.01607, audio_tagging_loss=0.009437, over 3029720.42 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:43:42,586 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253550 2023-11-21 22:43:44,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1690293.3333333333, ans=0.125 2023-11-21 22:43:54,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.40 vs. limit=22.5 2023-11-21 22:43:57,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1690360.0, ans=0.0 2023-11-21 22:44:06,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1690426.6666666667, ans=0.125 2023-11-21 22:44:10,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1690426.6666666667, ans=0.0 2023-11-21 22:44:15,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=1690493.3333333333, ans=0.02 2023-11-21 22:44:20,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1690493.3333333333, ans=0.0 2023-11-21 22:44:36,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.36 vs. limit=15.0 2023-11-21 22:44:41,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1690626.6666666667, ans=0.0 2023-11-21 22:44:42,880 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1100, loss[loss=0.07406, simple_loss=0.1003, pruned_loss=0.01325, audio_tagging_loss=0.01065, over 14266.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09533, pruned_loss=0.01602, audio_tagging_loss=0.00952, over 3036655.63 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 32.0 2023-11-21 22:44:45,374 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 22:44:47,915 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253600 2023-11-21 22:44:48,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1690626.6666666667, ans=0.0 2023-11-21 22:44:48,427 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=11.16 vs. limit=12.0 2023-11-21 22:45:01,572 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.847e+01 8.128e+01 8.851e+01 9.302e+01 1.185e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-21 22:45:11,132 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.02 vs. limit=22.5 2023-11-21 22:45:21,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0 2023-11-21 22:45:24,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1690826.6666666667, ans=0.1 2023-11-21 22:45:27,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1690826.6666666667, ans=0.125 2023-11-21 22:45:45,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1690960.0, ans=0.0 2023-11-21 22:45:46,097 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1150, loss[loss=0.05084, simple_loss=0.06822, pruned_loss=0.008778, audio_tagging_loss=0.007958, over 14304.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09538, pruned_loss=0.01593, audio_tagging_loss=0.009369, over 3033077.57 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 8.0 2023-11-21 22:45:51,102 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253650 2023-11-21 22:45:51,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1690960.0, ans=0.125 2023-11-21 22:45:51,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1690960.0, ans=0.125 2023-11-21 22:46:09,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1691026.6666666667, ans=0.2 2023-11-21 22:46:26,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1691160.0, ans=0.0 2023-11-21 22:46:35,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1691160.0, ans=0.125 2023-11-21 22:46:50,834 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1200, loss[loss=0.04912, simple_loss=0.05894, pruned_loss=0.006551, audio_tagging_loss=0.0131, over 14505.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09462, pruned_loss=0.01577, audio_tagging_loss=0.00949, over 3042807.46 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 16.0 2023-11-21 22:46:55,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253700 2023-11-21 22:47:06,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1691360.0, ans=0.125 2023-11-21 22:47:12,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1691360.0, ans=0.2 2023-11-21 22:47:12,907 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.597e+01 8.044e+01 8.657e+01 9.227e+01 1.115e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 22:47:20,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1691426.6666666667, ans=0.025 2023-11-21 22:47:27,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1691493.3333333333, ans=0.0 2023-11-21 22:47:42,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1691560.0, ans=0.125 2023-11-21 22:47:51,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1691560.0, ans=0.125 2023-11-21 22:47:52,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1691560.0, ans=0.1 2023-11-21 22:47:54,030 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=22.56 vs. limit=22.5 2023-11-21 22:47:54,542 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1250, loss[loss=0.06886, simple_loss=0.09115, pruned_loss=0.01447, audio_tagging_loss=0.008811, over 15032.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09455, pruned_loss=0.01571, audio_tagging_loss=0.009386, over 3038910.57 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 16.0 2023-11-21 22:48:00,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253750 2023-11-21 22:48:07,330 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.88 vs. limit=22.5 2023-11-21 22:48:27,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1691760.0, ans=0.0 2023-11-21 22:48:41,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1691826.6666666667, ans=0.1 2023-11-21 22:48:44,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1691826.6666666667, ans=0.125 2023-11-21 22:48:47,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1691893.3333333333, ans=0.1 2023-11-21 22:48:59,730 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1300, loss[loss=0.05838, simple_loss=0.07174, pruned_loss=0.01192, audio_tagging_loss=0.0106, over 16374.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09385, pruned_loss=0.01555, audio_tagging_loss=0.009459, over 3037824.22 frames. ], batch size: 65, lr: 3.14e-03, grad_scale: 16.0 2023-11-21 22:49:03,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1691960.0, ans=0.125 2023-11-21 22:49:04,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253800 2023-11-21 22:49:11,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0 2023-11-21 22:49:18,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1692026.6666666667, ans=0.1 2023-11-21 22:49:22,325 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 7.957e+01 8.431e+01 9.281e+01 1.105e+02, threshold=1.686e+02, percent-clipped=0.0 2023-11-21 22:49:23,973 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.331e-02 2023-11-21 22:49:24,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1692026.6666666667, ans=0.1 2023-11-21 22:49:37,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1692093.3333333333, ans=0.0 2023-11-21 22:50:04,320 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1350, loss[loss=0.06271, simple_loss=0.07419, pruned_loss=0.01092, audio_tagging_loss=0.0147, over 15637.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09371, pruned_loss=0.01554, audio_tagging_loss=0.00947, over 3047170.18 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 8.0 2023-11-21 22:50:10,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253850 2023-11-21 22:50:30,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.65 vs. limit=22.5 2023-11-21 22:50:51,079 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 22:51:09,543 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1400, loss[loss=0.06409, simple_loss=0.08134, pruned_loss=0.01346, audio_tagging_loss=0.009957, over 15229.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09427, pruned_loss=0.01565, audio_tagging_loss=0.009451, over 3044422.77 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 8.0 2023-11-21 22:51:15,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253900 2023-11-21 22:51:32,900 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.193e+01 8.723e+01 9.483e+01 1.296e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-21 22:51:49,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1692826.6666666667, ans=0.125 2023-11-21 22:51:58,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1692826.6666666667, ans=0.1 2023-11-21 22:52:15,000 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1450, loss[loss=0.07085, simple_loss=0.08538, pruned_loss=0.0167, audio_tagging_loss=0.01146, over 15019.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09469, pruned_loss=0.01575, audio_tagging_loss=0.009467, over 3045174.16 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 8.0 2023-11-21 22:52:19,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 253950 2023-11-21 22:52:37,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1693026.6666666667, ans=0.1 2023-11-21 22:52:51,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1693160.0, ans=0.125 2023-11-21 22:52:55,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1693160.0, ans=0.125 2023-11-21 22:53:19,101 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1500, loss[loss=0.06509, simple_loss=0.07501, pruned_loss=0.01695, audio_tagging_loss=0.01064, over 14527.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09491, pruned_loss=0.01587, audio_tagging_loss=0.009464, over 3049379.95 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 8.0 2023-11-21 22:53:24,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254000 2023-11-21 22:53:29,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1693293.3333333333, ans=0.0 2023-11-21 22:53:35,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1693360.0, ans=10.0 2023-11-21 22:53:40,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1693360.0, ans=0.0 2023-11-21 22:53:40,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1693360.0, ans=0.125 2023-11-21 22:53:41,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1693360.0, ans=0.0 2023-11-21 22:53:42,807 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.377e+01 8.863e+01 9.600e+01 1.162e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-21 22:54:08,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1693493.3333333333, ans=15.0 2023-11-21 22:54:17,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1693560.0, ans=0.125 2023-11-21 22:54:18,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.27 vs. limit=15.0 2023-11-21 22:54:23,600 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1550, loss[loss=0.06816, simple_loss=0.09046, pruned_loss=0.01373, audio_tagging_loss=0.009199, over 14510.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09565, pruned_loss=0.01589, audio_tagging_loss=0.009473, over 3048368.81 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 8.0 2023-11-21 22:54:26,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1693626.6666666667, ans=0.2 2023-11-21 22:54:26,455 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.45 vs. limit=10.0 2023-11-21 22:54:28,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254050 2023-11-21 22:54:35,718 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.04 vs. limit=15.0 2023-11-21 22:54:42,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1693693.3333333333, ans=0.0 2023-11-21 22:54:58,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1693760.0, ans=0.1 2023-11-21 22:55:15,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1693893.3333333333, ans=0.125 2023-11-21 22:55:16,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1693893.3333333333, ans=0.0 2023-11-21 22:55:16,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1693893.3333333333, ans=0.2 2023-11-21 22:55:18,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1693893.3333333333, ans=0.125 2023-11-21 22:55:26,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1693960.0, ans=0.1 2023-11-21 22:55:27,429 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1600, loss[loss=0.05859, simple_loss=0.08206, pruned_loss=0.009321, audio_tagging_loss=0.008236, over 14816.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09615, pruned_loss=0.01613, audio_tagging_loss=0.009511, over 3050037.97 frames. ], batch size: 55, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 22:55:32,295 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254100 2023-11-21 22:55:38,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1693960.0, ans=0.2 2023-11-21 22:55:41,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1694026.6666666667, ans=0.125 2023-11-21 22:55:44,723 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.99 vs. limit=15.0 2023-11-21 22:55:49,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.16 vs. limit=15.0 2023-11-21 22:55:50,495 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.195e+01 8.712e+01 9.406e+01 1.124e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-21 22:56:05,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.29 vs. limit=12.0 2023-11-21 22:56:31,736 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1650, loss[loss=0.103, simple_loss=0.1387, pruned_loss=0.02572, audio_tagging_loss=0.007911, over 16207.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09583, pruned_loss=0.01603, audio_tagging_loss=0.009551, over 3055197.30 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 22:56:31,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1694293.3333333333, ans=0.0 2023-11-21 22:56:33,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1694293.3333333333, ans=0.125 2023-11-21 22:56:36,645 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254150 2023-11-21 22:56:39,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1694293.3333333333, ans=0.125 2023-11-21 22:56:48,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1694360.0, ans=10.0 2023-11-21 22:56:51,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1694360.0, ans=0.1 2023-11-21 22:57:05,561 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.66 vs. limit=15.0 2023-11-21 22:57:09,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1694493.3333333333, ans=0.125 2023-11-21 22:57:23,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1694560.0, ans=0.1 2023-11-21 22:57:24,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1694560.0, ans=0.125 2023-11-21 22:57:36,315 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1700, loss[loss=0.06788, simple_loss=0.0873, pruned_loss=0.01417, audio_tagging_loss=0.01006, over 15060.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09538, pruned_loss=0.01604, audio_tagging_loss=0.009612, over 3055043.00 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 22:57:41,387 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254200 2023-11-21 22:57:59,282 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.322e+01 8.804e+01 9.363e+01 1.175e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-21 22:57:59,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1694693.3333333333, ans=0.07 2023-11-21 22:58:41,009 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1750, loss[loss=0.1051, simple_loss=0.1465, pruned_loss=0.02742, audio_tagging_loss=0.004408, over 15034.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09476, pruned_loss=0.01589, audio_tagging_loss=0.009535, over 3058742.82 frames. ], batch size: 53, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 22:58:45,967 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254250 2023-11-21 22:58:55,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1695026.6666666667, ans=0.125 2023-11-21 22:59:14,240 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.18 vs. limit=12.0 2023-11-21 22:59:24,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1695160.0, ans=0.125 2023-11-21 22:59:44,767 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1800, loss[loss=0.07589, simple_loss=0.09748, pruned_loss=0.01712, audio_tagging_loss=0.01003, over 15248.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09476, pruned_loss=0.01578, audio_tagging_loss=0.009378, over 3053486.44 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 22:59:50,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254300 2023-11-21 22:59:50,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.92 vs. limit=15.0 2023-11-21 23:00:08,559 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.157e+01 8.741e+01 9.334e+01 1.416e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-21 23:00:11,545 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.66 vs. limit=15.0 2023-11-21 23:00:15,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.62 vs. limit=22.5 2023-11-21 23:00:16,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.81 vs. limit=22.5 2023-11-21 23:00:46,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1695560.0, ans=0.1 2023-11-21 23:00:49,760 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1850, loss[loss=0.06615, simple_loss=0.09275, pruned_loss=0.01177, audio_tagging_loss=0.008002, over 15168.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09452, pruned_loss=0.01573, audio_tagging_loss=0.009304, over 3055926.75 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 23:00:52,455 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:00:54,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254350 2023-11-21 23:01:08,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1695693.3333333333, ans=0.5 2023-11-21 23:01:19,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.27 vs. limit=10.0 2023-11-21 23:01:26,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1695826.6666666667, ans=0.125 2023-11-21 23:01:54,046 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1900, loss[loss=0.08106, simple_loss=0.1111, pruned_loss=0.01925, audio_tagging_loss=0.006268, over 16375.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09433, pruned_loss=0.01568, audio_tagging_loss=0.009287, over 3058730.13 frames. ], batch size: 62, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 23:01:56,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1695960.0, ans=0.09899494936611666 2023-11-21 23:01:59,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254400 2023-11-21 23:02:16,781 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 8.079e+01 8.685e+01 9.740e+01 1.437e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-21 23:02:19,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0 2023-11-21 23:02:28,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1696093.3333333333, ans=15.0 2023-11-21 23:02:29,158 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0 2023-11-21 23:02:32,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1696160.0, ans=0.1 2023-11-21 23:02:49,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1696226.6666666667, ans=0.125 2023-11-21 23:02:58,594 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 1950, loss[loss=0.07218, simple_loss=0.09284, pruned_loss=0.01442, audio_tagging_loss=0.01134, over 14697.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09424, pruned_loss=0.01556, audio_tagging_loss=0.009334, over 3055153.84 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 23:03:03,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254450 2023-11-21 23:03:18,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.31 vs. limit=15.0 2023-11-21 23:03:37,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1696493.3333333333, ans=0.125 2023-11-21 23:03:49,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1696560.0, ans=0.125 2023-11-21 23:04:04,665 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2000, loss[loss=0.07, simple_loss=0.0856, pruned_loss=0.01716, audio_tagging_loss=0.01005, over 15735.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09368, pruned_loss=0.01561, audio_tagging_loss=0.009471, over 3046035.59 frames. ], batch size: 60, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:04:09,657 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254500 2023-11-21 23:04:27,265 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 7.983e+01 8.610e+01 9.428e+01 1.340e+02, threshold=1.722e+02, percent-clipped=0.0 2023-11-21 23:05:03,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1696893.3333333333, ans=22.5 2023-11-21 23:05:05,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1696893.3333333333, ans=0.125 2023-11-21 23:05:08,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1696960.0, ans=0.125 2023-11-21 23:05:09,035 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2050, loss[loss=0.07155, simple_loss=0.07881, pruned_loss=0.01912, audio_tagging_loss=0.01302, over 15690.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09497, pruned_loss=0.01574, audio_tagging_loss=0.009438, over 3056237.87 frames. ], batch size: 60, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:05:11,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1696960.0, ans=0.5 2023-11-21 23:05:14,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254550 2023-11-21 23:05:23,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1697026.6666666667, ans=0.0 2023-11-21 23:05:35,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1697093.3333333333, ans=0.1 2023-11-21 23:05:43,453 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.39 vs. limit=12.0 2023-11-21 23:05:48,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1697160.0, ans=0.125 2023-11-21 23:05:54,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1697160.0, ans=0.0 2023-11-21 23:06:04,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1697226.6666666667, ans=22.5 2023-11-21 23:06:10,993 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.90 vs. limit=10.0 2023-11-21 23:06:13,804 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2100, loss[loss=0.09636, simple_loss=0.1338, pruned_loss=0.02119, audio_tagging_loss=0.008264, over 16041.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09451, pruned_loss=0.0157, audio_tagging_loss=0.009404, over 3047938.83 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:06:18,718 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254600 2023-11-21 23:06:33,146 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=15.0 2023-11-21 23:06:37,857 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.098e+01 8.657e+01 9.385e+01 1.571e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-21 23:07:07,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1697560.0, ans=0.0 2023-11-21 23:07:13,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.18 vs. limit=15.0 2023-11-21 23:07:18,035 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2150, loss[loss=0.07926, simple_loss=0.1144, pruned_loss=0.01361, audio_tagging_loss=0.008446, over 15847.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09451, pruned_loss=0.01562, audio_tagging_loss=0.009434, over 3048421.04 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:07:23,687 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254650 2023-11-21 23:07:41,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1697693.3333333333, ans=0.0 2023-11-21 23:07:52,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1697760.0, ans=0.1 2023-11-21 23:07:56,835 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:07:59,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1697826.6666666667, ans=0.125 2023-11-21 23:08:23,097 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2200, loss[loss=0.06358, simple_loss=0.08629, pruned_loss=0.0103, audio_tagging_loss=0.01014, over 15128.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09536, pruned_loss=0.01582, audio_tagging_loss=0.009356, over 3045955.36 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:08:27,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1697960.0, ans=0.1 2023-11-21 23:08:28,032 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.17 vs. limit=15.0 2023-11-21 23:08:28,712 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254700 2023-11-21 23:08:47,049 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.502e+01 8.027e+01 8.507e+01 9.488e+01 1.436e+02, threshold=1.701e+02, percent-clipped=0.0 2023-11-21 23:08:56,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1698093.3333333333, ans=0.125 2023-11-21 23:09:05,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1698160.0, ans=0.125 2023-11-21 23:09:19,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1698226.6666666667, ans=0.125 2023-11-21 23:09:27,613 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2250, loss[loss=0.08666, simple_loss=0.1169, pruned_loss=0.01956, audio_tagging_loss=0.008664, over 15249.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09494, pruned_loss=0.01569, audio_tagging_loss=0.009391, over 3047883.69 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 23:09:32,672 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254750 2023-11-21 23:09:50,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1698360.0, ans=0.125 2023-11-21 23:10:01,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1698426.6666666667, ans=0.125 2023-11-21 23:10:12,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1698493.3333333333, ans=0.125 2023-11-21 23:10:12,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1698493.3333333333, ans=0.125 2023-11-21 23:10:15,454 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.29 vs. limit=15.0 2023-11-21 23:10:18,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1698560.0, ans=10.0 2023-11-21 23:10:26,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1698560.0, ans=0.0 2023-11-21 23:10:26,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1698560.0, ans=0.125 2023-11-21 23:10:32,045 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2300, loss[loss=0.09905, simple_loss=0.1345, pruned_loss=0.02434, audio_tagging_loss=0.00744, over 15295.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09598, pruned_loss=0.01584, audio_tagging_loss=0.009386, over 3055276.85 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 23:10:38,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254800 2023-11-21 23:10:40,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1698626.6666666667, ans=0.0 2023-11-21 23:10:57,739 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.213e+01 8.771e+01 9.442e+01 1.167e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-21 23:11:14,566 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.45 vs. limit=22.5 2023-11-21 23:11:16,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1698826.6666666667, ans=0.125 2023-11-21 23:11:29,818 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:11:30,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.55 vs. limit=22.5 2023-11-21 23:11:37,158 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2350, loss[loss=0.09411, simple_loss=0.1302, pruned_loss=0.02088, audio_tagging_loss=0.008115, over 14603.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09531, pruned_loss=0.01581, audio_tagging_loss=0.009504, over 3055220.42 frames. ], batch size: 53, lr: 3.13e-03, grad_scale: 16.0 2023-11-21 23:11:43,417 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254850 2023-11-21 23:11:44,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1698960.0, ans=0.2 2023-11-21 23:11:50,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1699026.6666666667, ans=0.125 2023-11-21 23:11:57,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.54 vs. limit=15.0 2023-11-21 23:11:59,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1699026.6666666667, ans=0.2 2023-11-21 23:12:04,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1699093.3333333333, ans=0.125 2023-11-21 23:12:05,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1699093.3333333333, ans=0.1 2023-11-21 23:12:22,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1699160.0, ans=0.0 2023-11-21 23:12:29,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1699226.6666666667, ans=0.1 2023-11-21 23:12:35,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1699226.6666666667, ans=0.0 2023-11-21 23:12:40,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1699226.6666666667, ans=0.2 2023-11-21 23:12:42,281 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2400, loss[loss=0.06632, simple_loss=0.08935, pruned_loss=0.01206, audio_tagging_loss=0.009581, over 16202.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09573, pruned_loss=0.01606, audio_tagging_loss=0.009632, over 3052053.01 frames. ], batch size: 60, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:12:47,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254900 2023-11-21 23:12:56,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1699360.0, ans=0.0 2023-11-21 23:13:06,531 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 7.976e+01 8.691e+01 9.517e+01 1.439e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-21 23:13:18,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0 2023-11-21 23:13:25,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1699493.3333333333, ans=0.0 2023-11-21 23:13:28,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1699493.3333333333, ans=0.125 2023-11-21 23:13:29,815 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:13:45,575 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2450, loss[loss=0.06969, simple_loss=0.09025, pruned_loss=0.01469, audio_tagging_loss=0.009882, over 13851.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09609, pruned_loss=0.01611, audio_tagging_loss=0.00958, over 3047691.53 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:13:51,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 254950 2023-11-21 23:13:56,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1699626.6666666667, ans=0.125 2023-11-21 23:14:28,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1699826.6666666667, ans=0.125 2023-11-21 23:14:32,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1699826.6666666667, ans=0.0 2023-11-21 23:14:50,015 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2500, loss[loss=0.06636, simple_loss=0.08138, pruned_loss=0.0146, audio_tagging_loss=0.01107, over 14837.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09656, pruned_loss=0.01615, audio_tagging_loss=0.009608, over 3055140.73 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:14:54,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255000 2023-11-21 23:14:55,154 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:15:02,950 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.06 vs. limit=6.0 2023-11-21 23:15:14,912 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.176e+01 8.833e+01 9.682e+01 1.336e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-21 23:15:28,713 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.85 vs. limit=15.0 2023-11-21 23:15:37,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1700160.0, ans=0.2 2023-11-21 23:15:55,683 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2550, loss[loss=0.06913, simple_loss=0.08648, pruned_loss=0.01671, audio_tagging_loss=0.009175, over 15806.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09559, pruned_loss=0.01604, audio_tagging_loss=0.009576, over 3052122.10 frames. ], batch size: 61, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:16:00,729 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255050 2023-11-21 23:16:07,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=15.0 2023-11-21 23:16:09,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1700360.0, ans=0.0 2023-11-21 23:16:30,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1700426.6666666667, ans=0.0 2023-11-21 23:16:34,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1700493.3333333333, ans=0.0 2023-11-21 23:16:39,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1700493.3333333333, ans=0.0 2023-11-21 23:16:54,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1700560.0, ans=0.125 2023-11-21 23:17:00,055 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2600, loss[loss=0.06903, simple_loss=0.09662, pruned_loss=0.01315, audio_tagging_loss=0.007573, over 14846.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09508, pruned_loss=0.01605, audio_tagging_loss=0.009465, over 3046600.24 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:17:05,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255100 2023-11-21 23:17:21,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1700693.3333333333, ans=0.0 2023-11-21 23:17:22,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1700693.3333333333, ans=0.0 2023-11-21 23:17:24,810 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.035e+01 8.256e+01 8.734e+01 9.365e+01 1.832e+02, threshold=1.747e+02, percent-clipped=1.0 2023-11-21 23:17:29,976 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=12.0 2023-11-21 23:17:57,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1700893.3333333333, ans=0.0 2023-11-21 23:17:58,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1700893.3333333333, ans=0.125 2023-11-21 23:17:58,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1700893.3333333333, ans=0.0 2023-11-21 23:18:00,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.04 vs. limit=22.5 2023-11-21 23:18:05,219 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2650, loss[loss=0.06789, simple_loss=0.08182, pruned_loss=0.01537, audio_tagging_loss=0.01161, over 16039.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09364, pruned_loss=0.01568, audio_tagging_loss=0.009469, over 3044040.82 frames. ], batch size: 63, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:18:05,852 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=15.0 2023-11-21 23:18:10,052 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255150 2023-11-21 23:18:11,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1700960.0, ans=0.0 2023-11-21 23:18:20,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1701026.6666666667, ans=0.125 2023-11-21 23:18:20,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1701026.6666666667, ans=0.1 2023-11-21 23:19:00,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1701226.6666666667, ans=0.0 2023-11-21 23:19:09,742 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2700, loss[loss=0.06313, simple_loss=0.08003, pruned_loss=0.01158, audio_tagging_loss=0.01153, over 14870.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09314, pruned_loss=0.01562, audio_tagging_loss=0.009528, over 3051033.95 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:19:10,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1701293.3333333333, ans=0.125 2023-11-21 23:19:14,921 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255200 2023-11-21 23:19:22,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1701360.0, ans=0.125 2023-11-21 23:19:27,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1701360.0, ans=0.125 2023-11-21 23:19:34,280 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.128e+01 8.561e+01 9.314e+01 1.273e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-21 23:19:37,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1701426.6666666667, ans=0.125 2023-11-21 23:20:15,119 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2750, loss[loss=0.08459, simple_loss=0.1102, pruned_loss=0.01925, audio_tagging_loss=0.01023, over 15444.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09356, pruned_loss=0.01575, audio_tagging_loss=0.009552, over 3048418.48 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:20:17,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1701626.6666666667, ans=0.0 2023-11-21 23:20:20,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255250 2023-11-21 23:20:21,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1701626.6666666667, ans=0.125 2023-11-21 23:20:22,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1701626.6666666667, ans=0.0 2023-11-21 23:20:37,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1701693.3333333333, ans=0.2 2023-11-21 23:20:50,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1701760.0, ans=0.125 2023-11-21 23:20:53,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1701826.6666666667, ans=0.1 2023-11-21 23:20:56,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1701826.6666666667, ans=0.0 2023-11-21 23:21:06,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.whiten.whitening_limit, batch_count=1701893.3333333333, ans=12.0 2023-11-21 23:21:08,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1701893.3333333333, ans=0.125 2023-11-21 23:21:09,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1701893.3333333333, ans=0.0 2023-11-21 23:21:10,847 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:21:20,686 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2800, loss[loss=0.0892, simple_loss=0.1282, pruned_loss=0.01714, audio_tagging_loss=0.007949, over 15974.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09349, pruned_loss=0.01572, audio_tagging_loss=0.009565, over 3040244.40 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:21:23,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1701960.0, ans=0.1 2023-11-21 23:21:25,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255300 2023-11-21 23:21:32,299 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.15 vs. limit=15.0 2023-11-21 23:21:36,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1702026.6666666667, ans=0.0 2023-11-21 23:21:44,828 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 7.892e+01 8.472e+01 9.341e+01 1.172e+02, threshold=1.694e+02, percent-clipped=0.0 2023-11-21 23:22:01,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1702160.0, ans=0.125 2023-11-21 23:22:03,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1702160.0, ans=0.0 2023-11-21 23:22:11,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.79 vs. limit=15.0 2023-11-21 23:22:14,989 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:22:19,090 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.18 vs. limit=22.5 2023-11-21 23:22:22,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.80 vs. limit=12.0 2023-11-21 23:22:25,125 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2850, loss[loss=0.06701, simple_loss=0.08618, pruned_loss=0.01649, audio_tagging_loss=0.007431, over 14287.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09282, pruned_loss=0.0155, audio_tagging_loss=0.009469, over 3035525.46 frames. ], batch size: 55, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:22:30,150 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255350 2023-11-21 23:22:31,977 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=15.0 2023-11-21 23:22:40,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1702360.0, ans=0.0 2023-11-21 23:22:49,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1702426.6666666667, ans=0.0 2023-11-21 23:22:50,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1702426.6666666667, ans=0.2 2023-11-21 23:23:08,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1702493.3333333333, ans=0.07 2023-11-21 23:23:28,853 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2900, loss[loss=0.06302, simple_loss=0.08206, pruned_loss=0.01223, audio_tagging_loss=0.009762, over 14809.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09313, pruned_loss=0.01568, audio_tagging_loss=0.009476, over 3035821.20 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:23:29,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1702626.6666666667, ans=0.025 2023-11-21 23:23:30,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1702626.6666666667, ans=0.0 2023-11-21 23:23:33,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1702626.6666666667, ans=0.95 2023-11-21 23:23:34,425 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255400 2023-11-21 23:23:46,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.73 vs. limit=15.0 2023-11-21 23:23:54,131 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.570e+01 8.449e+01 9.076e+01 9.852e+01 1.265e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-21 23:23:55,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1702760.0, ans=0.1 2023-11-21 23:24:33,955 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 2950, loss[loss=0.08624, simple_loss=0.1091, pruned_loss=0.02095, audio_tagging_loss=0.01073, over 15225.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09358, pruned_loss=0.0157, audio_tagging_loss=0.009565, over 3035826.53 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:24:34,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1702960.0, ans=0.0 2023-11-21 23:24:38,978 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255450 2023-11-21 23:25:21,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1703160.0, ans=0.125 2023-11-21 23:25:25,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1703226.6666666667, ans=0.2 2023-11-21 23:25:37,734 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3000, loss[loss=0.06978, simple_loss=0.09159, pruned_loss=0.01486, audio_tagging_loss=0.009128, over 14949.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09445, pruned_loss=0.01577, audio_tagging_loss=0.00943, over 3044269.14 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:25:37,735 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-21 23:26:10,413 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.5015, 2.9443, 3.7829, 3.4231], device='cuda:3') 2023-11-21 23:26:16,399 INFO [train_asr.py:1253] (3/4) Epoch 22, validation: loss=0.05907, simple_loss=0.0519, pruned_loss=0.005126, audio_tagging_loss=0.02799, over 4681554.00 frames. 2023-11-21 23:26:16,400 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-21 23:26:21,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255500 2023-11-21 23:26:27,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1703293.3333333333, ans=0.0 2023-11-21 23:26:41,247 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.989e+01 8.698e+01 9.491e+01 1.260e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-21 23:26:54,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1703493.3333333333, ans=0.125 2023-11-21 23:27:21,946 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3050, loss[loss=0.07508, simple_loss=0.09863, pruned_loss=0.01797, audio_tagging_loss=0.007792, over 14772.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09412, pruned_loss=0.01572, audio_tagging_loss=0.009451, over 3038226.79 frames. ], batch size: 53, lr: 3.13e-03, grad_scale: 32.0 2023-11-21 23:27:26,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255550 2023-11-21 23:27:27,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1703626.6666666667, ans=0.125 2023-11-21 23:27:57,687 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:28:02,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1703826.6666666667, ans=0.0 2023-11-21 23:28:11,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1703893.3333333333, ans=0.0 2023-11-21 23:28:13,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1703893.3333333333, ans=0.2 2023-11-21 23:28:23,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1703893.3333333333, ans=0.0 2023-11-21 23:28:25,770 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3100, loss[loss=0.0948, simple_loss=0.1193, pruned_loss=0.02536, audio_tagging_loss=0.0098, over 15852.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09477, pruned_loss=0.01587, audio_tagging_loss=0.009485, over 3044237.22 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:28:30,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255600 2023-11-21 23:28:35,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1703960.0, ans=0.125 2023-11-21 23:28:39,016 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.77 vs. limit=15.0 2023-11-21 23:28:45,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.05 vs. limit=22.5 2023-11-21 23:28:45,281 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.39 vs. limit=15.0 2023-11-21 23:28:45,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1704026.6666666667, ans=0.125 2023-11-21 23:28:51,160 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.488e+01 7.990e+01 8.561e+01 9.403e+01 1.205e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-21 23:29:00,274 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.89 vs. limit=22.5 2023-11-21 23:29:15,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1704160.0, ans=0.125 2023-11-21 23:29:23,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1704226.6666666667, ans=0.125 2023-11-21 23:29:29,635 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3150, loss[loss=0.06817, simple_loss=0.08075, pruned_loss=0.01978, audio_tagging_loss=0.008017, over 16059.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.0943, pruned_loss=0.01574, audio_tagging_loss=0.009502, over 3045612.89 frames. ], batch size: 61, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:29:34,684 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255650 2023-11-21 23:29:36,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1704293.3333333333, ans=0.1 2023-11-21 23:29:43,136 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.04 vs. limit=15.0 2023-11-21 23:29:46,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1704360.0, ans=0.125 2023-11-21 23:30:02,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.98 vs. limit=15.0 2023-11-21 23:30:09,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1704493.3333333333, ans=0.1 2023-11-21 23:30:27,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1704560.0, ans=0.1 2023-11-21 23:30:35,313 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3200, loss[loss=0.06928, simple_loss=0.08716, pruned_loss=0.013, audio_tagging_loss=0.0127, over 14239.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.0954, pruned_loss=0.01595, audio_tagging_loss=0.009547, over 3046040.85 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:30:38,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.34 vs. limit=15.0 2023-11-21 23:30:40,484 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255700 2023-11-21 23:30:48,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1704693.3333333333, ans=0.1 2023-11-21 23:30:52,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1704693.3333333333, ans=0.0 2023-11-21 23:30:52,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1704693.3333333333, ans=0.0 2023-11-21 23:31:01,652 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.275e+01 8.900e+01 9.462e+01 1.368e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-21 23:31:10,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1704760.0, ans=0.1 2023-11-21 23:31:33,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1704893.3333333333, ans=0.125 2023-11-21 23:31:39,792 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3250, loss[loss=0.1042, simple_loss=0.1427, pruned_loss=0.02652, audio_tagging_loss=0.006298, over 14687.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.0955, pruned_loss=0.01595, audio_tagging_loss=0.009773, over 3039940.15 frames. ], batch size: 53, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:31:39,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1704960.0, ans=0.125 2023-11-21 23:31:44,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255750 2023-11-21 23:31:45,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1704960.0, ans=0.2 2023-11-21 23:31:56,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1705026.6666666667, ans=0.125 2023-11-21 23:32:05,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1705093.3333333333, ans=0.2 2023-11-21 23:32:07,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1705093.3333333333, ans=0.125 2023-11-21 23:32:10,484 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:32:11,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1705093.3333333333, ans=0.04949747468305833 2023-11-21 23:32:27,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1705160.0, ans=0.1 2023-11-21 23:32:34,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1705226.6666666667, ans=0.125 2023-11-21 23:32:37,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1705226.6666666667, ans=0.07 2023-11-21 23:32:43,662 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3300, loss[loss=0.07553, simple_loss=0.1093, pruned_loss=0.01242, audio_tagging_loss=0.008462, over 15640.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.09633, pruned_loss=0.01608, audio_tagging_loss=0.009693, over 3046355.85 frames. ], batch size: 60, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:32:48,606 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255800 2023-11-21 23:33:02,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1705360.0, ans=0.125 2023-11-21 23:33:11,150 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.185e+01 8.316e+01 8.844e+01 9.343e+01 1.264e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-21 23:33:28,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1705493.3333333333, ans=0.0 2023-11-21 23:33:33,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1705560.0, ans=0.0 2023-11-21 23:33:45,478 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.71 vs. limit=15.0 2023-11-21 23:33:46,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.23 vs. limit=22.5 2023-11-21 23:33:47,421 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3350, loss[loss=0.07169, simple_loss=0.0969, pruned_loss=0.01328, audio_tagging_loss=0.009968, over 14714.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09537, pruned_loss=0.0159, audio_tagging_loss=0.00962, over 3048205.24 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:33:53,022 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255850 2023-11-21 23:33:54,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1705626.6666666667, ans=0.2 2023-11-21 23:34:05,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1705693.3333333333, ans=0.1 2023-11-21 23:34:33,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.09 vs. limit=10.0 2023-11-21 23:34:52,877 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3400, loss[loss=0.0665, simple_loss=0.08921, pruned_loss=0.01511, audio_tagging_loss=0.006781, over 15658.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09544, pruned_loss=0.01577, audio_tagging_loss=0.009483, over 3051788.03 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:34:57,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255900 2023-11-21 23:35:07,674 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:35:10,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1706026.6666666667, ans=0.125 2023-11-21 23:35:18,271 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 8.132e+01 8.792e+01 9.497e+01 1.160e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-21 23:35:48,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1706226.6666666667, ans=0.2 2023-11-21 23:35:56,095 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3450, loss[loss=0.04943, simple_loss=0.05596, pruned_loss=0.01077, audio_tagging_loss=0.01069, over 15942.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09595, pruned_loss=0.01598, audio_tagging_loss=0.009399, over 3056477.15 frames. ], batch size: 62, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:36:01,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 255950 2023-11-21 23:36:14,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1706360.0, ans=0.2 2023-11-21 23:37:00,055 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3500, loss[loss=0.06338, simple_loss=0.0824, pruned_loss=0.01288, audio_tagging_loss=0.009302, over 15920.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09555, pruned_loss=0.01599, audio_tagging_loss=0.009276, over 3048745.47 frames. ], batch size: 61, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:37:05,559 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256000 2023-11-21 23:37:25,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1706693.3333333333, ans=0.05 2023-11-21 23:37:31,126 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 7.990e+01 8.553e+01 9.404e+01 1.412e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-21 23:37:37,242 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:37:58,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1706893.3333333333, ans=0.2 2023-11-21 23:38:08,314 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3550, loss[loss=0.06841, simple_loss=0.08303, pruned_loss=0.01763, audio_tagging_loss=0.009257, over 14425.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.09623, pruned_loss=0.01614, audio_tagging_loss=0.009186, over 3054713.26 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:38:13,911 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256050 2023-11-21 23:38:21,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1707026.6666666667, ans=0.0 2023-11-21 23:38:23,683 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:39:05,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1707226.6666666667, ans=0.125 2023-11-21 23:39:08,284 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.38 vs. limit=15.0 2023-11-21 23:39:12,519 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3600, loss[loss=0.08766, simple_loss=0.1129, pruned_loss=0.01914, audio_tagging_loss=0.01205, over 14403.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09495, pruned_loss=0.01601, audio_tagging_loss=0.009155, over 3047821.19 frames. ], batch size: 54, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:39:13,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.98 vs. limit=15.0 2023-11-21 23:39:17,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256100 2023-11-21 23:39:18,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1707293.3333333333, ans=0.0 2023-11-21 23:39:20,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1707293.3333333333, ans=0.0 2023-11-21 23:39:36,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1707360.0, ans=0.125 2023-11-21 23:39:39,532 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.698e+01 8.079e+01 8.522e+01 9.097e+01 1.276e+02, threshold=1.704e+02, percent-clipped=0.0 2023-11-21 23:39:39,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1707426.6666666667, ans=0.125 2023-11-21 23:39:51,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1707493.3333333333, ans=0.0 2023-11-21 23:40:10,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1707560.0, ans=0.125 2023-11-21 23:40:16,167 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3650, loss[loss=0.07219, simple_loss=0.08926, pruned_loss=0.01748, audio_tagging_loss=0.01008, over 14769.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09539, pruned_loss=0.01608, audio_tagging_loss=0.00909, over 3042053.47 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:40:19,867 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.36 vs. limit=22.5 2023-11-21 23:40:21,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256150 2023-11-21 23:40:23,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.02 vs. limit=15.0 2023-11-21 23:40:36,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1707693.3333333333, ans=0.09899494936611666 2023-11-21 23:40:43,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1707760.0, ans=0.125 2023-11-21 23:40:53,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1707760.0, ans=0.125 2023-11-21 23:41:02,199 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:41:09,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.03 vs. limit=10.0 2023-11-21 23:41:18,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1707893.3333333333, ans=0.025 2023-11-21 23:41:21,411 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3700, loss[loss=0.09192, simple_loss=0.1182, pruned_loss=0.02235, audio_tagging_loss=0.01046, over 14637.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09617, pruned_loss=0.01627, audio_tagging_loss=0.009143, over 3044084.23 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:41:24,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1707960.0, ans=0.125 2023-11-21 23:41:27,188 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256200 2023-11-21 23:41:50,131 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.207e+01 8.757e+01 9.528e+01 1.164e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-21 23:41:54,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=5.31 vs. limit=15.0 2023-11-21 23:42:09,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1708160.0, ans=0.125 2023-11-21 23:42:26,873 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3750, loss[loss=0.0767, simple_loss=0.09781, pruned_loss=0.01855, audio_tagging_loss=0.009238, over 15931.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09644, pruned_loss=0.01643, audio_tagging_loss=0.009173, over 3041241.09 frames. ], batch size: 61, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:42:27,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.77 vs. limit=12.0 2023-11-21 23:42:31,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256250 2023-11-21 23:42:44,970 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.74 vs. limit=10.0 2023-11-21 23:43:11,592 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:43:14,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1708493.3333333333, ans=0.125 2023-11-21 23:43:29,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.62 vs. limit=15.0 2023-11-21 23:43:30,483 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3800, loss[loss=0.089, simple_loss=0.117, pruned_loss=0.01915, audio_tagging_loss=0.01137, over 16307.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09684, pruned_loss=0.01638, audio_tagging_loss=0.009266, over 3048018.17 frames. ], batch size: 62, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:43:35,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1708626.6666666667, ans=0.125 2023-11-21 23:43:36,296 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256300 2023-11-21 23:43:59,425 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.403e+01 8.924e+01 9.481e+01 1.120e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-21 23:44:35,825 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3850, loss[loss=0.07041, simple_loss=0.09232, pruned_loss=0.01612, audio_tagging_loss=0.008131, over 14969.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09534, pruned_loss=0.01613, audio_tagging_loss=0.009353, over 3045548.04 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:44:40,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256350 2023-11-21 23:44:46,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1708960.0, ans=0.0 2023-11-21 23:45:03,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1709093.3333333333, ans=0.125 2023-11-21 23:45:08,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1709093.3333333333, ans=0.0 2023-11-21 23:45:16,656 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.35 vs. limit=22.5 2023-11-21 23:45:24,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0 2023-11-21 23:45:30,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1709226.6666666667, ans=0.125 2023-11-21 23:45:34,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1709226.6666666667, ans=0.125 2023-11-21 23:45:34,641 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-21 23:45:39,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1709293.3333333333, ans=0.1 2023-11-21 23:45:40,178 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3900, loss[loss=0.07878, simple_loss=0.1019, pruned_loss=0.01735, audio_tagging_loss=0.01047, over 16022.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09588, pruned_loss=0.01639, audio_tagging_loss=0.009433, over 3040878.84 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:45:45,218 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256400 2023-11-21 23:45:59,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1709360.0, ans=0.0 2023-11-21 23:46:09,346 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.019e+01 8.673e+01 9.477e+01 1.418e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-21 23:46:23,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1709493.3333333333, ans=0.125 2023-11-21 23:46:30,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1709560.0, ans=0.125 2023-11-21 23:46:45,104 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 3950, loss[loss=0.06606, simple_loss=0.08377, pruned_loss=0.01555, audio_tagging_loss=0.008625, over 15615.00 frames. ], tot_loss[loss=0.0746, simple_loss=0.09719, pruned_loss=0.01661, audio_tagging_loss=0.009397, over 3045259.97 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 16.0 2023-11-21 23:46:50,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256450 2023-11-21 23:47:08,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1709693.3333333333, ans=0.0 2023-11-21 23:47:25,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1709826.6666666667, ans=0.125 2023-11-21 23:47:25,614 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.77 vs. limit=15.0 2023-11-21 23:47:42,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1709893.3333333333, ans=0.05 2023-11-21 23:47:49,834 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4000, loss[loss=0.05918, simple_loss=0.07318, pruned_loss=0.01339, audio_tagging_loss=0.009197, over 15443.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.0967, pruned_loss=0.01654, audio_tagging_loss=0.009563, over 3040049.68 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:47:52,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1709960.0, ans=0.0 2023-11-21 23:47:54,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256500 2023-11-21 23:47:56,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1709960.0, ans=0.2 2023-11-21 23:48:07,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1710026.6666666667, ans=0.125 2023-11-21 23:48:11,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1710026.6666666667, ans=15.0 2023-11-21 23:48:13,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1710093.3333333333, ans=0.0 2023-11-21 23:48:17,179 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.007e+01 8.495e+01 8.892e+01 9.387e+01 1.240e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-21 23:48:20,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1710093.3333333333, ans=0.1 2023-11-21 23:48:20,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1710093.3333333333, ans=0.0 2023-11-21 23:48:22,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1710093.3333333333, ans=0.0 2023-11-21 23:48:42,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1710226.6666666667, ans=0.07 2023-11-21 23:48:43,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1710226.6666666667, ans=0.125 2023-11-21 23:48:47,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1710226.6666666667, ans=0.125 2023-11-21 23:48:53,505 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4050, loss[loss=0.07172, simple_loss=0.09423, pruned_loss=0.0141, audio_tagging_loss=0.01051, over 15033.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09523, pruned_loss=0.01614, audio_tagging_loss=0.009779, over 3039646.13 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:48:57,225 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:48:57,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1710293.3333333333, ans=0.0 2023-11-21 23:48:58,520 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256550 2023-11-21 23:49:21,322 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.67 vs. limit=10.0 2023-11-21 23:49:25,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1710426.6666666667, ans=0.125 2023-11-21 23:49:41,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1710493.3333333333, ans=0.0 2023-11-21 23:49:51,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1710560.0, ans=0.0 2023-11-21 23:49:57,002 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4100, loss[loss=0.0678, simple_loss=0.09703, pruned_loss=0.0129, audio_tagging_loss=0.006386, over 14944.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09554, pruned_loss=0.01606, audio_tagging_loss=0.009692, over 3046384.63 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:50:02,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256600 2023-11-21 23:50:22,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1710760.0, ans=0.125 2023-11-21 23:50:23,702 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.70 vs. limit=15.0 2023-11-21 23:50:24,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1710760.0, ans=0.125 2023-11-21 23:50:26,962 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.295e+01 8.154e+01 8.730e+01 9.411e+01 1.441e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-21 23:50:28,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1710760.0, ans=0.125 2023-11-21 23:50:41,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1710826.6666666667, ans=0.125 2023-11-21 23:50:56,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1710893.3333333333, ans=0.125 2023-11-21 23:50:57,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.37 vs. limit=15.0 2023-11-21 23:50:57,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1710893.3333333333, ans=0.0 2023-11-21 23:51:03,077 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4150, loss[loss=0.05726, simple_loss=0.06551, pruned_loss=0.01537, audio_tagging_loss=0.009135, over 15169.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09636, pruned_loss=0.01624, audio_tagging_loss=0.009594, over 3055746.30 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:51:08,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256650 2023-11-21 23:51:08,634 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.20 vs. limit=6.0 2023-11-21 23:51:12,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.74 vs. limit=22.5 2023-11-21 23:51:16,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1711026.6666666667, ans=0.0 2023-11-21 23:51:21,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1711026.6666666667, ans=0.2 2023-11-21 23:51:30,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1711093.3333333333, ans=0.1 2023-11-21 23:51:50,774 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-21 23:51:57,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1711226.6666666667, ans=0.125 2023-11-21 23:52:01,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1711226.6666666667, ans=0.95 2023-11-21 23:52:08,108 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4200, loss[loss=0.07068, simple_loss=0.09614, pruned_loss=0.01527, audio_tagging_loss=0.007342, over 15391.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09625, pruned_loss=0.01613, audio_tagging_loss=0.009364, over 3054329.23 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:52:13,149 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256700 2023-11-21 23:52:19,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1711360.0, ans=0.2 2023-11-21 23:52:23,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1711360.0, ans=0.1 2023-11-21 23:52:36,104 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 7.996e+01 8.840e+01 9.314e+01 1.225e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-21 23:52:38,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1711426.6666666667, ans=0.125 2023-11-21 23:52:45,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1711493.3333333333, ans=0.0 2023-11-21 23:53:11,827 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4250, loss[loss=0.07584, simple_loss=0.1064, pruned_loss=0.01582, audio_tagging_loss=0.006824, over 15174.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09602, pruned_loss=0.016, audio_tagging_loss=0.009484, over 3064279.78 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:53:16,993 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256750 2023-11-21 23:53:31,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1711693.3333333333, ans=0.125 2023-11-21 23:53:37,691 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.44 vs. limit=10.0 2023-11-21 23:53:53,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1711826.6666666667, ans=0.05 2023-11-21 23:53:58,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1711826.6666666667, ans=0.1 2023-11-21 23:54:04,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1711893.3333333333, ans=0.125 2023-11-21 23:54:07,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1711893.3333333333, ans=0.035 2023-11-21 23:54:08,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1711893.3333333333, ans=0.1 2023-11-21 23:54:10,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1711893.3333333333, ans=0.025 2023-11-21 23:54:16,092 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4300, loss[loss=0.09056, simple_loss=0.1191, pruned_loss=0.02083, audio_tagging_loss=0.01017, over 16008.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09726, pruned_loss=0.01635, audio_tagging_loss=0.009382, over 3062191.80 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:54:20,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1711960.0, ans=0.125 2023-11-21 23:54:22,250 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256800 2023-11-21 23:54:28,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1712026.6666666667, ans=0.2 2023-11-21 23:54:45,105 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.276e+01 8.988e+01 9.758e+01 2.160e+02, threshold=1.798e+02, percent-clipped=1.0 2023-11-21 23:55:18,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1712226.6666666667, ans=0.2 2023-11-21 23:55:22,054 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4350, loss[loss=0.0693, simple_loss=0.09288, pruned_loss=0.01525, audio_tagging_loss=0.007616, over 16345.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09696, pruned_loss=0.0162, audio_tagging_loss=0.009342, over 3049156.41 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:55:26,986 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256850 2023-11-21 23:55:31,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.02 vs. limit=15.0 2023-11-21 23:56:15,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1712560.0, ans=0.125 2023-11-21 23:56:25,329 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4400, loss[loss=0.07719, simple_loss=0.111, pruned_loss=0.01333, audio_tagging_loss=0.008351, over 16074.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09782, pruned_loss=0.01627, audio_tagging_loss=0.009256, over 3054669.07 frames. ], batch size: 60, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:56:25,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1712626.6666666667, ans=0.0 2023-11-21 23:56:29,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1712626.6666666667, ans=0.125 2023-11-21 23:56:30,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256900 2023-11-21 23:56:34,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1712626.6666666667, ans=0.2 2023-11-21 23:56:55,223 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.141e+01 8.812e+01 9.440e+01 1.228e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-21 23:57:29,781 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4450, loss[loss=0.06098, simple_loss=0.07526, pruned_loss=0.01347, audio_tagging_loss=0.009879, over 15374.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09757, pruned_loss=0.01617, audio_tagging_loss=0.009222, over 3063868.46 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:57:35,262 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 256950 2023-11-21 23:57:36,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1712960.0, ans=0.125 2023-11-21 23:58:08,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1713160.0, ans=0.125 2023-11-21 23:58:21,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1713226.6666666667, ans=0.1 2023-11-21 23:58:30,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.98 vs. limit=15.0 2023-11-21 23:58:31,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1713226.6666666667, ans=0.1 2023-11-21 23:58:31,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=1713226.6666666667, ans=0.02 2023-11-21 23:58:32,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1713226.6666666667, ans=0.125 2023-11-21 23:58:34,705 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4500, loss[loss=0.06367, simple_loss=0.08785, pruned_loss=0.01152, audio_tagging_loss=0.008223, over 15848.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09751, pruned_loss=0.01619, audio_tagging_loss=0.009151, over 3059403.65 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:58:40,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257000 2023-11-21 23:58:59,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1713426.6666666667, ans=0.125 2023-11-21 23:59:01,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1713426.6666666667, ans=0.1 2023-11-21 23:59:01,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1713426.6666666667, ans=0.1 2023-11-21 23:59:03,857 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.360e+01 9.049e+01 9.861e+01 1.318e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-21 23:59:08,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1713426.6666666667, ans=0.125 2023-11-21 23:59:12,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1713493.3333333333, ans=0.1 2023-11-21 23:59:39,443 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4550, loss[loss=0.06936, simple_loss=0.09023, pruned_loss=0.01659, audio_tagging_loss=0.007657, over 14093.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09736, pruned_loss=0.01615, audio_tagging_loss=0.009095, over 3053134.46 frames. ], batch size: 54, lr: 3.12e-03, grad_scale: 32.0 2023-11-21 23:59:44,409 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257050 2023-11-21 23:59:50,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1713693.3333333333, ans=0.125 2023-11-22 00:00:09,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1713760.0, ans=0.2 2023-11-22 00:00:27,883 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 00:00:40,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1713893.3333333333, ans=0.2 2023-11-22 00:00:42,549 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4600, loss[loss=0.05969, simple_loss=0.08146, pruned_loss=0.01106, audio_tagging_loss=0.007901, over 14850.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09622, pruned_loss=0.01603, audio_tagging_loss=0.009255, over 3044594.44 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 32.0 2023-11-22 00:00:48,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257100 2023-11-22 00:00:51,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1713960.0, ans=0.125 2023-11-22 00:01:13,015 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 7.906e+01 8.473e+01 9.187e+01 1.183e+02, threshold=1.695e+02, percent-clipped=0.0 2023-11-22 00:01:22,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1714160.0, ans=0.0 2023-11-22 00:01:31,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1714160.0, ans=0.0 2023-11-22 00:01:31,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1714160.0, ans=0.1 2023-11-22 00:01:34,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1714226.6666666667, ans=0.125 2023-11-22 00:01:36,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1714226.6666666667, ans=0.125 2023-11-22 00:01:41,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1714226.6666666667, ans=0.1 2023-11-22 00:01:47,036 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4650, loss[loss=0.06595, simple_loss=0.08571, pruned_loss=0.013, audio_tagging_loss=0.01009, over 15020.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09528, pruned_loss=0.01583, audio_tagging_loss=0.00931, over 3041115.56 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0 2023-11-22 00:01:48,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=1714293.3333333333, ans=0.1 2023-11-22 00:01:53,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257150 2023-11-22 00:01:57,184 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:02:11,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1714426.6666666667, ans=0.1 2023-11-22 00:02:33,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1714493.3333333333, ans=0.1 2023-11-22 00:02:45,321 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:02:51,272 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4700, loss[loss=0.06502, simple_loss=0.08161, pruned_loss=0.01477, audio_tagging_loss=0.00944, over 15004.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.0962, pruned_loss=0.01609, audio_tagging_loss=0.009382, over 3047636.29 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0 2023-11-22 00:02:56,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257200 2023-11-22 00:03:03,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1714693.3333333333, ans=0.125 2023-11-22 00:03:21,300 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.209e+01 8.969e+01 9.577e+01 1.305e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-22 00:03:45,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1714893.3333333333, ans=0.125 2023-11-22 00:03:52,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1714893.3333333333, ans=0.1 2023-11-22 00:03:56,079 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4750, loss[loss=0.06709, simple_loss=0.0903, pruned_loss=0.01429, audio_tagging_loss=0.007654, over 15424.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09526, pruned_loss=0.01601, audio_tagging_loss=0.009521, over 3038783.01 frames. ], batch size: 60, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:04:00,980 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.79 vs. limit=22.5 2023-11-22 00:04:01,727 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257250 2023-11-22 00:04:07,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1714960.0, ans=0.1 2023-11-22 00:04:41,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1715160.0, ans=0.1 2023-11-22 00:04:54,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1715226.6666666667, ans=0.125 2023-11-22 00:05:00,625 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4800, loss[loss=0.07591, simple_loss=0.09514, pruned_loss=0.01807, audio_tagging_loss=0.01027, over 15499.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09492, pruned_loss=0.01583, audio_tagging_loss=0.009546, over 3045589.10 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:05:02,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1715293.3333333333, ans=0.125 2023-11-22 00:05:05,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257300 2023-11-22 00:05:14,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1715360.0, ans=0.125 2023-11-22 00:05:26,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1715426.6666666667, ans=0.1 2023-11-22 00:05:30,010 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.329e+01 8.144e+01 8.815e+01 9.467e+01 1.388e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-22 00:05:33,004 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.50 vs. limit=15.0 2023-11-22 00:05:35,657 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.31 vs. limit=22.5 2023-11-22 00:05:36,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1715426.6666666667, ans=0.2 2023-11-22 00:05:56,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1715560.0, ans=0.2 2023-11-22 00:06:05,295 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4850, loss[loss=0.06209, simple_loss=0.08604, pruned_loss=0.009844, audio_tagging_loss=0.009225, over 15555.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09363, pruned_loss=0.01555, audio_tagging_loss=0.009677, over 3041156.09 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:06:09,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1715626.6666666667, ans=0.0 2023-11-22 00:06:10,256 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257350 2023-11-22 00:06:44,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1715826.6666666667, ans=0.04949747468305833 2023-11-22 00:06:48,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1715826.6666666667, ans=0.0 2023-11-22 00:07:09,119 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4900, loss[loss=0.06641, simple_loss=0.08478, pruned_loss=0.01546, audio_tagging_loss=0.008555, over 15542.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09415, pruned_loss=0.01576, audio_tagging_loss=0.009565, over 3035828.32 frames. ], batch size: 60, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:07:10,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1715960.0, ans=0.125 2023-11-22 00:07:14,208 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257400 2023-11-22 00:07:38,984 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.481e+01 8.110e+01 8.831e+01 9.694e+01 1.184e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 00:07:41,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1716093.3333333333, ans=0.2 2023-11-22 00:07:42,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.76 vs. limit=10.0 2023-11-22 00:07:44,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1716093.3333333333, ans=0.125 2023-11-22 00:07:52,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1716160.0, ans=0.125 2023-11-22 00:08:14,177 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 4950, loss[loss=0.06497, simple_loss=0.07943, pruned_loss=0.0137, audio_tagging_loss=0.01155, over 14285.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09503, pruned_loss=0.01576, audio_tagging_loss=0.009513, over 3041070.38 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:08:19,135 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257450 2023-11-22 00:08:19,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1716293.3333333333, ans=0.0 2023-11-22 00:08:47,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1716426.6666666667, ans=0.0 2023-11-22 00:09:10,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1716560.0, ans=0.0 2023-11-22 00:09:18,237 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5000, loss[loss=0.05278, simple_loss=0.07139, pruned_loss=0.008061, audio_tagging_loss=0.009025, over 13972.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09506, pruned_loss=0.01558, audio_tagging_loss=0.009456, over 3039627.10 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:09:23,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257500 2023-11-22 00:09:31,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1716693.3333333333, ans=0.125 2023-11-22 00:09:48,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.146e+01 8.929e+01 9.625e+01 1.123e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 00:09:48,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1716760.0, ans=0.04949747468305833 2023-11-22 00:09:50,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=12.0 2023-11-22 00:09:57,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1716826.6666666667, ans=0.0 2023-11-22 00:10:07,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.74 vs. limit=15.0 2023-11-22 00:10:22,323 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5050, loss[loss=0.04605, simple_loss=0.04722, pruned_loss=0.009707, audio_tagging_loss=0.01273, over 14524.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09461, pruned_loss=0.01566, audio_tagging_loss=0.009472, over 3039752.68 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:10:27,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257550 2023-11-22 00:10:28,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1716960.0, ans=0.0 2023-11-22 00:10:59,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1717093.3333333333, ans=0.125 2023-11-22 00:11:01,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1717160.0, ans=0.04949747468305833 2023-11-22 00:11:17,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1717226.6666666667, ans=0.0 2023-11-22 00:11:28,011 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5100, loss[loss=0.07272, simple_loss=0.09113, pruned_loss=0.01528, audio_tagging_loss=0.01187, over 16837.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09586, pruned_loss=0.01593, audio_tagging_loss=0.009374, over 3044175.10 frames. ], batch size: 63, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:11:28,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1717293.3333333333, ans=0.125 2023-11-22 00:11:33,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257600 2023-11-22 00:11:49,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1717360.0, ans=0.125 2023-11-22 00:11:58,356 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.309e+01 7.923e+01 8.751e+01 9.756e+01 1.347e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 00:11:58,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1717426.6666666667, ans=0.125 2023-11-22 00:12:01,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1717426.6666666667, ans=0.125 2023-11-22 00:12:34,127 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5150, loss[loss=0.06998, simple_loss=0.1026, pruned_loss=0.01199, audio_tagging_loss=0.006659, over 15812.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09464, pruned_loss=0.01567, audio_tagging_loss=0.00945, over 3039202.24 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:12:39,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257650 2023-11-22 00:12:53,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1717693.3333333333, ans=0.125 2023-11-22 00:13:16,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1717826.6666666667, ans=0.0 2023-11-22 00:13:31,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1717893.3333333333, ans=0.04949747468305833 2023-11-22 00:13:37,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1717960.0, ans=0.125 2023-11-22 00:13:39,420 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5200, loss[loss=0.05619, simple_loss=0.0719, pruned_loss=0.00934, audio_tagging_loss=0.0109, over 16385.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09547, pruned_loss=0.01589, audio_tagging_loss=0.0094, over 3041544.90 frames. ], batch size: 62, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:13:44,453 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257700 2023-11-22 00:13:49,753 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.01 vs. limit=10.0 2023-11-22 00:13:59,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1718026.6666666667, ans=0.125 2023-11-22 00:14:09,399 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.143e+01 8.728e+01 9.361e+01 2.351e+02, threshold=1.746e+02, percent-clipped=1.0 2023-11-22 00:14:31,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1718226.6666666667, ans=0.1 2023-11-22 00:14:43,503 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5250, loss[loss=0.05046, simple_loss=0.05734, pruned_loss=0.01137, audio_tagging_loss=0.01042, over 15986.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09659, pruned_loss=0.01603, audio_tagging_loss=0.009271, over 3040533.07 frames. ], batch size: 62, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:14:49,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257750 2023-11-22 00:14:50,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1718293.3333333333, ans=0.125 2023-11-22 00:14:53,364 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.39 vs. limit=15.0 2023-11-22 00:15:09,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1718426.6666666667, ans=0.2 2023-11-22 00:15:17,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1718426.6666666667, ans=0.0 2023-11-22 00:15:19,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1718426.6666666667, ans=0.2 2023-11-22 00:15:35,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1718560.0, ans=0.04949747468305833 2023-11-22 00:15:35,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1718560.0, ans=0.125 2023-11-22 00:15:36,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1718560.0, ans=0.125 2023-11-22 00:15:43,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1718560.0, ans=0.1 2023-11-22 00:15:48,191 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5300, loss[loss=0.0878, simple_loss=0.1258, pruned_loss=0.01676, audio_tagging_loss=0.008157, over 15198.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09619, pruned_loss=0.01588, audio_tagging_loss=0.009242, over 3039424.03 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:15:53,216 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257800 2023-11-22 00:15:59,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1718693.3333333333, ans=0.2 2023-11-22 00:16:17,183 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.246e+01 8.912e+01 9.804e+01 1.159e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-22 00:16:23,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1718760.0, ans=0.0 2023-11-22 00:16:31,529 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.56 vs. limit=22.5 2023-11-22 00:16:52,132 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5350, loss[loss=0.06836, simple_loss=0.0939, pruned_loss=0.01357, audio_tagging_loss=0.007846, over 14693.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09509, pruned_loss=0.01565, audio_tagging_loss=0.009313, over 3036489.71 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:16:52,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1718960.0, ans=0.125 2023-11-22 00:16:54,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1718960.0, ans=0.125 2023-11-22 00:16:57,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257850 2023-11-22 00:17:36,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1719160.0, ans=0.125 2023-11-22 00:17:47,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1719226.6666666667, ans=0.0 2023-11-22 00:17:49,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.63 vs. limit=15.0 2023-11-22 00:17:57,063 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5400, loss[loss=0.08225, simple_loss=0.1066, pruned_loss=0.02055, audio_tagging_loss=0.008401, over 15418.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09516, pruned_loss=0.01559, audio_tagging_loss=0.009322, over 3039321.89 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:18:02,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257900 2023-11-22 00:18:10,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1719360.0, ans=0.0 2023-11-22 00:18:11,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1719360.0, ans=0.0 2023-11-22 00:18:17,139 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.15 vs. limit=12.0 2023-11-22 00:18:22,369 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.67 vs. limit=10.0 2023-11-22 00:18:27,539 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 7.882e+01 8.565e+01 9.189e+01 1.192e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-22 00:18:54,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1719560.0, ans=0.0 2023-11-22 00:19:01,109 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5450, loss[loss=0.09732, simple_loss=0.1204, pruned_loss=0.02791, audio_tagging_loss=0.009195, over 15235.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09655, pruned_loss=0.01602, audio_tagging_loss=0.009255, over 3039689.16 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:19:06,710 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 257950 2023-11-22 00:19:21,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1719693.3333333333, ans=0.0 2023-11-22 00:19:53,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1719893.3333333333, ans=0.0 2023-11-22 00:20:00,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1719893.3333333333, ans=0.0 2023-11-22 00:20:04,968 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=22.5 2023-11-22 00:20:05,630 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5500, loss[loss=0.07477, simple_loss=0.1011, pruned_loss=0.01588, audio_tagging_loss=0.008363, over 15583.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09707, pruned_loss=0.01608, audio_tagging_loss=0.009284, over 3041781.21 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:20:10,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258000 2023-11-22 00:20:13,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1719960.0, ans=0.1 2023-11-22 00:20:17,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1720026.6666666667, ans=0.0 2023-11-22 00:20:36,642 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.285e+01 8.796e+01 9.521e+01 1.194e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 00:20:50,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1720160.0, ans=0.125 2023-11-22 00:21:03,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1720226.6666666667, ans=0.125 2023-11-22 00:21:09,030 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5550, loss[loss=0.08307, simple_loss=0.09586, pruned_loss=0.01943, audio_tagging_loss=0.01571, over 15605.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09595, pruned_loss=0.01585, audio_tagging_loss=0.009382, over 3041951.76 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:21:14,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258050 2023-11-22 00:21:16,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1720293.3333333333, ans=0.125 2023-11-22 00:21:19,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1720293.3333333333, ans=0.125 2023-11-22 00:21:21,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1720293.3333333333, ans=0.0 2023-11-22 00:22:01,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1720560.0, ans=0.2 2023-11-22 00:22:10,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1720560.0, ans=0.0 2023-11-22 00:22:10,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.50 vs. limit=15.0 2023-11-22 00:22:13,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.70 vs. limit=15.0 2023-11-22 00:22:13,627 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5600, loss[loss=0.07898, simple_loss=0.1017, pruned_loss=0.01462, audio_tagging_loss=0.01352, over 15790.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09518, pruned_loss=0.01568, audio_tagging_loss=0.009538, over 3048252.69 frames. ], batch size: 60, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:22:19,192 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258100 2023-11-22 00:22:19,938 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.03 vs. limit=15.0 2023-11-22 00:22:37,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1720760.0, ans=0.0 2023-11-22 00:22:43,448 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 7.788e+01 8.528e+01 9.166e+01 1.353e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-22 00:22:51,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1720826.6666666667, ans=0.125 2023-11-22 00:22:53,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.77 vs. limit=15.0 2023-11-22 00:22:59,878 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 00:23:01,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1720826.6666666667, ans=0.1 2023-11-22 00:23:09,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1720893.3333333333, ans=0.0 2023-11-22 00:23:11,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1720893.3333333333, ans=0.125 2023-11-22 00:23:16,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1720960.0, ans=0.125 2023-11-22 00:23:17,562 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5650, loss[loss=0.05885, simple_loss=0.06508, pruned_loss=0.01074, audio_tagging_loss=0.01558, over 14177.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09546, pruned_loss=0.01593, audio_tagging_loss=0.009639, over 3051262.46 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:23:22,662 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258150 2023-11-22 00:23:30,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1721026.6666666667, ans=0.0 2023-11-22 00:23:43,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.98 vs. limit=10.0 2023-11-22 00:23:51,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1721093.3333333333, ans=0.07 2023-11-22 00:24:21,024 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5700, loss[loss=0.06231, simple_loss=0.07988, pruned_loss=0.01278, audio_tagging_loss=0.009595, over 16262.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09561, pruned_loss=0.01615, audio_tagging_loss=0.009669, over 3052898.92 frames. ], batch size: 64, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:24:25,940 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258200 2023-11-22 00:24:38,430 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.83 vs. limit=15.0 2023-11-22 00:24:41,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1721360.0, ans=0.0 2023-11-22 00:24:49,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1721426.6666666667, ans=0.125 2023-11-22 00:24:52,712 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.074e+01 8.821e+01 9.592e+01 1.274e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 00:25:26,455 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5750, loss[loss=0.1012, simple_loss=0.117, pruned_loss=0.03356, audio_tagging_loss=0.009093, over 15106.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.09553, pruned_loss=0.01615, audio_tagging_loss=0.009513, over 3054920.95 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:25:29,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1721626.6666666667, ans=0.125 2023-11-22 00:25:31,507 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258250 2023-11-22 00:25:36,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.50 vs. limit=15.0 2023-11-22 00:25:37,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1721626.6666666667, ans=0.125 2023-11-22 00:25:42,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1721693.3333333333, ans=0.2 2023-11-22 00:25:56,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1721760.0, ans=0.0 2023-11-22 00:26:07,968 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:26:11,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1721826.6666666667, ans=0.0 2023-11-22 00:26:28,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1721960.0, ans=0.125 2023-11-22 00:26:30,006 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5800, loss[loss=0.07592, simple_loss=0.09943, pruned_loss=0.01625, audio_tagging_loss=0.009953, over 14881.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09576, pruned_loss=0.01608, audio_tagging_loss=0.009402, over 3049095.30 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:26:34,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258300 2023-11-22 00:26:37,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1721960.0, ans=0.1 2023-11-22 00:26:39,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1721960.0, ans=22.5 2023-11-22 00:26:40,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1721960.0, ans=0.1 2023-11-22 00:26:47,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1722026.6666666667, ans=0.04949747468305833 2023-11-22 00:27:01,816 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.499e+01 8.125e+01 8.820e+01 9.414e+01 1.363e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 00:27:04,669 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:27:11,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1722160.0, ans=0.125 2023-11-22 00:27:13,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1722160.0, ans=0.07 2023-11-22 00:27:21,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1722226.6666666667, ans=0.05 2023-11-22 00:27:33,615 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5850, loss[loss=0.0656, simple_loss=0.08683, pruned_loss=0.01319, audio_tagging_loss=0.008997, over 17020.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09593, pruned_loss=0.01598, audio_tagging_loss=0.009306, over 3051031.10 frames. ], batch size: 63, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:27:38,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258350 2023-11-22 00:27:45,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1722360.0, ans=0.015 2023-11-22 00:27:52,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1722360.0, ans=0.0 2023-11-22 00:27:53,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1722360.0, ans=0.125 2023-11-22 00:28:08,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.78 vs. limit=15.0 2023-11-22 00:28:24,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1722560.0, ans=0.0 2023-11-22 00:28:33,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1722560.0, ans=0.1 2023-11-22 00:28:38,104 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5900, loss[loss=0.07091, simple_loss=0.08827, pruned_loss=0.01475, audio_tagging_loss=0.01202, over 15394.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09692, pruned_loss=0.01602, audio_tagging_loss=0.009228, over 3050666.49 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:28:43,224 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258400 2023-11-22 00:29:10,518 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.273e+01 8.751e+01 9.457e+01 2.979e+02, threshold=1.750e+02, percent-clipped=1.0 2023-11-22 00:29:14,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1722760.0, ans=0.2 2023-11-22 00:29:22,382 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:29:26,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1722826.6666666667, ans=0.1 2023-11-22 00:29:43,313 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 5950, loss[loss=0.07659, simple_loss=0.09711, pruned_loss=0.01868, audio_tagging_loss=0.009356, over 14700.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09691, pruned_loss=0.01594, audio_tagging_loss=0.009242, over 3046754.88 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 16.0 2023-11-22 00:29:46,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1722960.0, ans=0.2 2023-11-22 00:29:48,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258450 2023-11-22 00:30:00,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1723026.6666666667, ans=0.0 2023-11-22 00:30:00,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1723026.6666666667, ans=0.1 2023-11-22 00:30:04,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1723026.6666666667, ans=0.1 2023-11-22 00:30:19,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1723093.3333333333, ans=0.125 2023-11-22 00:30:44,111 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=15.41 vs. limit=15.0 2023-11-22 00:30:46,891 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6000, loss[loss=0.06261, simple_loss=0.07783, pruned_loss=0.01609, audio_tagging_loss=0.007608, over 14495.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09597, pruned_loss=0.01583, audio_tagging_loss=0.009322, over 3038316.62 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:30:46,892 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 00:31:27,175 INFO [train_asr.py:1253] (3/4) Epoch 22, validation: loss=0.05958, simple_loss=0.05193, pruned_loss=0.005178, audio_tagging_loss=0.02843, over 4681554.00 frames. 2023-11-22 00:31:27,176 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 00:31:31,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1723293.3333333333, ans=0.0 2023-11-22 00:31:32,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258500 2023-11-22 00:31:43,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1723360.0, ans=0.125 2023-11-22 00:31:58,331 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.255e+01 8.695e+01 9.611e+01 1.390e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 00:32:04,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1723493.3333333333, ans=0.125 2023-11-22 00:32:14,260 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 00:32:29,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1723626.6666666667, ans=0.125 2023-11-22 00:32:31,179 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6050, loss[loss=0.05395, simple_loss=0.06377, pruned_loss=0.01099, audio_tagging_loss=0.01107, over 15341.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09555, pruned_loss=0.01575, audio_tagging_loss=0.009366, over 3040528.10 frames. ], batch size: 60, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:32:35,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1723626.6666666667, ans=0.0 2023-11-22 00:32:36,090 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258550 2023-11-22 00:32:53,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1723693.3333333333, ans=0.125 2023-11-22 00:33:02,219 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.81 vs. limit=15.0 2023-11-22 00:33:27,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1723893.3333333333, ans=0.0 2023-11-22 00:33:34,440 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6100, loss[loss=0.08422, simple_loss=0.1176, pruned_loss=0.01945, audio_tagging_loss=0.005954, over 16267.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09639, pruned_loss=0.01595, audio_tagging_loss=0.00932, over 3047684.99 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:33:39,396 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258600 2023-11-22 00:33:42,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.07 vs. limit=10.0 2023-11-22 00:33:46,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1724026.6666666667, ans=0.035 2023-11-22 00:34:04,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1724093.3333333333, ans=0.125 2023-11-22 00:34:07,005 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.187e+01 8.838e+01 9.553e+01 1.618e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-22 00:34:39,402 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6150, loss[loss=0.06903, simple_loss=0.08911, pruned_loss=0.01663, audio_tagging_loss=0.007845, over 15405.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09616, pruned_loss=0.01605, audio_tagging_loss=0.009365, over 3050115.30 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:34:44,419 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258650 2023-11-22 00:34:44,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1724293.3333333333, ans=0.125 2023-11-22 00:34:57,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.58 vs. limit=12.0 2023-11-22 00:35:27,579 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.47 vs. limit=10.0 2023-11-22 00:35:33,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1724560.0, ans=0.125 2023-11-22 00:35:37,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1724560.0, ans=0.1 2023-11-22 00:35:43,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.35 vs. limit=12.0 2023-11-22 00:35:43,768 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6200, loss[loss=0.05192, simple_loss=0.05389, pruned_loss=0.009973, audio_tagging_loss=0.015, over 15175.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09499, pruned_loss=0.016, audio_tagging_loss=0.009443, over 3031350.80 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:35:48,823 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258700 2023-11-22 00:36:12,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1724760.0, ans=0.125 2023-11-22 00:36:15,755 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.055e+01 8.630e+01 9.332e+01 1.528e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-22 00:36:20,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1724760.0, ans=0.125 2023-11-22 00:36:48,003 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6250, loss[loss=0.06874, simple_loss=0.08882, pruned_loss=0.01405, audio_tagging_loss=0.01028, over 16799.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09471, pruned_loss=0.01595, audio_tagging_loss=0.009558, over 3035763.89 frames. ], batch size: 61, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:36:49,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1724960.0, ans=0.125 2023-11-22 00:36:53,064 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258750 2023-11-22 00:36:59,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1725026.6666666667, ans=0.0 2023-11-22 00:37:03,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.03 vs. limit=15.0 2023-11-22 00:37:19,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1725093.3333333333, ans=0.0 2023-11-22 00:37:33,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1725160.0, ans=0.2 2023-11-22 00:37:34,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1725160.0, ans=0.2 2023-11-22 00:37:38,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1725226.6666666667, ans=0.0 2023-11-22 00:37:40,237 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.57 vs. limit=15.0 2023-11-22 00:37:41,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1725226.6666666667, ans=0.0 2023-11-22 00:37:46,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.48 vs. limit=10.0 2023-11-22 00:37:52,630 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6300, loss[loss=0.08316, simple_loss=0.1116, pruned_loss=0.01934, audio_tagging_loss=0.008033, over 14233.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09507, pruned_loss=0.0159, audio_tagging_loss=0.009643, over 3039806.57 frames. ], batch size: 53, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:37:54,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1725293.3333333333, ans=0.125 2023-11-22 00:37:58,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258800 2023-11-22 00:38:00,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.83 vs. limit=15.0 2023-11-22 00:38:00,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.42 vs. limit=6.0 2023-11-22 00:38:10,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1725360.0, ans=0.1 2023-11-22 00:38:24,170 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:38:25,689 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.059e+01 8.458e+01 8.852e+01 9.578e+01 1.235e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-22 00:38:27,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1725426.6666666667, ans=0.1 2023-11-22 00:38:38,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.29 vs. limit=6.0 2023-11-22 00:38:41,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1725493.3333333333, ans=0.04949747468305833 2023-11-22 00:38:55,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1725560.0, ans=0.0 2023-11-22 00:38:58,134 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6350, loss[loss=0.05876, simple_loss=0.07094, pruned_loss=0.01227, audio_tagging_loss=0.01102, over 15956.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09473, pruned_loss=0.01582, audio_tagging_loss=0.009626, over 3037923.94 frames. ], batch size: 62, lr: 3.11e-03, grad_scale: 32.0 2023-11-22 00:38:59,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.89 vs. limit=15.0 2023-11-22 00:39:01,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1725626.6666666667, ans=0.0 2023-11-22 00:39:03,169 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258850 2023-11-22 00:39:11,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1725693.3333333333, ans=0.0 2023-11-22 00:39:39,473 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.00 vs. limit=15.0 2023-11-22 00:40:01,462 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6400, loss[loss=0.08729, simple_loss=0.124, pruned_loss=0.01884, audio_tagging_loss=0.006437, over 15733.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09532, pruned_loss=0.01569, audio_tagging_loss=0.009685, over 3032626.92 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 00:40:06,994 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258900 2023-11-22 00:40:14,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1726026.6666666667, ans=10.0 2023-11-22 00:40:18,473 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.47 vs. limit=10.0 2023-11-22 00:40:24,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1726026.6666666667, ans=0.0 2023-11-22 00:40:34,293 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.092e+01 8.951e+01 9.607e+01 2.279e+02, threshold=1.790e+02, percent-clipped=1.0 2023-11-22 00:40:40,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1726160.0, ans=0.125 2023-11-22 00:40:47,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1726160.0, ans=0.0 2023-11-22 00:40:59,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1726226.6666666667, ans=0.125 2023-11-22 00:41:05,262 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6450, loss[loss=0.08302, simple_loss=0.1079, pruned_loss=0.01822, audio_tagging_loss=0.01084, over 16116.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09503, pruned_loss=0.01564, audio_tagging_loss=0.009893, over 3037573.93 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 00:41:11,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 258950 2023-11-22 00:41:21,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.26 vs. limit=15.0 2023-11-22 00:41:25,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1726360.0, ans=0.2 2023-11-22 00:41:25,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1726360.0, ans=0.0 2023-11-22 00:41:40,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1726426.6666666667, ans=0.0 2023-11-22 00:42:07,293 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.60 vs. limit=15.0 2023-11-22 00:42:10,282 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6500, loss[loss=0.0835, simple_loss=0.1156, pruned_loss=0.01724, audio_tagging_loss=0.008449, over 15419.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09405, pruned_loss=0.01542, audio_tagging_loss=0.009826, over 3042795.26 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 00:42:14,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1726626.6666666667, ans=0.025 2023-11-22 00:42:15,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259000 2023-11-22 00:42:21,916 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.41 vs. limit=15.0 2023-11-22 00:42:23,251 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.17 vs. limit=15.0 2023-11-22 00:42:37,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1726760.0, ans=0.0 2023-11-22 00:42:42,953 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.155e+01 8.144e+01 8.732e+01 9.307e+01 1.366e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-22 00:42:47,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1726760.0, ans=0.125 2023-11-22 00:43:00,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1726826.6666666667, ans=0.125 2023-11-22 00:43:09,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.35 vs. limit=22.5 2023-11-22 00:43:11,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1726893.3333333333, ans=0.2 2023-11-22 00:43:15,083 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6550, loss[loss=0.07258, simple_loss=0.09736, pruned_loss=0.01618, audio_tagging_loss=0.007718, over 14799.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09373, pruned_loss=0.01549, audio_tagging_loss=0.009619, over 3036395.26 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:43:15,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1726960.0, ans=0.125 2023-11-22 00:43:20,268 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259050 2023-11-22 00:43:31,075 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.18 vs. limit=15.0 2023-11-22 00:43:54,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1727160.0, ans=0.0 2023-11-22 00:44:07,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1727226.6666666667, ans=0.125 2023-11-22 00:44:08,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.26 vs. limit=15.0 2023-11-22 00:44:17,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1727293.3333333333, ans=0.0 2023-11-22 00:44:18,547 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6600, loss[loss=0.07888, simple_loss=0.1114, pruned_loss=0.01442, audio_tagging_loss=0.00878, over 15863.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09427, pruned_loss=0.01566, audio_tagging_loss=0.009433, over 3044340.91 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:44:24,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259100 2023-11-22 00:44:26,130 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:44:27,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1727293.3333333333, ans=0.0 2023-11-22 00:44:49,714 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.99 vs. limit=15.0 2023-11-22 00:44:52,439 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.221e+01 8.740e+01 9.559e+01 1.333e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 00:44:54,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.70 vs. limit=22.5 2023-11-22 00:44:58,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1727493.3333333333, ans=0.125 2023-11-22 00:45:00,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1727493.3333333333, ans=0.125 2023-11-22 00:45:06,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1727493.3333333333, ans=0.125 2023-11-22 00:45:12,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.00 vs. limit=15.0 2023-11-22 00:45:13,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1727560.0, ans=0.125 2023-11-22 00:45:23,535 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6650, loss[loss=0.06483, simple_loss=0.08075, pruned_loss=0.01416, audio_tagging_loss=0.01029, over 14962.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09422, pruned_loss=0.01563, audio_tagging_loss=0.009431, over 3042251.39 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:45:29,254 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259150 2023-11-22 00:45:32,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.82 vs. limit=15.0 2023-11-22 00:45:40,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1727693.3333333333, ans=0.0 2023-11-22 00:45:49,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=15.0 2023-11-22 00:45:49,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.14 vs. limit=15.0 2023-11-22 00:45:51,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=15.0 2023-11-22 00:45:52,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=1727760.0, ans=0.1 2023-11-22 00:45:59,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1727760.0, ans=0.07 2023-11-22 00:46:08,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1727826.6666666667, ans=0.125 2023-11-22 00:46:16,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1727893.3333333333, ans=0.125 2023-11-22 00:46:27,271 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6700, loss[loss=0.07349, simple_loss=0.09035, pruned_loss=0.01938, audio_tagging_loss=0.008934, over 14745.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09374, pruned_loss=0.01558, audio_tagging_loss=0.009447, over 3044803.72 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 8.0 2023-11-22 00:46:32,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259200 2023-11-22 00:47:01,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=1728093.3333333333, ans=0.05 2023-11-22 00:47:01,888 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 7.937e+01 8.531e+01 9.262e+01 1.112e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-22 00:47:03,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1728093.3333333333, ans=0.125 2023-11-22 00:47:06,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1728160.0, ans=0.125 2023-11-22 00:47:11,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1728160.0, ans=0.125 2023-11-22 00:47:17,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1728226.6666666667, ans=0.0 2023-11-22 00:47:20,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1728226.6666666667, ans=0.125 2023-11-22 00:47:30,804 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6750, loss[loss=0.0757, simple_loss=0.09659, pruned_loss=0.01801, audio_tagging_loss=0.009399, over 14301.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09368, pruned_loss=0.0156, audio_tagging_loss=0.009403, over 3042025.79 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 8.0 2023-11-22 00:47:35,715 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259250 2023-11-22 00:47:41,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1728293.3333333333, ans=0.1 2023-11-22 00:47:47,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1728360.0, ans=0.1 2023-11-22 00:47:49,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1728360.0, ans=0.2 2023-11-22 00:48:02,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1728426.6666666667, ans=0.0 2023-11-22 00:48:05,615 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=15.0 2023-11-22 00:48:08,189 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.54 vs. limit=22.5 2023-11-22 00:48:31,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.93 vs. limit=15.0 2023-11-22 00:48:33,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1728560.0, ans=0.125 2023-11-22 00:48:33,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1728560.0, ans=0.2 2023-11-22 00:48:36,254 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6800, loss[loss=0.04391, simple_loss=0.0561, pruned_loss=0.004784, audio_tagging_loss=0.01108, over 14900.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09325, pruned_loss=0.01553, audio_tagging_loss=0.009377, over 3039302.86 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:48:41,235 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259300 2023-11-22 00:48:54,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1728693.3333333333, ans=0.125 2023-11-22 00:48:58,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1728693.3333333333, ans=0.2 2023-11-22 00:49:00,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1728760.0, ans=0.5 2023-11-22 00:49:09,691 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.092e+01 8.012e+01 8.646e+01 9.288e+01 1.206e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-22 00:49:13,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1728826.6666666667, ans=0.125 2023-11-22 00:49:21,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1728826.6666666667, ans=0.125 2023-11-22 00:49:38,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1728893.3333333333, ans=0.2 2023-11-22 00:49:40,209 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6850, loss[loss=0.05835, simple_loss=0.07406, pruned_loss=0.01157, audio_tagging_loss=0.009749, over 16401.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09382, pruned_loss=0.01551, audio_tagging_loss=0.00931, over 3037021.70 frames. ], batch size: 63, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:49:42,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1728960.0, ans=0.0 2023-11-22 00:49:45,212 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259350 2023-11-22 00:49:49,106 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 00:50:17,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.94 vs. limit=15.0 2023-11-22 00:50:20,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1729160.0, ans=0.125 2023-11-22 00:50:20,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1729160.0, ans=0.125 2023-11-22 00:50:24,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1729160.0, ans=0.0 2023-11-22 00:50:25,052 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.81 vs. limit=15.0 2023-11-22 00:50:42,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1729293.3333333333, ans=0.1 2023-11-22 00:50:43,754 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6900, loss[loss=0.06225, simple_loss=0.08237, pruned_loss=0.01209, audio_tagging_loss=0.008972, over 14858.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09493, pruned_loss=0.01558, audio_tagging_loss=0.009234, over 3037203.90 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:50:48,766 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259400 2023-11-22 00:51:10,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1729426.6666666667, ans=0.125 2023-11-22 00:51:15,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1729426.6666666667, ans=0.0 2023-11-22 00:51:19,569 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.107e+01 8.289e+01 8.755e+01 9.420e+01 1.385e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-22 00:51:19,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1729426.6666666667, ans=0.1 2023-11-22 00:51:21,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1729426.6666666667, ans=0.0 2023-11-22 00:51:22,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1729493.3333333333, ans=0.0 2023-11-22 00:51:35,570 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 00:51:40,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1729560.0, ans=0.125 2023-11-22 00:51:47,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1729560.0, ans=0.2 2023-11-22 00:51:49,209 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 6950, loss[loss=0.06634, simple_loss=0.08869, pruned_loss=0.01194, audio_tagging_loss=0.01005, over 14485.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09515, pruned_loss=0.01569, audio_tagging_loss=0.009288, over 3038314.05 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:51:54,342 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259450 2023-11-22 00:52:33,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1729826.6666666667, ans=0.125 2023-11-22 00:52:34,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1729826.6666666667, ans=0.2 2023-11-22 00:52:43,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1729893.3333333333, ans=0.0 2023-11-22 00:52:53,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1729960.0, ans=0.125 2023-11-22 00:52:54,410 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7000, loss[loss=0.07979, simple_loss=0.1094, pruned_loss=0.01683, audio_tagging_loss=0.008269, over 15804.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09518, pruned_loss=0.01572, audio_tagging_loss=0.009328, over 3032095.86 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:52:55,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1729960.0, ans=0.2 2023-11-22 00:52:58,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1729960.0, ans=0.1 2023-11-22 00:52:59,409 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259500 2023-11-22 00:53:04,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1729960.0, ans=0.125 2023-11-22 00:53:11,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.59 vs. limit=10.0 2023-11-22 00:53:11,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1730026.6666666667, ans=0.125 2023-11-22 00:53:14,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1730026.6666666667, ans=0.125 2023-11-22 00:53:28,627 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 7.948e+01 8.560e+01 9.274e+01 1.120e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-22 00:53:51,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1730226.6666666667, ans=0.2 2023-11-22 00:53:58,425 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7050, loss[loss=0.06526, simple_loss=0.07908, pruned_loss=0.01356, audio_tagging_loss=0.01216, over 13777.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09456, pruned_loss=0.01557, audio_tagging_loss=0.009365, over 3036234.60 frames. ], batch size: 53, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:54:02,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1730293.3333333333, ans=0.0 2023-11-22 00:54:03,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259550 2023-11-22 00:54:17,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1730360.0, ans=0.0 2023-11-22 00:54:32,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1730426.6666666667, ans=0.09899494936611666 2023-11-22 00:54:37,844 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.19 vs. limit=15.0 2023-11-22 00:54:48,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1730560.0, ans=0.125 2023-11-22 00:55:02,339 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7100, loss[loss=0.06717, simple_loss=0.08977, pruned_loss=0.01473, audio_tagging_loss=0.007557, over 14240.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09508, pruned_loss=0.01578, audio_tagging_loss=0.009357, over 3040139.88 frames. ], batch size: 53, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:55:05,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1730626.6666666667, ans=0.0 2023-11-22 00:55:07,508 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259600 2023-11-22 00:55:13,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.63 vs. limit=15.0 2023-11-22 00:55:14,448 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.59 vs. limit=22.5 2023-11-22 00:55:19,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1730693.3333333333, ans=0.0 2023-11-22 00:55:21,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1730693.3333333333, ans=0.1 2023-11-22 00:55:33,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1730760.0, ans=0.125 2023-11-22 00:55:37,158 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.168e+01 8.717e+01 9.423e+01 1.311e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-22 00:55:46,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1730826.6666666667, ans=0.07 2023-11-22 00:55:59,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1730893.3333333333, ans=0.125 2023-11-22 00:56:07,817 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7150, loss[loss=0.08887, simple_loss=0.117, pruned_loss=0.02334, audio_tagging_loss=0.007045, over 14744.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09554, pruned_loss=0.01579, audio_tagging_loss=0.009351, over 3044146.01 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 00:56:09,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1730960.0, ans=0.0 2023-11-22 00:56:12,780 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259650 2023-11-22 00:56:13,335 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0 2023-11-22 00:56:30,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1731026.6666666667, ans=0.025 2023-11-22 00:56:45,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1731160.0, ans=0.125 2023-11-22 00:56:46,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1731160.0, ans=0.0 2023-11-22 00:57:00,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1731226.6666666667, ans=0.0 2023-11-22 00:57:07,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1731226.6666666667, ans=0.125 2023-11-22 00:57:11,645 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7200, loss[loss=0.06285, simple_loss=0.0781, pruned_loss=0.01242, audio_tagging_loss=0.01138, over 14220.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09604, pruned_loss=0.01589, audio_tagging_loss=0.009452, over 3043132.13 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 00:57:16,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259700 2023-11-22 00:57:16,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1731293.3333333333, ans=0.125 2023-11-22 00:57:32,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1731360.0, ans=0.125 2023-11-22 00:57:46,568 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 8.081e+01 8.874e+01 9.710e+01 1.298e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 00:58:02,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1731560.0, ans=0.125 2023-11-22 00:58:14,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1731626.6666666667, ans=0.2 2023-11-22 00:58:15,117 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7250, loss[loss=0.07071, simple_loss=0.1043, pruned_loss=0.01242, audio_tagging_loss=0.006129, over 15490.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09611, pruned_loss=0.01572, audio_tagging_loss=0.009483, over 3047488.01 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 00:58:21,213 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259750 2023-11-22 00:58:33,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=1731693.3333333333, ans=10.0 2023-11-22 00:58:35,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1731693.3333333333, ans=0.125 2023-11-22 00:58:54,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1731826.6666666667, ans=0.1 2023-11-22 00:59:09,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1731893.3333333333, ans=0.1 2023-11-22 00:59:15,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1731893.3333333333, ans=0.125 2023-11-22 00:59:19,756 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7300, loss[loss=0.06951, simple_loss=0.09301, pruned_loss=0.01451, audio_tagging_loss=0.008492, over 15002.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09607, pruned_loss=0.0157, audio_tagging_loss=0.009513, over 3042391.07 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 00:59:22,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1731960.0, ans=0.0 2023-11-22 00:59:24,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259800 2023-11-22 00:59:53,080 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.552e+01 8.104e+01 8.588e+01 9.277e+01 1.159e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-22 00:59:59,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1732160.0, ans=0.2 2023-11-22 01:00:01,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1732160.0, ans=0.125 2023-11-22 01:00:13,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1732226.6666666667, ans=0.125 2023-11-22 01:00:23,336 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7350, loss[loss=0.08321, simple_loss=0.116, pruned_loss=0.01941, audio_tagging_loss=0.005818, over 15705.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09626, pruned_loss=0.01581, audio_tagging_loss=0.009322, over 3049653.30 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 01:00:28,281 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259850 2023-11-22 01:00:38,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1732360.0, ans=0.125 2023-11-22 01:00:49,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1732426.6666666667, ans=0.04949747468305833 2023-11-22 01:01:06,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1732493.3333333333, ans=0.125 2023-11-22 01:01:12,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1732493.3333333333, ans=0.125 2023-11-22 01:01:17,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1732560.0, ans=0.1 2023-11-22 01:01:20,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.15 vs. limit=22.5 2023-11-22 01:01:26,434 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7400, loss[loss=0.09019, simple_loss=0.1217, pruned_loss=0.02055, audio_tagging_loss=0.008815, over 15406.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09537, pruned_loss=0.01564, audio_tagging_loss=0.009384, over 3043891.57 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 01:01:26,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1732626.6666666667, ans=0.125 2023-11-22 01:01:28,375 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.89 vs. limit=15.0 2023-11-22 01:01:32,012 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259900 2023-11-22 01:01:34,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.81 vs. limit=12.0 2023-11-22 01:02:02,678 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.170e+01 8.751e+01 9.458e+01 1.086e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 01:02:04,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1732826.6666666667, ans=0.0 2023-11-22 01:02:06,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1732826.6666666667, ans=0.0 2023-11-22 01:02:17,032 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.23 vs. limit=15.0 2023-11-22 01:02:30,999 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7450, loss[loss=0.06958, simple_loss=0.095, pruned_loss=0.01377, audio_tagging_loss=0.008314, over 14843.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09572, pruned_loss=0.01582, audio_tagging_loss=0.009257, over 3045193.67 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 01:02:31,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1732960.0, ans=0.1 2023-11-22 01:02:36,543 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 259950 2023-11-22 01:03:35,344 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7500, loss[loss=0.07502, simple_loss=0.1109, pruned_loss=0.01312, audio_tagging_loss=0.006438, over 15130.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09516, pruned_loss=0.01569, audio_tagging_loss=0.009263, over 3043436.22 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 01:03:40,975 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260000 2023-11-22 01:04:14,757 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.000e+01 8.690e+01 9.325e+01 1.229e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-22 01:04:27,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1733493.3333333333, ans=0.2 2023-11-22 01:04:42,074 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7550, loss[loss=0.07737, simple_loss=0.09923, pruned_loss=0.01776, audio_tagging_loss=0.009999, over 14872.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09476, pruned_loss=0.01572, audio_tagging_loss=0.009255, over 3048689.16 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 01:04:42,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1733626.6666666667, ans=0.125 2023-11-22 01:04:45,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1733626.6666666667, ans=0.125 2023-11-22 01:04:47,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260050 2023-11-22 01:05:18,004 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.82 vs. limit=22.5 2023-11-22 01:05:45,974 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7600, loss[loss=0.07352, simple_loss=0.09427, pruned_loss=0.01827, audio_tagging_loss=0.008114, over 16479.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09499, pruned_loss=0.01581, audio_tagging_loss=0.009301, over 3052825.22 frames. ], batch size: 62, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:05:51,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260100 2023-11-22 01:05:59,553 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.60 vs. limit=22.5 2023-11-22 01:06:07,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1734026.6666666667, ans=0.025 2023-11-22 01:06:14,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=1734093.3333333333, ans=15.0 2023-11-22 01:06:18,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1734093.3333333333, ans=0.5 2023-11-22 01:06:18,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1734093.3333333333, ans=15.0 2023-11-22 01:06:21,720 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.674e+01 8.289e+01 8.897e+01 9.780e+01 1.167e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-22 01:06:45,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1734226.6666666667, ans=0.125 2023-11-22 01:06:49,639 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7650, loss[loss=0.05482, simple_loss=0.06292, pruned_loss=0.01109, audio_tagging_loss=0.01227, over 14046.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09408, pruned_loss=0.01563, audio_tagging_loss=0.009423, over 3044770.66 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:06:53,906 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.87 vs. limit=22.5 2023-11-22 01:06:54,518 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260150 2023-11-22 01:06:58,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1734293.3333333333, ans=0.125 2023-11-22 01:07:06,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1734360.0, ans=0.0 2023-11-22 01:07:28,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1734493.3333333333, ans=0.2 2023-11-22 01:07:32,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1734493.3333333333, ans=0.1 2023-11-22 01:07:35,671 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.27 vs. limit=6.0 2023-11-22 01:07:51,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1734626.6666666667, ans=0.125 2023-11-22 01:07:53,002 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7700, loss[loss=0.06009, simple_loss=0.07688, pruned_loss=0.01209, audio_tagging_loss=0.009562, over 14139.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09376, pruned_loss=0.01558, audio_tagging_loss=0.009467, over 3038956.04 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:07:57,951 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260200 2023-11-22 01:08:04,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1734626.6666666667, ans=0.125 2023-11-22 01:08:17,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1734693.3333333333, ans=0.125 2023-11-22 01:08:25,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1734760.0, ans=0.5 2023-11-22 01:08:29,045 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 7.976e+01 8.890e+01 9.341e+01 1.175e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 01:08:30,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1734826.6666666667, ans=0.125 2023-11-22 01:08:36,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1734826.6666666667, ans=0.0 2023-11-22 01:08:49,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1734893.3333333333, ans=0.125 2023-11-22 01:08:49,960 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.21 vs. limit=15.0 2023-11-22 01:08:57,206 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.22 vs. limit=15.0 2023-11-22 01:08:57,787 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7750, loss[loss=0.09661, simple_loss=0.1266, pruned_loss=0.02545, audio_tagging_loss=0.007876, over 15588.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.0941, pruned_loss=0.01577, audio_tagging_loss=0.009519, over 3038536.81 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:09:02,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260250 2023-11-22 01:09:08,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1734960.0, ans=0.1 2023-11-22 01:09:22,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1735093.3333333333, ans=0.125 2023-11-22 01:09:39,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1735160.0, ans=0.5 2023-11-22 01:09:50,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.55 vs. limit=12.0 2023-11-22 01:10:01,260 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7800, loss[loss=0.06755, simple_loss=0.0867, pruned_loss=0.01559, audio_tagging_loss=0.008613, over 14501.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09529, pruned_loss=0.01604, audio_tagging_loss=0.009486, over 3033566.68 frames. ], batch size: 54, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:10:03,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1735293.3333333333, ans=0.1 2023-11-22 01:10:06,224 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260300 2023-11-22 01:10:08,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1735293.3333333333, ans=0.0 2023-11-22 01:10:33,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1735426.6666666667, ans=0.0 2023-11-22 01:10:35,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.41 vs. limit=22.5 2023-11-22 01:10:37,017 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 8.146e+01 9.007e+01 9.749e+01 1.228e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-22 01:10:37,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1735426.6666666667, ans=0.125 2023-11-22 01:10:45,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1735493.3333333333, ans=0.125 2023-11-22 01:10:51,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1735560.0, ans=0.1 2023-11-22 01:11:04,753 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7850, loss[loss=0.06212, simple_loss=0.08624, pruned_loss=0.009495, audio_tagging_loss=0.009506, over 15848.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09489, pruned_loss=0.01594, audio_tagging_loss=0.009509, over 3035797.24 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:11:09,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260350 2023-11-22 01:11:15,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1735626.6666666667, ans=0.0 2023-11-22 01:11:21,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1735693.3333333333, ans=0.125 2023-11-22 01:12:00,968 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 01:12:09,845 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7900, loss[loss=0.07984, simple_loss=0.1075, pruned_loss=0.0162, audio_tagging_loss=0.009895, over 15609.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09499, pruned_loss=0.01594, audio_tagging_loss=0.009639, over 3035240.88 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 01:12:14,946 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260400 2023-11-22 01:12:26,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1736026.6666666667, ans=0.0 2023-11-22 01:12:33,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1736026.6666666667, ans=0.2 2023-11-22 01:12:42,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.53 vs. limit=22.5 2023-11-22 01:12:46,390 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.401e+01 8.796e+01 9.438e+01 1.194e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 01:13:01,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1736226.6666666667, ans=0.0 2023-11-22 01:13:03,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1736226.6666666667, ans=0.125 2023-11-22 01:13:13,831 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 7950, loss[loss=0.08379, simple_loss=0.1151, pruned_loss=0.0176, audio_tagging_loss=0.008627, over 15746.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09504, pruned_loss=0.01581, audio_tagging_loss=0.009686, over 3037104.89 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0 2023-11-22 01:13:18,760 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260450 2023-11-22 01:13:27,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1736360.0, ans=0.0 2023-11-22 01:13:29,892 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 01:13:39,090 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.20 vs. limit=6.0 2023-11-22 01:13:39,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1736426.6666666667, ans=0.125 2023-11-22 01:13:45,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.39 vs. limit=12.0 2023-11-22 01:13:48,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1736426.6666666667, ans=0.2 2023-11-22 01:13:59,987 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 01:14:00,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1736493.3333333333, ans=0.125 2023-11-22 01:14:16,703 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8000, loss[loss=0.06243, simple_loss=0.08153, pruned_loss=0.01025, audio_tagging_loss=0.01141, over 15077.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.09493, pruned_loss=0.01581, audio_tagging_loss=0.00986, over 3035218.36 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:14:21,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260500 2023-11-22 01:14:38,827 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.26 vs. limit=10.0 2023-11-22 01:14:40,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1736693.3333333333, ans=0.1 2023-11-22 01:14:46,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1736760.0, ans=0.0 2023-11-22 01:14:46,687 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.02 vs. limit=15.0 2023-11-22 01:14:54,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 7.877e+01 8.568e+01 9.236e+01 1.291e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-22 01:15:01,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1736826.6666666667, ans=0.1 2023-11-22 01:15:03,644 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.02 vs. limit=22.5 2023-11-22 01:15:20,834 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8050, loss[loss=0.08542, simple_loss=0.1167, pruned_loss=0.01685, audio_tagging_loss=0.0102, over 16229.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09419, pruned_loss=0.01559, audio_tagging_loss=0.009851, over 3027634.48 frames. ], batch size: 59, lr: 3.10e-03, grad_scale: 32.0 2023-11-22 01:15:26,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260550 2023-11-22 01:15:41,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1737026.6666666667, ans=0.125 2023-11-22 01:16:26,003 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8100, loss[loss=0.05634, simple_loss=0.07224, pruned_loss=0.0133, audio_tagging_loss=0.006925, over 14066.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09435, pruned_loss=0.01559, audio_tagging_loss=0.00966, over 3026292.40 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:16:30,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260600 2023-11-22 01:16:48,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1737360.0, ans=0.125 2023-11-22 01:16:50,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1737426.6666666667, ans=0.2 2023-11-22 01:17:04,360 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.436e+01 7.880e+01 8.583e+01 9.266e+01 1.211e+02, threshold=1.717e+02, percent-clipped=0.0 2023-11-22 01:17:21,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1737560.0, ans=0.1 2023-11-22 01:17:24,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1737560.0, ans=0.125 2023-11-22 01:17:30,012 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8150, loss[loss=0.08863, simple_loss=0.1205, pruned_loss=0.02086, audio_tagging_loss=0.007495, over 15800.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09551, pruned_loss=0.01578, audio_tagging_loss=0.009407, over 3028719.53 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:17:34,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260650 2023-11-22 01:17:39,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1737626.6666666667, ans=15.0 2023-11-22 01:17:58,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1737760.0, ans=0.125 2023-11-22 01:18:10,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1737826.6666666667, ans=0.2 2023-11-22 01:18:10,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1737826.6666666667, ans=0.5 2023-11-22 01:18:10,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1737826.6666666667, ans=0.2 2023-11-22 01:18:16,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1737826.6666666667, ans=0.1 2023-11-22 01:18:26,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1737893.3333333333, ans=0.125 2023-11-22 01:18:33,275 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8200, loss[loss=0.08124, simple_loss=0.1071, pruned_loss=0.01983, audio_tagging_loss=0.007832, over 15069.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09639, pruned_loss=0.01596, audio_tagging_loss=0.009258, over 3044719.82 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:18:36,361 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 01:18:38,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1737960.0, ans=0.0 2023-11-22 01:18:39,507 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260700 2023-11-22 01:18:40,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1737960.0, ans=0.125 2023-11-22 01:18:42,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.58 vs. limit=15.0 2023-11-22 01:18:47,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1738026.6666666667, ans=0.0 2023-11-22 01:18:47,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.87 vs. limit=22.5 2023-11-22 01:18:57,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1738026.6666666667, ans=0.07 2023-11-22 01:19:00,708 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.76 vs. limit=12.0 2023-11-22 01:19:12,443 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.958e+01 8.128e+01 8.787e+01 9.499e+01 1.177e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 01:19:21,669 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0 2023-11-22 01:19:24,051 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.71 vs. limit=15.0 2023-11-22 01:19:38,530 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8250, loss[loss=0.07326, simple_loss=0.1067, pruned_loss=0.01202, audio_tagging_loss=0.00789, over 15720.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09567, pruned_loss=0.01579, audio_tagging_loss=0.009327, over 3046623.72 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:19:42,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1738293.3333333333, ans=0.125 2023-11-22 01:19:43,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260750 2023-11-22 01:19:47,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1738293.3333333333, ans=0.0 2023-11-22 01:19:47,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1738293.3333333333, ans=0.07 2023-11-22 01:19:55,886 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 01:20:01,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1738426.6666666667, ans=0.2 2023-11-22 01:20:35,217 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.22 vs. limit=15.0 2023-11-22 01:20:37,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1738560.0, ans=0.2 2023-11-22 01:20:41,962 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8300, loss[loss=0.06925, simple_loss=0.08499, pruned_loss=0.01577, audio_tagging_loss=0.01098, over 14876.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09656, pruned_loss=0.01599, audio_tagging_loss=0.009341, over 3049733.06 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:20:46,942 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260800 2023-11-22 01:21:02,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1738693.3333333333, ans=0.1 2023-11-22 01:21:05,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1738693.3333333333, ans=0.0 2023-11-22 01:21:10,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1738760.0, ans=0.125 2023-11-22 01:21:10,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1738760.0, ans=0.125 2023-11-22 01:21:21,281 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.003e+01 8.893e+01 9.644e+01 1.813e+02, threshold=1.779e+02, percent-clipped=1.0 2023-11-22 01:21:32,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1738893.3333333333, ans=0.1 2023-11-22 01:21:34,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1738893.3333333333, ans=0.2 2023-11-22 01:21:39,202 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.59 vs. limit=6.0 2023-11-22 01:21:41,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.43 vs. limit=12.0 2023-11-22 01:21:46,178 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8350, loss[loss=0.0509, simple_loss=0.06601, pruned_loss=0.007172, audio_tagging_loss=0.01072, over 14971.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09593, pruned_loss=0.01582, audio_tagging_loss=0.009285, over 3052207.64 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:21:51,709 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260850 2023-11-22 01:22:10,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1739026.6666666667, ans=0.0 2023-11-22 01:22:33,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1739160.0, ans=0.1 2023-11-22 01:22:50,539 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8400, loss[loss=0.08381, simple_loss=0.1127, pruned_loss=0.01702, audio_tagging_loss=0.01046, over 15442.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09605, pruned_loss=0.0158, audio_tagging_loss=0.009258, over 3051020.90 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:22:56,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260900 2023-11-22 01:23:19,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1739426.6666666667, ans=0.2 2023-11-22 01:23:28,561 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 7.994e+01 8.596e+01 9.193e+01 1.208e+02, threshold=1.719e+02, percent-clipped=0.0 2023-11-22 01:23:41,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1739560.0, ans=0.125 2023-11-22 01:23:51,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1739560.0, ans=0.0 2023-11-22 01:23:54,251 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.05 vs. limit=22.5 2023-11-22 01:23:54,877 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8450, loss[loss=0.08791, simple_loss=0.1111, pruned_loss=0.02392, audio_tagging_loss=0.008457, over 15239.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.09571, pruned_loss=0.01588, audio_tagging_loss=0.009387, over 3052482.98 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:23:59,851 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 260950 2023-11-22 01:24:15,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1739693.3333333333, ans=0.1 2023-11-22 01:24:24,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1739760.0, ans=0.125 2023-11-22 01:24:29,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.84 vs. limit=15.0 2023-11-22 01:24:35,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1739826.6666666667, ans=0.0 2023-11-22 01:24:37,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1739826.6666666667, ans=0.1 2023-11-22 01:24:58,320 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8500, loss[loss=0.07176, simple_loss=0.09518, pruned_loss=0.0141, audio_tagging_loss=0.01007, over 15321.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09607, pruned_loss=0.01584, audio_tagging_loss=0.009337, over 3052614.49 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:25:00,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1739960.0, ans=0.125 2023-11-22 01:25:03,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261000 2023-11-22 01:25:36,965 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 8.117e+01 8.646e+01 9.659e+01 1.248e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-22 01:25:53,692 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.94 vs. limit=12.0 2023-11-22 01:25:57,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.05 vs. limit=22.5 2023-11-22 01:26:02,755 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8550, loss[loss=0.09422, simple_loss=0.1226, pruned_loss=0.02334, audio_tagging_loss=0.009593, over 14891.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09642, pruned_loss=0.01598, audio_tagging_loss=0.009366, over 3063013.10 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:26:08,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261050 2023-11-22 01:26:13,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1740293.3333333333, ans=0.125 2023-11-22 01:26:29,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0 2023-11-22 01:26:52,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1740560.0, ans=0.07 2023-11-22 01:26:56,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1740560.0, ans=0.0 2023-11-22 01:27:06,901 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8600, loss[loss=0.06367, simple_loss=0.08351, pruned_loss=0.0134, audio_tagging_loss=0.008516, over 15468.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09624, pruned_loss=0.01604, audio_tagging_loss=0.009389, over 3059142.49 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:27:10,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1740626.6666666667, ans=0.125 2023-11-22 01:27:12,479 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261100 2023-11-22 01:27:35,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1740760.0, ans=0.125 2023-11-22 01:27:45,978 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.115e+01 8.738e+01 9.336e+01 1.223e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 01:27:46,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.whiten.whitening_limit, batch_count=1740826.6666666667, ans=12.0 2023-11-22 01:27:57,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1740893.3333333333, ans=0.1 2023-11-22 01:27:58,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0 2023-11-22 01:28:04,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.98 vs. limit=15.0 2023-11-22 01:28:06,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1740893.3333333333, ans=0.125 2023-11-22 01:28:09,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1740960.0, ans=0.0 2023-11-22 01:28:09,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1740960.0, ans=0.2 2023-11-22 01:28:10,849 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8650, loss[loss=0.07415, simple_loss=0.09901, pruned_loss=0.01409, audio_tagging_loss=0.01055, over 16781.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09533, pruned_loss=0.01576, audio_tagging_loss=0.009504, over 3054436.75 frames. ], batch size: 61, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:28:12,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1740960.0, ans=0.125 2023-11-22 01:28:13,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1740960.0, ans=0.1 2023-11-22 01:28:15,359 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.71 vs. limit=15.0 2023-11-22 01:28:15,851 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261150 2023-11-22 01:29:10,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1741226.6666666667, ans=0.0 2023-11-22 01:29:14,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.72 vs. limit=15.0 2023-11-22 01:29:15,061 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8700, loss[loss=0.09452, simple_loss=0.1293, pruned_loss=0.02176, audio_tagging_loss=0.008129, over 15289.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09564, pruned_loss=0.01594, audio_tagging_loss=0.009551, over 3046601.73 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:29:16,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1741293.3333333333, ans=0.0 2023-11-22 01:29:16,636 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 01:29:20,598 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261200 2023-11-22 01:29:23,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1741293.3333333333, ans=0.125 2023-11-22 01:29:25,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1741293.3333333333, ans=0.1 2023-11-22 01:29:34,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1741360.0, ans=0.0 2023-11-22 01:29:53,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1741493.3333333333, ans=0.125 2023-11-22 01:29:54,207 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.323e+01 9.034e+01 9.944e+01 3.984e+02, threshold=1.807e+02, percent-clipped=2.0 2023-11-22 01:29:56,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1741493.3333333333, ans=0.0 2023-11-22 01:30:06,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1741560.0, ans=0.125 2023-11-22 01:30:16,430 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0 2023-11-22 01:30:19,272 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8750, loss[loss=0.07787, simple_loss=0.09424, pruned_loss=0.01759, audio_tagging_loss=0.01316, over 14483.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09531, pruned_loss=0.01594, audio_tagging_loss=0.009611, over 3046340.39 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:30:24,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261250 2023-11-22 01:30:25,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1741626.6666666667, ans=0.0 2023-11-22 01:30:30,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1741693.3333333333, ans=0.125 2023-11-22 01:30:34,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1741693.3333333333, ans=0.1 2023-11-22 01:30:37,547 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.06 vs. limit=15.0 2023-11-22 01:31:00,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1741826.6666666667, ans=0.1 2023-11-22 01:31:23,447 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8800, loss[loss=0.05566, simple_loss=0.06709, pruned_loss=0.01131, audio_tagging_loss=0.0108, over 14127.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09662, pruned_loss=0.01614, audio_tagging_loss=0.009619, over 3051589.93 frames. ], batch size: 54, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:31:23,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1741960.0, ans=0.0 2023-11-22 01:31:28,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261300 2023-11-22 01:31:35,004 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.31 vs. limit=15.0 2023-11-22 01:32:03,194 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.567e+01 8.285e+01 8.942e+01 9.871e+01 1.431e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-22 01:32:15,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1742226.6666666667, ans=0.125 2023-11-22 01:32:19,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1742226.6666666667, ans=0.125 2023-11-22 01:32:28,149 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8850, loss[loss=0.07423, simple_loss=0.09556, pruned_loss=0.01693, audio_tagging_loss=0.009526, over 14737.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09697, pruned_loss=0.01618, audio_tagging_loss=0.009658, over 3055297.86 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:32:33,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261350 2023-11-22 01:32:42,109 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 01:33:08,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1742493.3333333333, ans=0.07 2023-11-22 01:33:17,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.48 vs. limit=22.5 2023-11-22 01:33:25,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1742560.0, ans=0.0 2023-11-22 01:33:29,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1742560.0, ans=0.2 2023-11-22 01:33:31,966 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8900, loss[loss=0.07344, simple_loss=0.09559, pruned_loss=0.01615, audio_tagging_loss=0.009494, over 14891.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09618, pruned_loss=0.01579, audio_tagging_loss=0.009602, over 3055253.84 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:33:36,921 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261400 2023-11-22 01:33:42,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1742626.6666666667, ans=0.2 2023-11-22 01:33:43,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1742693.3333333333, ans=0.2 2023-11-22 01:33:57,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1742760.0, ans=0.2 2023-11-22 01:34:11,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1742826.6666666667, ans=0.125 2023-11-22 01:34:13,142 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.357e+01 8.809e+01 9.586e+01 1.233e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 01:34:16,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.29 vs. limit=12.0 2023-11-22 01:34:29,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1742893.3333333333, ans=0.125 2023-11-22 01:34:29,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.16 vs. limit=6.0 2023-11-22 01:34:33,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1742893.3333333333, ans=0.1 2023-11-22 01:34:35,048 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 8950, loss[loss=0.0765, simple_loss=0.09984, pruned_loss=0.01911, audio_tagging_loss=0.007461, over 14505.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09644, pruned_loss=0.0159, audio_tagging_loss=0.009432, over 3053990.34 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:34:37,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1742960.0, ans=0.0 2023-11-22 01:34:40,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261450 2023-11-22 01:34:43,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.05 vs. limit=15.0 2023-11-22 01:34:49,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1743026.6666666667, ans=0.0 2023-11-22 01:35:05,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1743093.3333333333, ans=0.125 2023-11-22 01:35:39,544 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9000, loss[loss=0.06901, simple_loss=0.09413, pruned_loss=0.01245, audio_tagging_loss=0.009495, over 14471.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09609, pruned_loss=0.01576, audio_tagging_loss=0.009418, over 3043161.63 frames. ], batch size: 54, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:35:39,544 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 01:36:12,723 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3434, 5.0526, 4.7203, 5.1942], device='cuda:3') 2023-11-22 01:36:20,049 INFO [train_asr.py:1253] (3/4) Epoch 22, validation: loss=0.0605, simple_loss=0.05183, pruned_loss=0.005175, audio_tagging_loss=0.02941, over 4681554.00 frames. 2023-11-22 01:36:20,050 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 01:36:22,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1743293.3333333333, ans=0.1 2023-11-22 01:36:24,863 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261500 2023-11-22 01:36:26,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1743293.3333333333, ans=0.125 2023-11-22 01:36:39,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1743360.0, ans=0.09899494936611666 2023-11-22 01:36:53,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1743426.6666666667, ans=0.09899494936611666 2023-11-22 01:36:54,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1743426.6666666667, ans=0.0 2023-11-22 01:36:54,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1743426.6666666667, ans=0.2 2023-11-22 01:36:55,233 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.01 vs. limit=15.0 2023-11-22 01:37:01,166 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.315e+01 8.919e+01 9.797e+01 1.182e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 01:37:02,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1743493.3333333333, ans=0.2 2023-11-22 01:37:11,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1743560.0, ans=0.0 2023-11-22 01:37:22,990 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9050, loss[loss=0.1043, simple_loss=0.1276, pruned_loss=0.03247, audio_tagging_loss=0.008025, over 15296.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09601, pruned_loss=0.01589, audio_tagging_loss=0.009453, over 3045268.65 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:37:27,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261550 2023-11-22 01:37:31,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1743626.6666666667, ans=0.125 2023-11-22 01:37:38,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.84 vs. limit=15.0 2023-11-22 01:37:51,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1743760.0, ans=0.0 2023-11-22 01:37:54,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1743760.0, ans=0.125 2023-11-22 01:38:07,836 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0 2023-11-22 01:38:19,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1743893.3333333333, ans=0.125 2023-11-22 01:38:24,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1743893.3333333333, ans=0.035 2023-11-22 01:38:25,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.78 vs. limit=15.0 2023-11-22 01:38:27,021 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9100, loss[loss=0.07323, simple_loss=0.08979, pruned_loss=0.01884, audio_tagging_loss=0.009496, over 16585.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09546, pruned_loss=0.01572, audio_tagging_loss=0.009354, over 3053555.86 frames. ], batch size: 66, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:38:31,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261600 2023-11-22 01:38:35,175 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.49 vs. limit=15.0 2023-11-22 01:38:36,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1743960.0, ans=0.1 2023-11-22 01:38:48,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1744026.6666666667, ans=0.125 2023-11-22 01:39:06,384 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.063e+01 8.742e+01 9.387e+01 1.203e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 01:39:07,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1744160.0, ans=0.125 2023-11-22 01:39:30,546 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9150, loss[loss=0.09478, simple_loss=0.1308, pruned_loss=0.01992, audio_tagging_loss=0.009442, over 15520.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.0962, pruned_loss=0.01579, audio_tagging_loss=0.009283, over 3063363.27 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:39:32,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.43 vs. limit=15.0 2023-11-22 01:39:35,523 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261650 2023-11-22 01:39:38,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1744293.3333333333, ans=0.2 2023-11-22 01:39:50,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1744360.0, ans=0.1 2023-11-22 01:39:54,262 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.15 vs. limit=10.0 2023-11-22 01:40:02,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1744426.6666666667, ans=0.1 2023-11-22 01:40:16,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1744493.3333333333, ans=0.125 2023-11-22 01:40:33,660 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9200, loss[loss=0.08632, simple_loss=0.1151, pruned_loss=0.01803, audio_tagging_loss=0.01073, over 14512.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09631, pruned_loss=0.01576, audio_tagging_loss=0.009258, over 3056680.21 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:40:36,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.60 vs. limit=15.0 2023-11-22 01:40:38,712 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261700 2023-11-22 01:40:49,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1744693.3333333333, ans=0.125 2023-11-22 01:40:52,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1744693.3333333333, ans=0.0 2023-11-22 01:41:00,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0 2023-11-22 01:41:14,625 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.066e+01 8.592e+01 9.287e+01 1.258e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-22 01:41:30,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1744893.3333333333, ans=0.125 2023-11-22 01:41:31,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.34 vs. limit=10.0 2023-11-22 01:41:37,059 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9250, loss[loss=0.07184, simple_loss=0.09996, pruned_loss=0.01416, audio_tagging_loss=0.007694, over 16568.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09586, pruned_loss=0.01575, audio_tagging_loss=0.009216, over 3056387.82 frames. ], batch size: 63, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:41:39,615 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.67 vs. limit=15.0 2023-11-22 01:41:43,183 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261750 2023-11-22 01:41:55,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1745026.6666666667, ans=0.125 2023-11-22 01:42:06,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1745093.3333333333, ans=0.125 2023-11-22 01:42:09,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1745093.3333333333, ans=0.125 2023-11-22 01:42:13,399 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 01:42:36,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1745226.6666666667, ans=0.125 2023-11-22 01:42:41,994 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9300, loss[loss=0.06314, simple_loss=0.08306, pruned_loss=0.01214, audio_tagging_loss=0.009481, over 13984.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09511, pruned_loss=0.01538, audio_tagging_loss=0.009303, over 3051513.96 frames. ], batch size: 54, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:42:46,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261800 2023-11-22 01:43:11,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.30 vs. limit=10.0 2023-11-22 01:43:24,445 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.431e+01 7.861e+01 8.461e+01 9.251e+01 1.268e+02, threshold=1.692e+02, percent-clipped=0.0 2023-11-22 01:43:30,488 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.27 vs. limit=22.5 2023-11-22 01:43:37,834 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.21 vs. limit=22.5 2023-11-22 01:43:45,726 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9350, loss[loss=0.06232, simple_loss=0.07708, pruned_loss=0.008942, audio_tagging_loss=0.01484, over 15448.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09511, pruned_loss=0.01555, audio_tagging_loss=0.009335, over 3047148.76 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:43:50,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261850 2023-11-22 01:44:05,865 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.13 vs. limit=10.0 2023-11-22 01:44:23,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1745826.6666666667, ans=0.0 2023-11-22 01:44:43,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1745893.3333333333, ans=0.125 2023-11-22 01:44:46,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.81 vs. limit=12.0 2023-11-22 01:44:47,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.90 vs. limit=15.0 2023-11-22 01:44:49,967 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9400, loss[loss=0.08596, simple_loss=0.1092, pruned_loss=0.02263, audio_tagging_loss=0.008735, over 16059.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09564, pruned_loss=0.01561, audio_tagging_loss=0.009401, over 3045287.61 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:44:55,576 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261900 2023-11-22 01:44:59,650 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.30 vs. limit=15.0 2023-11-22 01:45:18,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1746093.3333333333, ans=0.125 2023-11-22 01:45:32,795 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.919e+01 8.299e+01 8.911e+01 9.594e+01 2.055e+02, threshold=1.782e+02, percent-clipped=1.0 2023-11-22 01:45:35,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1746160.0, ans=0.0 2023-11-22 01:45:52,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1746226.6666666667, ans=0.125 2023-11-22 01:45:52,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1746226.6666666667, ans=0.2 2023-11-22 01:45:56,239 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9450, loss[loss=0.07098, simple_loss=0.09089, pruned_loss=0.01387, audio_tagging_loss=0.01166, over 15846.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09536, pruned_loss=0.01555, audio_tagging_loss=0.009495, over 3049540.78 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:45:56,322 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 01:45:59,585 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.45 vs. limit=6.0 2023-11-22 01:46:01,380 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 261950 2023-11-22 01:46:09,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1746360.0, ans=0.125 2023-11-22 01:46:14,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746360.0, ans=0.1 2023-11-22 01:46:20,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746426.6666666667, ans=0.1 2023-11-22 01:46:24,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.55 vs. limit=15.0 2023-11-22 01:46:42,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1746493.3333333333, ans=0.2 2023-11-22 01:46:54,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1746560.0, ans=0.2 2023-11-22 01:46:54,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1746560.0, ans=0.015 2023-11-22 01:46:55,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746560.0, ans=0.1 2023-11-22 01:46:57,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1746560.0, ans=0.125 2023-11-22 01:46:59,808 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9500, loss[loss=0.07254, simple_loss=0.09033, pruned_loss=0.01514, audio_tagging_loss=0.01223, over 15942.00 frames. ], tot_loss[loss=0.07285, simple_loss=0.09524, pruned_loss=0.01558, audio_tagging_loss=0.00965, over 3050479.31 frames. ], batch size: 62, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:47:04,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262000 2023-11-22 01:47:42,655 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.239e+01 8.709e+01 9.390e+01 1.179e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-22 01:47:48,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1746826.6666666667, ans=0.125 2023-11-22 01:48:03,599 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9550, loss[loss=0.06225, simple_loss=0.07902, pruned_loss=0.01288, audio_tagging_loss=0.009861, over 14746.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09483, pruned_loss=0.01555, audio_tagging_loss=0.009708, over 3041049.51 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0 2023-11-22 01:48:09,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262050 2023-11-22 01:48:10,478 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.68 vs. limit=15.0 2023-11-22 01:48:16,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1747026.6666666667, ans=0.0 2023-11-22 01:48:20,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1747026.6666666667, ans=0.125 2023-11-22 01:48:23,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.65 vs. limit=22.5 2023-11-22 01:48:44,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1747160.0, ans=0.125 2023-11-22 01:49:00,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1747226.6666666667, ans=0.125 2023-11-22 01:49:08,518 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9600, loss[loss=0.08629, simple_loss=0.1176, pruned_loss=0.01942, audio_tagging_loss=0.008079, over 14922.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09565, pruned_loss=0.01564, audio_tagging_loss=0.009629, over 3038095.42 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:49:14,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262100 2023-11-22 01:49:15,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1747293.3333333333, ans=0.125 2023-11-22 01:49:18,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1747293.3333333333, ans=0.1 2023-11-22 01:49:51,172 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 8.033e+01 8.722e+01 9.429e+01 2.117e+02, threshold=1.744e+02, percent-clipped=1.0 2023-11-22 01:49:51,991 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.79 vs. limit=22.5 2023-11-22 01:50:08,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1747560.0, ans=0.125 2023-11-22 01:50:13,373 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9650, loss[loss=0.07664, simple_loss=0.1001, pruned_loss=0.01583, audio_tagging_loss=0.01074, over 14794.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.0958, pruned_loss=0.01569, audio_tagging_loss=0.009613, over 3036349.92 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:50:14,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1747626.6666666667, ans=0.125 2023-11-22 01:50:18,236 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262150 2023-11-22 01:50:23,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1747626.6666666667, ans=0.125 2023-11-22 01:50:28,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1747693.3333333333, ans=0.0 2023-11-22 01:50:45,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1747760.0, ans=0.2 2023-11-22 01:50:47,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1747760.0, ans=0.125 2023-11-22 01:51:16,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.92 vs. limit=22.5 2023-11-22 01:51:16,682 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9700, loss[loss=0.09398, simple_loss=0.1169, pruned_loss=0.02658, audio_tagging_loss=0.008964, over 14951.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09647, pruned_loss=0.01586, audio_tagging_loss=0.009432, over 3037973.25 frames. ], batch size: 54, lr: 3.09e-03, grad_scale: 32.0 2023-11-22 01:51:21,581 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262200 2023-11-22 01:51:58,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1748160.0, ans=0.125 2023-11-22 01:52:00,820 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.573e+01 8.169e+01 8.750e+01 9.625e+01 1.305e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 01:52:17,581 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.90 vs. limit=15.0 2023-11-22 01:52:18,271 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 01:52:21,639 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9750, loss[loss=0.0557, simple_loss=0.0692, pruned_loss=0.01291, audio_tagging_loss=0.008189, over 14459.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09549, pruned_loss=0.01569, audio_tagging_loss=0.009277, over 3039775.81 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 01:52:21,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1748293.3333333333, ans=10.0 2023-11-22 01:52:26,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.31 vs. limit=22.5 2023-11-22 01:52:27,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262250 2023-11-22 01:52:29,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1748293.3333333333, ans=0.125 2023-11-22 01:53:01,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1748493.3333333333, ans=0.0 2023-11-22 01:53:06,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1748493.3333333333, ans=0.125 2023-11-22 01:53:18,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1748560.0, ans=0.0 2023-11-22 01:53:18,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1748560.0, ans=0.125 2023-11-22 01:53:26,052 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9800, loss[loss=0.07084, simple_loss=0.09598, pruned_loss=0.01442, audio_tagging_loss=0.008427, over 14311.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09542, pruned_loss=0.01569, audio_tagging_loss=0.009264, over 3038981.10 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 01:53:31,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262300 2023-11-22 01:53:40,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1748693.3333333333, ans=0.125 2023-11-22 01:53:54,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.64 vs. limit=6.0 2023-11-22 01:54:10,084 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.436e+01 9.164e+01 9.957e+01 1.259e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-22 01:54:24,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1748893.3333333333, ans=0.125 2023-11-22 01:54:26,535 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 01:54:30,298 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9850, loss[loss=0.06306, simple_loss=0.08357, pruned_loss=0.01069, audio_tagging_loss=0.01058, over 15485.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09708, pruned_loss=0.01592, audio_tagging_loss=0.009058, over 3044015.65 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 01:54:34,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1748960.0, ans=0.2 2023-11-22 01:54:35,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262350 2023-11-22 01:54:42,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1749026.6666666667, ans=0.125 2023-11-22 01:54:42,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1749026.6666666667, ans=0.125 2023-11-22 01:54:42,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1749026.6666666667, ans=0.125 2023-11-22 01:54:45,727 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.05 vs. limit=10.0 2023-11-22 01:54:52,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1749026.6666666667, ans=0.125 2023-11-22 01:54:56,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1749093.3333333333, ans=0.09899494936611666 2023-11-22 01:55:15,049 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.90 vs. limit=15.0 2023-11-22 01:55:35,350 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9900, loss[loss=0.05759, simple_loss=0.0751, pruned_loss=0.01055, audio_tagging_loss=0.009491, over 15591.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.096, pruned_loss=0.01569, audio_tagging_loss=0.009116, over 3050545.04 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 01:55:40,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262400 2023-11-22 01:55:45,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1749293.3333333333, ans=0.0 2023-11-22 01:55:52,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1749360.0, ans=0.0 2023-11-22 01:56:02,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1749426.6666666667, ans=0.125 2023-11-22 01:56:15,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.55 vs. limit=12.0 2023-11-22 01:56:19,003 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.153e+01 9.023e+01 9.668e+01 1.710e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-22 01:56:22,857 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.32 vs. limit=15.0 2023-11-22 01:56:28,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1749560.0, ans=0.125 2023-11-22 01:56:35,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1749560.0, ans=0.125 2023-11-22 01:56:38,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1749626.6666666667, ans=0.125 2023-11-22 01:56:39,848 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 9950, loss[loss=0.07399, simple_loss=0.1025, pruned_loss=0.01231, audio_tagging_loss=0.01041, over 14516.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09617, pruned_loss=0.0158, audio_tagging_loss=0.009102, over 3047473.88 frames. ], batch size: 53, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 01:56:44,762 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262450 2023-11-22 01:56:47,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1749626.6666666667, ans=0.0 2023-11-22 01:56:49,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1749626.6666666667, ans=0.125 2023-11-22 01:56:58,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1749693.3333333333, ans=0.5 2023-11-22 01:57:14,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1749760.0, ans=0.0 2023-11-22 01:57:20,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1749826.6666666667, ans=0.125 2023-11-22 01:57:43,622 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10000, loss[loss=0.07504, simple_loss=0.09966, pruned_loss=0.01521, audio_tagging_loss=0.009998, over 15567.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.09547, pruned_loss=0.01556, audio_tagging_loss=0.009175, over 3041997.19 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 32.0 2023-11-22 01:57:47,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1749960.0, ans=0.2 2023-11-22 01:57:48,579 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262500 2023-11-22 01:57:55,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.68 vs. limit=15.0 2023-11-22 01:57:56,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1750026.6666666667, ans=0.0 2023-11-22 01:58:14,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.32 vs. limit=15.0 2023-11-22 01:58:17,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1750093.3333333333, ans=0.125 2023-11-22 01:58:27,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1750160.0, ans=0.0 2023-11-22 01:58:28,343 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.434e+01 8.075e+01 8.690e+01 9.364e+01 1.327e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-22 01:58:36,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1750226.6666666667, ans=0.2 2023-11-22 01:58:47,711 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10050, loss[loss=0.07892, simple_loss=0.1041, pruned_loss=0.01578, audio_tagging_loss=0.01108, over 15422.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09546, pruned_loss=0.01566, audio_tagging_loss=0.00919, over 3042469.43 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 01:58:53,383 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262550 2023-11-22 01:58:56,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1750293.3333333333, ans=0.0 2023-11-22 01:59:16,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1750426.6666666667, ans=0.125 2023-11-22 01:59:43,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1750560.0, ans=0.125 2023-11-22 01:59:49,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1750560.0, ans=0.125 2023-11-22 01:59:52,857 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10100, loss[loss=0.07251, simple_loss=0.09575, pruned_loss=0.01547, audio_tagging_loss=0.009167, over 15254.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09646, pruned_loss=0.01587, audio_tagging_loss=0.009272, over 3047463.75 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 01:59:57,980 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262600 2023-11-22 02:00:15,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1750693.3333333333, ans=0.125 2023-11-22 02:00:38,610 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.193e+01 8.828e+01 9.579e+01 1.146e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 02:00:47,439 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 02:00:52,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1750893.3333333333, ans=0.1 2023-11-22 02:00:56,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1750960.0, ans=15.0 2023-11-22 02:00:57,225 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10150, loss[loss=0.06384, simple_loss=0.08208, pruned_loss=0.01282, audio_tagging_loss=0.009976, over 15242.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09522, pruned_loss=0.01577, audio_tagging_loss=0.00944, over 3053449.92 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:01:01,985 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262650 2023-11-22 02:01:15,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1751026.6666666667, ans=0.125 2023-11-22 02:01:30,323 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 02:02:01,548 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10200, loss[loss=0.07881, simple_loss=0.1022, pruned_loss=0.01738, audio_tagging_loss=0.01031, over 16096.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09462, pruned_loss=0.01569, audio_tagging_loss=0.009586, over 3053089.50 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:02:03,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1751293.3333333333, ans=0.0 2023-11-22 02:02:07,150 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262700 2023-11-22 02:02:23,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1751360.0, ans=0.125 2023-11-22 02:02:26,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1751426.6666666667, ans=0.125 2023-11-22 02:02:27,342 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 02:02:46,388 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 7.988e+01 8.675e+01 9.636e+01 1.224e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-22 02:03:06,722 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10250, loss[loss=0.06126, simple_loss=0.08649, pruned_loss=0.008356, audio_tagging_loss=0.009655, over 16282.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09545, pruned_loss=0.01574, audio_tagging_loss=0.009616, over 3064706.65 frames. ], batch size: 61, lr: 3.08e-03, grad_scale: 8.0 2023-11-22 02:03:08,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1751626.6666666667, ans=0.1 2023-11-22 02:03:09,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1751626.6666666667, ans=0.0 2023-11-22 02:03:11,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262750 2023-11-22 02:03:26,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=12.0 2023-11-22 02:03:29,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.87 vs. limit=12.0 2023-11-22 02:03:30,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.46 vs. limit=15.0 2023-11-22 02:03:35,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1751760.0, ans=0.0 2023-11-22 02:03:39,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1751760.0, ans=0.125 2023-11-22 02:03:46,476 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.18 vs. limit=15.0 2023-11-22 02:04:01,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1751893.3333333333, ans=0.125 2023-11-22 02:04:09,759 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10300, loss[loss=0.06364, simple_loss=0.08334, pruned_loss=0.01163, audio_tagging_loss=0.01034, over 15399.00 frames. ], tot_loss[loss=0.07262, simple_loss=0.09495, pruned_loss=0.01551, audio_tagging_loss=0.009626, over 3056000.88 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 8.0 2023-11-22 02:04:11,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1751960.0, ans=0.125 2023-11-22 02:04:14,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262800 2023-11-22 02:04:35,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=1752093.3333333333, ans=0.2 2023-11-22 02:04:37,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1752093.3333333333, ans=0.2 2023-11-22 02:04:40,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1752093.3333333333, ans=0.1 2023-11-22 02:04:47,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.96 vs. limit=15.0 2023-11-22 02:04:55,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1752160.0, ans=0.125 2023-11-22 02:04:56,378 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.259e+01 8.792e+01 9.429e+01 1.253e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 02:05:03,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1752226.6666666667, ans=0.0 2023-11-22 02:05:05,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1752226.6666666667, ans=0.125 2023-11-22 02:05:08,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.39 vs. limit=15.0 2023-11-22 02:05:14,238 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10350, loss[loss=0.08693, simple_loss=0.1158, pruned_loss=0.01912, audio_tagging_loss=0.009883, over 15472.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09617, pruned_loss=0.01582, audio_tagging_loss=0.009655, over 3054292.09 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 8.0 2023-11-22 02:05:16,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1752293.3333333333, ans=0.125 2023-11-22 02:05:16,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.60 vs. limit=22.5 2023-11-22 02:05:20,467 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262850 2023-11-22 02:05:25,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1752293.3333333333, ans=0.2 2023-11-22 02:05:30,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1752360.0, ans=0.1 2023-11-22 02:05:41,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1752426.6666666667, ans=0.125 2023-11-22 02:05:54,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1752493.3333333333, ans=0.2 2023-11-22 02:05:59,404 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:06:13,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1752560.0, ans=0.125 2023-11-22 02:06:17,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1752560.0, ans=0.0 2023-11-22 02:06:19,265 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10400, loss[loss=0.06515, simple_loss=0.08645, pruned_loss=0.01373, audio_tagging_loss=0.008192, over 15261.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09564, pruned_loss=0.01566, audio_tagging_loss=0.009764, over 3047541.84 frames. ], batch size: 59, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:06:24,739 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262900 2023-11-22 02:06:44,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1752760.0, ans=0.125 2023-11-22 02:07:02,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.92 vs. limit=15.0 2023-11-22 02:07:04,964 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.068e+01 8.726e+01 9.431e+01 1.284e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-22 02:07:09,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1752893.3333333333, ans=0.0 2023-11-22 02:07:22,673 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10450, loss[loss=0.06491, simple_loss=0.07982, pruned_loss=0.01488, audio_tagging_loss=0.01013, over 15600.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09451, pruned_loss=0.01554, audio_tagging_loss=0.009835, over 3046093.46 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:07:27,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 262950 2023-11-22 02:07:51,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1753093.3333333333, ans=0.0 2023-11-22 02:07:54,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1753093.3333333333, ans=0.125 2023-11-22 02:08:18,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1753226.6666666667, ans=0.025 2023-11-22 02:08:21,363 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:08:22,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1753226.6666666667, ans=0.025 2023-11-22 02:08:25,954 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10500, loss[loss=0.06205, simple_loss=0.07536, pruned_loss=0.01358, audio_tagging_loss=0.01079, over 13623.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.0949, pruned_loss=0.01565, audio_tagging_loss=0.009635, over 3043770.61 frames. ], batch size: 53, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:08:29,047 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.68 vs. limit=22.5 2023-11-22 02:08:31,502 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263000 2023-11-22 02:09:10,678 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.45 vs. limit=15.0 2023-11-22 02:09:12,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.420e+01 8.257e+01 8.868e+01 9.626e+01 1.177e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 02:09:29,701 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.66 vs. limit=15.0 2023-11-22 02:09:31,577 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10550, loss[loss=0.06631, simple_loss=0.0743, pruned_loss=0.01822, audio_tagging_loss=0.01093, over 13184.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09537, pruned_loss=0.01579, audio_tagging_loss=0.009469, over 3038565.71 frames. ], batch size: 52, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:09:35,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1753626.6666666667, ans=0.1 2023-11-22 02:09:37,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263050 2023-11-22 02:10:16,457 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:10:16,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1753826.6666666667, ans=0.0 2023-11-22 02:10:17,963 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.39 vs. limit=22.5 2023-11-22 02:10:31,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1753893.3333333333, ans=0.1 2023-11-22 02:10:32,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1753893.3333333333, ans=0.125 2023-11-22 02:10:32,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1753893.3333333333, ans=0.1 2023-11-22 02:10:33,660 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:10:35,913 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10600, loss[loss=0.06176, simple_loss=0.08026, pruned_loss=0.01209, audio_tagging_loss=0.00955, over 14957.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09478, pruned_loss=0.01579, audio_tagging_loss=0.009405, over 3032590.53 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:10:40,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263100 2023-11-22 02:10:43,923 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.51 vs. limit=15.0 2023-11-22 02:10:44,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1753960.0, ans=0.125 2023-11-22 02:11:19,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1754160.0, ans=10.0 2023-11-22 02:11:21,768 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 8.070e+01 8.614e+01 9.337e+01 1.249e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-22 02:11:29,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1754226.6666666667, ans=0.0 2023-11-22 02:11:38,650 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10650, loss[loss=0.07449, simple_loss=0.09569, pruned_loss=0.01693, audio_tagging_loss=0.009715, over 15069.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09518, pruned_loss=0.01583, audio_tagging_loss=0.009341, over 3037863.26 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:11:41,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1754293.3333333333, ans=0.125 2023-11-22 02:11:43,590 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263150 2023-11-22 02:11:49,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.80 vs. limit=15.0 2023-11-22 02:11:53,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1754360.0, ans=0.125 2023-11-22 02:12:37,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1754560.0, ans=0.1 2023-11-22 02:12:42,120 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10700, loss[loss=0.07798, simple_loss=0.1029, pruned_loss=0.01612, audio_tagging_loss=0.01043, over 14740.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09499, pruned_loss=0.01586, audio_tagging_loss=0.009367, over 3034141.61 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:12:47,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263200 2023-11-22 02:13:00,103 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:13:04,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1754693.3333333333, ans=0.025 2023-11-22 02:13:29,042 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.759e+01 7.890e+01 8.587e+01 9.339e+01 1.239e+02, threshold=1.717e+02, percent-clipped=0.0 2023-11-22 02:13:40,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1754893.3333333333, ans=0.125 2023-11-22 02:13:47,139 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10750, loss[loss=0.09033, simple_loss=0.1283, pruned_loss=0.01826, audio_tagging_loss=0.007928, over 15188.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09498, pruned_loss=0.01584, audio_tagging_loss=0.009376, over 3034314.55 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:13:52,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263250 2023-11-22 02:14:15,145 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.23 vs. limit=22.5 2023-11-22 02:14:21,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1755093.3333333333, ans=10.0 2023-11-22 02:14:46,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1755226.6666666667, ans=0.125 2023-11-22 02:14:49,839 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10800, loss[loss=0.07093, simple_loss=0.09304, pruned_loss=0.01399, audio_tagging_loss=0.01042, over 15251.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09542, pruned_loss=0.01579, audio_tagging_loss=0.009305, over 3036231.16 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:14:51,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.90 vs. limit=10.0 2023-11-22 02:14:54,748 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263300 2023-11-22 02:14:59,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1755293.3333333333, ans=0.125 2023-11-22 02:15:16,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1755426.6666666667, ans=0.125 2023-11-22 02:15:23,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1755426.6666666667, ans=0.125 2023-11-22 02:15:25,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1755426.6666666667, ans=0.125 2023-11-22 02:15:37,524 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 8.311e+01 8.848e+01 9.855e+01 1.832e+02, threshold=1.770e+02, percent-clipped=2.0 2023-11-22 02:15:54,662 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10850, loss[loss=0.08028, simple_loss=0.1059, pruned_loss=0.01927, audio_tagging_loss=0.008041, over 15388.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09451, pruned_loss=0.01552, audio_tagging_loss=0.009383, over 3037684.22 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:15:57,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1755626.6666666667, ans=0.2 2023-11-22 02:15:59,598 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263350 2023-11-22 02:16:13,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1755693.3333333333, ans=0.2 2023-11-22 02:16:32,358 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.06 vs. limit=15.0 2023-11-22 02:16:39,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.68 vs. limit=15.0 2023-11-22 02:16:56,561 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:16:57,473 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 02:16:58,678 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10900, loss[loss=0.09045, simple_loss=0.1304, pruned_loss=0.01886, audio_tagging_loss=0.006405, over 14920.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09412, pruned_loss=0.01534, audio_tagging_loss=0.00949, over 3034035.58 frames. ], batch size: 53, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:17:00,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1755960.0, ans=0.1 2023-11-22 02:17:04,294 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263400 2023-11-22 02:17:11,297 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.75 vs. limit=15.0 2023-11-22 02:17:40,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.94 vs. limit=22.5 2023-11-22 02:17:46,988 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 8.172e+01 8.770e+01 9.474e+01 2.328e+02, threshold=1.754e+02, percent-clipped=1.0 2023-11-22 02:18:01,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1756226.6666666667, ans=0.0 2023-11-22 02:18:02,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1756293.3333333333, ans=0.2 2023-11-22 02:18:03,854 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 10950, loss[loss=0.05516, simple_loss=0.06026, pruned_loss=0.01029, audio_tagging_loss=0.01474, over 14662.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09436, pruned_loss=0.01531, audio_tagging_loss=0.009548, over 3037048.09 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:18:08,979 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263450 2023-11-22 02:18:08,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1756293.3333333333, ans=0.125 2023-11-22 02:18:10,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1756293.3333333333, ans=0.1 2023-11-22 02:18:25,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1756360.0, ans=0.0 2023-11-22 02:18:36,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1756426.6666666667, ans=0.125 2023-11-22 02:18:44,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1756493.3333333333, ans=0.125 2023-11-22 02:18:57,844 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.94 vs. limit=15.0 2023-11-22 02:18:59,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1756560.0, ans=0.125 2023-11-22 02:19:05,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1756560.0, ans=0.125 2023-11-22 02:19:07,475 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11000, loss[loss=0.06562, simple_loss=0.0845, pruned_loss=0.01444, audio_tagging_loss=0.008941, over 15434.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09426, pruned_loss=0.01528, audio_tagging_loss=0.009696, over 3043225.18 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:19:13,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263500 2023-11-22 02:19:21,197 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 02:19:27,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.49 vs. limit=15.0 2023-11-22 02:19:27,963 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.67 vs. limit=22.5 2023-11-22 02:19:28,717 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:19:45,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1756826.6666666667, ans=0.125 2023-11-22 02:19:55,170 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.467e+01 8.140e+01 8.725e+01 9.449e+01 1.514e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-22 02:20:07,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1756893.3333333333, ans=0.07 2023-11-22 02:20:12,235 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11050, loss[loss=0.05894, simple_loss=0.07579, pruned_loss=0.009737, audio_tagging_loss=0.01131, over 14240.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09442, pruned_loss=0.01542, audio_tagging_loss=0.009674, over 3038235.22 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:20:17,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263550 2023-11-22 02:20:18,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1756960.0, ans=0.125 2023-11-22 02:21:16,226 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11100, loss[loss=0.08162, simple_loss=0.0975, pruned_loss=0.01998, audio_tagging_loss=0.01289, over 14602.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09571, pruned_loss=0.01599, audio_tagging_loss=0.009633, over 3034068.32 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:21:21,248 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263600 2023-11-22 02:21:33,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1757360.0, ans=0.1 2023-11-22 02:21:42,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1757426.6666666667, ans=0.0 2023-11-22 02:21:45,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.78 vs. limit=6.0 2023-11-22 02:21:49,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1757426.6666666667, ans=0.0 2023-11-22 02:22:00,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1757493.3333333333, ans=0.125 2023-11-22 02:22:02,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1757493.3333333333, ans=0.0 2023-11-22 02:22:03,459 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.113e+01 8.819e+01 9.747e+01 1.374e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 02:22:05,969 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.87 vs. limit=12.0 2023-11-22 02:22:20,502 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11150, loss[loss=0.05868, simple_loss=0.07054, pruned_loss=0.01423, audio_tagging_loss=0.009189, over 14569.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.0965, pruned_loss=0.01616, audio_tagging_loss=0.009684, over 3034751.66 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0 2023-11-22 02:22:24,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.87 vs. limit=10.0 2023-11-22 02:22:25,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263650 2023-11-22 02:22:28,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1757626.6666666667, ans=0.2 2023-11-22 02:22:33,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1757693.3333333333, ans=0.5 2023-11-22 02:22:48,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1757760.0, ans=0.2 2023-11-22 02:23:19,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1757893.3333333333, ans=0.0 2023-11-22 02:23:25,254 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11200, loss[loss=0.06952, simple_loss=0.0903, pruned_loss=0.01447, audio_tagging_loss=0.009901, over 14620.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09608, pruned_loss=0.01592, audio_tagging_loss=0.009787, over 3040864.36 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 32.0 2023-11-22 02:23:30,080 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263700 2023-11-22 02:24:10,511 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:24:11,343 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.183e+01 8.908e+01 9.614e+01 1.304e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-22 02:24:27,699 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11250, loss[loss=0.07835, simple_loss=0.1033, pruned_loss=0.01658, audio_tagging_loss=0.01013, over 17283.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09609, pruned_loss=0.01584, audio_tagging_loss=0.009756, over 3044576.15 frames. ], batch size: 66, lr: 3.08e-03, grad_scale: 32.0 2023-11-22 02:24:32,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263750 2023-11-22 02:24:34,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1758293.3333333333, ans=0.125 2023-11-22 02:24:39,974 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.70 vs. limit=15.0 2023-11-22 02:24:40,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1758360.0, ans=0.0 2023-11-22 02:24:44,978 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:24:59,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1758426.6666666667, ans=0.125 2023-11-22 02:25:05,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1758493.3333333333, ans=0.0 2023-11-22 02:25:08,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.42 vs. limit=12.0 2023-11-22 02:25:15,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1758493.3333333333, ans=0.0 2023-11-22 02:25:16,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1758493.3333333333, ans=0.125 2023-11-22 02:25:26,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1758560.0, ans=0.125 2023-11-22 02:25:31,822 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11300, loss[loss=0.0714, simple_loss=0.08986, pruned_loss=0.0166, audio_tagging_loss=0.009875, over 15978.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09478, pruned_loss=0.01556, audio_tagging_loss=0.009574, over 3047857.71 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 32.0 2023-11-22 02:25:32,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1758626.6666666667, ans=0.04949747468305833 2023-11-22 02:25:36,981 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263800 2023-11-22 02:25:37,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1758626.6666666667, ans=0.0 2023-11-22 02:26:10,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.23 vs. limit=22.5 2023-11-22 02:26:18,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1758826.6666666667, ans=0.0 2023-11-22 02:26:19,799 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.464e+01 8.064e+01 8.666e+01 9.719e+01 1.287e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-22 02:26:36,257 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11350, loss[loss=0.07043, simple_loss=0.0965, pruned_loss=0.0139, audio_tagging_loss=0.008278, over 16464.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09449, pruned_loss=0.01547, audio_tagging_loss=0.009401, over 3046581.88 frames. ], batch size: 61, lr: 3.08e-03, grad_scale: 32.0 2023-11-22 02:26:39,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1758960.0, ans=0.125 2023-11-22 02:26:41,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263850 2023-11-22 02:26:55,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1759026.6666666667, ans=0.0 2023-11-22 02:27:35,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1759226.6666666667, ans=0.1 2023-11-22 02:27:39,813 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11400, loss[loss=0.06505, simple_loss=0.09051, pruned_loss=0.01194, audio_tagging_loss=0.00785, over 15413.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09521, pruned_loss=0.01565, audio_tagging_loss=0.009292, over 3046187.88 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 32.0 2023-11-22 02:27:44,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263900 2023-11-22 02:27:45,193 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:28:26,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1759493.3333333333, ans=0.125 2023-11-22 02:28:27,241 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 8.049e+01 8.601e+01 9.494e+01 1.206e+02, threshold=1.720e+02, percent-clipped=0.0 2023-11-22 02:28:30,453 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=15.0 2023-11-22 02:28:33,958 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.50 vs. limit=22.5 2023-11-22 02:28:39,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1759560.0, ans=0.2 2023-11-22 02:28:43,101 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11450, loss[loss=0.06865, simple_loss=0.08689, pruned_loss=0.01621, audio_tagging_loss=0.008992, over 15079.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09541, pruned_loss=0.01569, audio_tagging_loss=0.009128, over 3046371.23 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 32.0 2023-11-22 02:28:48,602 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 263950 2023-11-22 02:29:16,171 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.03 vs. limit=15.0 2023-11-22 02:29:28,288 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.84 vs. limit=10.0 2023-11-22 02:29:47,460 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11500, loss[loss=0.06036, simple_loss=0.07985, pruned_loss=0.01259, audio_tagging_loss=0.007849, over 14944.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09549, pruned_loss=0.01568, audio_tagging_loss=0.009072, over 3050091.08 frames. ], batch size: 55, lr: 3.07e-03, grad_scale: 32.0 2023-11-22 02:29:50,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1759960.0, ans=0.0 2023-11-22 02:29:53,111 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264000 2023-11-22 02:29:53,571 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.47 vs. limit=12.0 2023-11-22 02:29:54,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1759960.0, ans=0.1 2023-11-22 02:30:00,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1759960.0, ans=0.125 2023-11-22 02:30:03,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1760026.6666666667, ans=0.2 2023-11-22 02:30:24,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1760093.3333333333, ans=0.125 2023-11-22 02:30:38,030 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.114e+01 8.860e+01 9.560e+01 1.133e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-22 02:30:54,559 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11550, loss[loss=0.07097, simple_loss=0.0939, pruned_loss=0.01538, audio_tagging_loss=0.008643, over 16149.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09596, pruned_loss=0.01588, audio_tagging_loss=0.009127, over 3055649.90 frames. ], batch size: 62, lr: 3.07e-03, grad_scale: 32.0 2023-11-22 02:30:55,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.90 vs. limit=22.5 2023-11-22 02:30:59,463 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264050 2023-11-22 02:31:36,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1760493.3333333333, ans=0.125 2023-11-22 02:31:37,437 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 02:31:57,995 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11600, loss[loss=0.05839, simple_loss=0.07069, pruned_loss=0.012, audio_tagging_loss=0.01104, over 14735.00 frames. ], tot_loss[loss=0.07219, simple_loss=0.09466, pruned_loss=0.01559, audio_tagging_loss=0.009273, over 3059046.66 frames. ], batch size: 56, lr: 3.07e-03, grad_scale: 32.0 2023-11-22 02:31:59,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1760626.6666666667, ans=15.0 2023-11-22 02:32:02,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1760626.6666666667, ans=0.0 2023-11-22 02:32:03,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264100 2023-11-22 02:32:17,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1760693.3333333333, ans=0.0 2023-11-22 02:32:45,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1760826.6666666667, ans=0.125 2023-11-22 02:32:47,755 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.153e+01 8.705e+01 9.370e+01 1.172e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-22 02:32:54,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1760893.3333333333, ans=0.95 2023-11-22 02:33:02,418 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11650, loss[loss=0.07374, simple_loss=0.09659, pruned_loss=0.01616, audio_tagging_loss=0.009284, over 15157.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09498, pruned_loss=0.01566, audio_tagging_loss=0.009211, over 3057730.91 frames. ], batch size: 58, lr: 3.07e-03, grad_scale: 16.0 2023-11-22 02:33:07,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264150 2023-11-22 02:33:09,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1760960.0, ans=0.0 2023-11-22 02:34:06,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1761293.3333333333, ans=0.5 2023-11-22 02:34:07,583 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11700, loss[loss=0.07748, simple_loss=0.1031, pruned_loss=0.01824, audio_tagging_loss=0.007669, over 15107.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09447, pruned_loss=0.01566, audio_tagging_loss=0.009268, over 3054548.11 frames. ], batch size: 59, lr: 3.07e-03, grad_scale: 16.0 2023-11-22 02:34:12,573 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264200 2023-11-22 02:34:29,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1761360.0, ans=0.2 2023-11-22 02:34:35,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1761426.6666666667, ans=0.0 2023-11-22 02:34:42,033 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.31 vs. limit=6.0 2023-11-22 02:34:55,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1761493.3333333333, ans=0.125 2023-11-22 02:34:57,767 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.071e+01 8.694e+01 9.444e+01 1.226e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 02:35:08,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1761560.0, ans=0.125 2023-11-22 02:35:11,127 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11750, loss[loss=0.06014, simple_loss=0.07136, pruned_loss=0.01301, audio_tagging_loss=0.01145, over 14624.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09343, pruned_loss=0.01557, audio_tagging_loss=0.009391, over 3055030.47 frames. ], batch size: 58, lr: 3.07e-03, grad_scale: 16.0 2023-11-22 02:35:13,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1761626.6666666667, ans=0.0 2023-11-22 02:35:16,069 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264250 2023-11-22 02:35:24,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1761693.3333333333, ans=0.125 2023-11-22 02:35:29,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1761693.3333333333, ans=0.125 2023-11-22 02:35:33,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1761693.3333333333, ans=0.1 2023-11-22 02:36:08,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1761893.3333333333, ans=0.07 2023-11-22 02:36:15,725 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11800, loss[loss=0.06493, simple_loss=0.08478, pruned_loss=0.01273, audio_tagging_loss=0.009809, over 15694.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09325, pruned_loss=0.0156, audio_tagging_loss=0.009391, over 3050455.77 frames. ], batch size: 61, lr: 3.07e-03, grad_scale: 16.0 2023-11-22 02:36:21,267 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264300 2023-11-22 02:36:21,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1761960.0, ans=0.0 2023-11-22 02:36:34,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1762026.6666666667, ans=0.125 2023-11-22 02:36:35,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1762026.6666666667, ans=0.125 2023-11-22 02:36:38,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1762026.6666666667, ans=0.0 2023-11-22 02:36:41,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1762093.3333333333, ans=0.0 2023-11-22 02:37:00,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.33 vs. limit=10.0 2023-11-22 02:37:06,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.285e+01 8.657e+01 9.300e+01 1.154e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-22 02:37:21,179 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11850, loss[loss=0.09818, simple_loss=0.1228, pruned_loss=0.02722, audio_tagging_loss=0.009574, over 16076.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09445, pruned_loss=0.01576, audio_tagging_loss=0.00935, over 3048236.90 frames. ], batch size: 58, lr: 3.07e-03, grad_scale: 16.0 2023-11-22 02:37:26,196 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264350 2023-11-22 02:37:54,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1762426.6666666667, ans=0.0 2023-11-22 02:38:04,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1762493.3333333333, ans=0.125 2023-11-22 02:38:05,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1762493.3333333333, ans=0.09899494936611666 2023-11-22 02:38:11,284 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.36 vs. limit=15.0 2023-11-22 02:38:23,851 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11900, loss[loss=0.06175, simple_loss=0.0757, pruned_loss=0.01101, audio_tagging_loss=0.01289, over 15833.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09428, pruned_loss=0.01565, audio_tagging_loss=0.009481, over 3052181.90 frames. ], batch size: 61, lr: 3.07e-03, grad_scale: 8.0 2023-11-22 02:38:28,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264400 2023-11-22 02:38:42,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1762693.3333333333, ans=0.1 2023-11-22 02:38:50,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1762760.0, ans=0.125 2023-11-22 02:39:00,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1762760.0, ans=0.125 2023-11-22 02:39:00,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1762760.0, ans=0.125 2023-11-22 02:39:12,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1762826.6666666667, ans=0.0 2023-11-22 02:39:15,102 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.906e+01 8.188e+01 8.826e+01 9.362e+01 1.336e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-22 02:39:15,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1762893.3333333333, ans=0.2 2023-11-22 02:39:18,314 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.85 vs. limit=15.0 2023-11-22 02:39:20,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1762893.3333333333, ans=0.09899494936611666 2023-11-22 02:39:27,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1762960.0, ans=0.125 2023-11-22 02:39:28,029 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 11950, loss[loss=0.05343, simple_loss=0.06883, pruned_loss=0.008188, audio_tagging_loss=0.01082, over 15736.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09413, pruned_loss=0.01551, audio_tagging_loss=0.009605, over 3043950.51 frames. ], batch size: 60, lr: 3.07e-03, grad_scale: 8.0 2023-11-22 02:39:33,681 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264450 2023-11-22 02:39:48,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1763026.6666666667, ans=0.2 2023-11-22 02:39:59,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.60 vs. limit=15.0 2023-11-22 02:40:07,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.41 vs. limit=15.0 2023-11-22 02:40:14,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1763160.0, ans=0.2 2023-11-22 02:40:22,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1763226.6666666667, ans=0.05 2023-11-22 02:40:25,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1763226.6666666667, ans=0.0 2023-11-22 02:40:31,120 INFO [train_asr.py:1221] (3/4) Epoch 22, batch 12000, loss[loss=0.07942, simple_loss=0.1099, pruned_loss=0.01491, audio_tagging_loss=0.009563, over 15558.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09386, pruned_loss=0.01553, audio_tagging_loss=0.009707, over 3042295.28 frames. ], batch size: 56, lr: 3.07e-03, grad_scale: 16.0 2023-11-22 02:40:31,120 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 02:40:54,997 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.0437, 2.3449, 2.8958, 2.5118], device='cuda:3') 2023-11-22 02:41:14,442 INFO [train_asr.py:1253] (3/4) Epoch 22, validation: loss=0.05922, simple_loss=0.05191, pruned_loss=0.005254, audio_tagging_loss=0.02801, over 4681554.00 frames. 2023-11-22 02:41:14,443 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 02:41:18,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1763293.3333333333, ans=0.125 2023-11-22 02:41:19,297 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264500 2023-11-22 02:41:23,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0 2023-11-22 02:41:27,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1763360.0, ans=0.1 2023-11-22 02:42:19,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.64 vs. limit=15.0 2023-11-22 02:42:19,898 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 0, loss[loss=0.07604, simple_loss=0.07622, pruned_loss=0.01375, audio_tagging_loss=0.02418, over 16215.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.07622, pruned_loss=0.01375, audio_tagging_loss=0.02418, over 16215.00 frames. ], batch size: 62, lr: 3.00e-03, grad_scale: 32.0 2023-11-22 02:42:19,899 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 02:42:55,351 INFO [train_asr.py:1253] (3/4) Epoch 23, validation: loss=0.05874, simple_loss=0.05183, pruned_loss=0.005194, audio_tagging_loss=0.02763, over 4681554.00 frames. 2023-11-22 02:42:55,353 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 02:43:13,342 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.370e+01 9.150e+01 9.856e+01 1.621e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-22 02:43:17,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1763540.0, ans=0.0 2023-11-22 02:43:30,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264550 2023-11-22 02:43:47,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1763740.0, ans=0.05 2023-11-22 02:43:57,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1763740.0, ans=0.125 2023-11-22 02:43:59,714 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 50, loss[loss=0.09224, simple_loss=0.1149, pruned_loss=0.01783, audio_tagging_loss=0.01697, over 15716.00 frames. ], tot_loss[loss=0.07902, simple_loss=0.09159, pruned_loss=0.01479, audio_tagging_loss=0.01844, over 689331.84 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:43:59,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1763806.6666666667, ans=0.2 2023-11-22 02:44:00,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.49 vs. limit=15.0 2023-11-22 02:44:03,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1763806.6666666667, ans=0.035 2023-11-22 02:44:06,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.77 vs. limit=15.0 2023-11-22 02:44:24,015 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.35 vs. limit=6.0 2023-11-22 02:44:35,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264600 2023-11-22 02:44:35,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1763940.0, ans=0.125 2023-11-22 02:44:38,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1764006.6666666667, ans=0.1 2023-11-22 02:44:50,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1764073.3333333333, ans=0.125 2023-11-22 02:45:00,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1764073.3333333333, ans=0.025 2023-11-22 02:45:00,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1764073.3333333333, ans=0.95 2023-11-22 02:45:01,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1764073.3333333333, ans=0.125 2023-11-22 02:45:05,752 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 100, loss[loss=0.07776, simple_loss=0.09982, pruned_loss=0.01317, audio_tagging_loss=0.01468, over 15121.00 frames. ], tot_loss[loss=0.07974, simple_loss=0.09371, pruned_loss=0.01528, audio_tagging_loss=0.0176, over 1208634.72 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:45:23,743 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 9.017e+01 9.520e+01 1.030e+02 1.259e+02, threshold=1.904e+02, percent-clipped=0.0 2023-11-22 02:45:40,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264650 2023-11-22 02:46:00,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1764406.6666666667, ans=0.1 2023-11-22 02:46:03,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1764406.6666666667, ans=0.0 2023-11-22 02:46:04,867 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.47 vs. limit=6.0 2023-11-22 02:46:08,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1764406.6666666667, ans=0.125 2023-11-22 02:46:10,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.63 vs. limit=15.0 2023-11-22 02:46:11,380 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 150, loss[loss=0.08111, simple_loss=0.1193, pruned_loss=0.01414, audio_tagging_loss=0.007301, over 15941.00 frames. ], tot_loss[loss=0.0795, simple_loss=0.09624, pruned_loss=0.01565, audio_tagging_loss=0.01573, over 1618989.92 frames. ], batch size: 59, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:46:31,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.45 vs. limit=12.0 2023-11-22 02:46:39,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1764606.6666666667, ans=0.0 2023-11-22 02:46:46,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264700 2023-11-22 02:47:04,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1764740.0, ans=0.125 2023-11-22 02:47:07,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1764740.0, ans=0.0 2023-11-22 02:47:16,099 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 200, loss[loss=0.08561, simple_loss=0.1156, pruned_loss=0.01912, audio_tagging_loss=0.008662, over 15423.00 frames. ], tot_loss[loss=0.07766, simple_loss=0.0959, pruned_loss=0.01575, audio_tagging_loss=0.01396, over 1932675.76 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:47:20,393 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.45 vs. limit=15.0 2023-11-22 02:47:30,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1764873.3333333333, ans=0.125 2023-11-22 02:47:33,131 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.91 vs. limit=15.0 2023-11-22 02:47:33,578 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.877e+01 8.445e+01 8.941e+01 9.618e+01 1.553e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-22 02:47:38,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1764873.3333333333, ans=0.0 2023-11-22 02:47:43,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.70 vs. limit=15.0 2023-11-22 02:47:50,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1764940.0, ans=0.125 2023-11-22 02:47:50,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264750 2023-11-22 02:47:53,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1765006.6666666667, ans=0.0 2023-11-22 02:47:56,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.59 vs. limit=12.0 2023-11-22 02:48:20,498 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 250, loss[loss=0.08456, simple_loss=0.1087, pruned_loss=0.02083, audio_tagging_loss=0.009378, over 15206.00 frames. ], tot_loss[loss=0.07648, simple_loss=0.09623, pruned_loss=0.01587, audio_tagging_loss=0.0125, over 2181910.03 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:48:20,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1765140.0, ans=0.0 2023-11-22 02:48:53,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1765273.3333333333, ans=0.1 2023-11-22 02:48:55,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264800 2023-11-22 02:49:26,629 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 300, loss[loss=0.07745, simple_loss=0.1042, pruned_loss=0.01606, audio_tagging_loss=0.009294, over 14458.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.0966, pruned_loss=0.01595, audio_tagging_loss=0.01159, over 2370039.05 frames. ], batch size: 53, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:49:33,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1765473.3333333333, ans=0.1 2023-11-22 02:49:44,446 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.292e+01 9.030e+01 9.528e+01 1.689e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-22 02:49:45,293 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.22 vs. limit=22.5 2023-11-22 02:49:55,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.46 vs. limit=22.5 2023-11-22 02:49:59,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1765606.6666666667, ans=0.125 2023-11-22 02:50:00,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1765606.6666666667, ans=0.125 2023-11-22 02:50:01,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264850 2023-11-22 02:50:01,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1765606.6666666667, ans=0.0 2023-11-22 02:50:11,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1765673.3333333333, ans=0.125 2023-11-22 02:50:11,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1765673.3333333333, ans=0.125 2023-11-22 02:50:31,834 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 350, loss[loss=0.06286, simple_loss=0.07535, pruned_loss=0.01497, audio_tagging_loss=0.01021, over 15159.00 frames. ], tot_loss[loss=0.0749, simple_loss=0.0961, pruned_loss=0.01577, audio_tagging_loss=0.01108, over 2525584.75 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:50:34,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1765806.6666666667, ans=0.1 2023-11-22 02:50:45,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1765873.3333333333, ans=0.125 2023-11-22 02:50:48,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.51 vs. limit=15.0 2023-11-22 02:50:57,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1765940.0, ans=0.125 2023-11-22 02:50:57,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1765940.0, ans=0.0 2023-11-22 02:51:07,131 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264900 2023-11-22 02:51:36,547 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 400, loss[loss=0.04504, simple_loss=0.04949, pruned_loss=0.007377, audio_tagging_loss=0.01292, over 15096.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09614, pruned_loss=0.01593, audio_tagging_loss=0.01064, over 2638580.50 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 32.0 2023-11-22 02:51:38,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1766140.0, ans=0.0 2023-11-22 02:51:55,097 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.193e+01 8.089e+01 8.847e+01 9.636e+01 1.262e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 02:52:11,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 264950 2023-11-22 02:52:17,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1766340.0, ans=0.125 2023-11-22 02:52:41,728 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 450, loss[loss=0.05833, simple_loss=0.06556, pruned_loss=0.01091, audio_tagging_loss=0.01463, over 15657.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09555, pruned_loss=0.01568, audio_tagging_loss=0.01041, over 2734210.13 frames. ], batch size: 62, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:52:43,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1766473.3333333333, ans=0.125 2023-11-22 02:52:45,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.21 vs. limit=15.0 2023-11-22 02:53:01,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1766540.0, ans=0.2 2023-11-22 02:53:10,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1766606.6666666667, ans=0.0 2023-11-22 02:53:17,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265000 2023-11-22 02:53:17,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1766606.6666666667, ans=0.0 2023-11-22 02:53:17,766 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.75 vs. limit=15.0 2023-11-22 02:53:45,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1766806.6666666667, ans=0.035 2023-11-22 02:53:46,791 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 500, loss[loss=0.07288, simple_loss=0.09869, pruned_loss=0.01638, audio_tagging_loss=0.007151, over 16112.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09526, pruned_loss=0.01582, audio_tagging_loss=0.01014, over 2798900.70 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:53:56,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.22 vs. limit=15.0 2023-11-22 02:54:06,405 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.266e+01 9.127e+01 9.856e+01 1.336e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-22 02:54:06,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1766873.3333333333, ans=0.05 2023-11-22 02:54:11,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1766940.0, ans=0.125 2023-11-22 02:54:21,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265050 2023-11-22 02:54:32,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.86 vs. limit=15.0 2023-11-22 02:54:45,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.12 vs. limit=15.0 2023-11-22 02:54:50,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1767140.0, ans=0.125 2023-11-22 02:54:51,571 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 550, loss[loss=0.08832, simple_loss=0.1168, pruned_loss=0.0227, audio_tagging_loss=0.007239, over 16346.00 frames. ], tot_loss[loss=0.07319, simple_loss=0.09509, pruned_loss=0.01564, audio_tagging_loss=0.01, over 2855049.91 frames. ], batch size: 62, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:54:54,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1767140.0, ans=0.1 2023-11-22 02:54:59,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1767140.0, ans=0.2 2023-11-22 02:54:59,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1767140.0, ans=0.2 2023-11-22 02:55:02,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1767206.6666666667, ans=0.125 2023-11-22 02:55:11,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1767206.6666666667, ans=0.0 2023-11-22 02:55:14,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1767206.6666666667, ans=0.125 2023-11-22 02:55:26,330 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265100 2023-11-22 02:55:31,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1767340.0, ans=0.125 2023-11-22 02:55:34,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1767340.0, ans=10.0 2023-11-22 02:55:41,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1767340.0, ans=0.125 2023-11-22 02:55:54,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1767473.3333333333, ans=0.0 2023-11-22 02:55:55,627 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 600, loss[loss=0.08255, simple_loss=0.1161, pruned_loss=0.01555, audio_tagging_loss=0.008964, over 17608.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09558, pruned_loss=0.01574, audio_tagging_loss=0.009847, over 2901952.35 frames. ], batch size: 65, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:56:15,830 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.111e+01 8.755e+01 9.380e+01 1.139e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-22 02:56:18,770 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.98 vs. limit=6.0 2023-11-22 02:56:31,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265150 2023-11-22 02:57:01,418 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 650, loss[loss=0.0356, simple_loss=0.03529, pruned_loss=0.006854, audio_tagging_loss=0.0111, over 14104.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09514, pruned_loss=0.01576, audio_tagging_loss=0.009732, over 2931288.82 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:57:01,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1767806.6666666667, ans=0.125 2023-11-22 02:57:16,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.62 vs. limit=22.5 2023-11-22 02:57:25,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1767873.3333333333, ans=0.04949747468305833 2023-11-22 02:57:26,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1767940.0, ans=0.0 2023-11-22 02:57:35,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265200 2023-11-22 02:57:36,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1767940.0, ans=0.125 2023-11-22 02:57:37,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1767940.0, ans=0.0 2023-11-22 02:57:39,080 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:57:47,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1768006.6666666667, ans=0.0 2023-11-22 02:57:51,822 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.56 vs. limit=15.0 2023-11-22 02:58:06,613 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 700, loss[loss=0.08422, simple_loss=0.1182, pruned_loss=0.01862, audio_tagging_loss=0.006512, over 15539.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09535, pruned_loss=0.01572, audio_tagging_loss=0.009688, over 2948938.83 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 02:58:16,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1768140.0, ans=0.0 2023-11-22 02:58:25,111 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 02:58:26,073 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.011e+01 8.675e+01 9.413e+01 1.282e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-22 02:58:42,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265250 2023-11-22 02:58:47,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1768340.0, ans=0.125 2023-11-22 02:59:08,842 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.01 vs. limit=15.0 2023-11-22 02:59:11,962 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 750, loss[loss=0.08088, simple_loss=0.1171, pruned_loss=0.01519, audio_tagging_loss=0.007129, over 16589.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.0964, pruned_loss=0.01598, audio_tagging_loss=0.009609, over 2984743.57 frames. ], batch size: 60, lr: 3.00e-03, grad_scale: 8.0 2023-11-22 02:59:17,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1768473.3333333333, ans=0.125 2023-11-22 02:59:25,968 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.05 vs. limit=15.0 2023-11-22 02:59:29,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1768540.0, ans=0.125 2023-11-22 02:59:46,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1768606.6666666667, ans=0.125 2023-11-22 02:59:47,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265300 2023-11-22 02:59:52,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1768673.3333333333, ans=0.125 2023-11-22 03:00:04,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1768740.0, ans=0.0 2023-11-22 03:00:05,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1768740.0, ans=0.025 2023-11-22 03:00:12,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.59 vs. limit=15.0 2023-11-22 03:00:17,348 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 800, loss[loss=0.05724, simple_loss=0.0767, pruned_loss=0.008928, audio_tagging_loss=0.009964, over 15067.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09725, pruned_loss=0.0162, audio_tagging_loss=0.009652, over 3002894.39 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 03:00:39,219 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.565e+01 9.192e+01 1.025e+02 1.525e+02, threshold=1.838e+02, percent-clipped=0.0 2023-11-22 03:00:43,673 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.30 vs. limit=10.0 2023-11-22 03:00:44,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.67 vs. limit=15.0 2023-11-22 03:00:52,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1768940.0, ans=0.0 2023-11-22 03:00:53,115 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265350 2023-11-22 03:00:56,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1769006.6666666667, ans=0.1 2023-11-22 03:00:57,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.42 vs. limit=10.0 2023-11-22 03:00:58,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1769006.6666666667, ans=0.2 2023-11-22 03:01:07,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1769006.6666666667, ans=0.2 2023-11-22 03:01:09,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.72 vs. limit=12.0 2023-11-22 03:01:23,369 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 850, loss[loss=0.08203, simple_loss=0.1083, pruned_loss=0.02003, audio_tagging_loss=0.007874, over 16414.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09635, pruned_loss=0.01605, audio_tagging_loss=0.009722, over 3015813.74 frames. ], batch size: 61, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 03:01:38,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.26 vs. limit=22.5 2023-11-22 03:01:44,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1769206.6666666667, ans=0.2 2023-11-22 03:01:58,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265400 2023-11-22 03:02:07,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=1769340.0, ans=15.0 2023-11-22 03:02:15,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1769406.6666666667, ans=0.035 2023-11-22 03:02:28,368 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 900, loss[loss=0.06701, simple_loss=0.09642, pruned_loss=0.01218, audio_tagging_loss=0.006616, over 13714.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09583, pruned_loss=0.0158, audio_tagging_loss=0.009745, over 3016977.34 frames. ], batch size: 52, lr: 3.00e-03, grad_scale: 8.0 2023-11-22 03:02:46,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1769540.0, ans=0.125 2023-11-22 03:02:49,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1769540.0, ans=0.125 2023-11-22 03:02:50,267 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.073e+01 8.740e+01 9.361e+01 1.142e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 03:02:52,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1769540.0, ans=0.0 2023-11-22 03:02:57,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.60 vs. limit=12.0 2023-11-22 03:03:04,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265450 2023-11-22 03:03:17,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1769673.3333333333, ans=0.125 2023-11-22 03:03:19,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1769740.0, ans=0.2 2023-11-22 03:03:33,882 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 950, loss[loss=0.0669, simple_loss=0.08246, pruned_loss=0.0164, audio_tagging_loss=0.009278, over 15235.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09551, pruned_loss=0.01578, audio_tagging_loss=0.00964, over 3026673.03 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 8.0 2023-11-22 03:04:08,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265500 2023-11-22 03:04:25,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1770073.3333333333, ans=0.0 2023-11-22 03:04:39,187 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1000, loss[loss=0.09278, simple_loss=0.1186, pruned_loss=0.02384, audio_tagging_loss=0.009647, over 15293.00 frames. ], tot_loss[loss=0.07296, simple_loss=0.09522, pruned_loss=0.0158, audio_tagging_loss=0.009542, over 3026202.03 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 8.0 2023-11-22 03:04:56,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1770206.6666666667, ans=0.1 2023-11-22 03:04:59,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1770206.6666666667, ans=0.125 2023-11-22 03:05:00,673 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.104e+01 8.710e+01 9.492e+01 1.362e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-22 03:05:05,590 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:05:05,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1770273.3333333333, ans=0.125 2023-11-22 03:05:06,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.10 vs. limit=15.0 2023-11-22 03:05:13,768 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265550 2023-11-22 03:05:19,196 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.02 vs. limit=22.5 2023-11-22 03:05:29,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1770406.6666666667, ans=0.0 2023-11-22 03:05:43,702 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1050, loss[loss=0.07789, simple_loss=0.101, pruned_loss=0.0185, audio_tagging_loss=0.008892, over 14768.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09439, pruned_loss=0.01557, audio_tagging_loss=0.009576, over 3025326.41 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 8.0 2023-11-22 03:06:19,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265600 2023-11-22 03:06:27,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1770673.3333333333, ans=0.1 2023-11-22 03:06:30,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1770673.3333333333, ans=0.05 2023-11-22 03:06:37,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1770740.0, ans=0.0 2023-11-22 03:06:48,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1770806.6666666667, ans=0.125 2023-11-22 03:06:49,319 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1100, loss[loss=0.06696, simple_loss=0.07519, pruned_loss=0.01599, audio_tagging_loss=0.01338, over 15045.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09404, pruned_loss=0.01552, audio_tagging_loss=0.009489, over 3032458.67 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 8.0 2023-11-22 03:06:51,776 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:07:02,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1770873.3333333333, ans=0.2 2023-11-22 03:07:11,626 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 7.983e+01 8.697e+01 9.396e+01 1.258e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 03:07:14,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1770940.0, ans=0.2 2023-11-22 03:07:16,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1770940.0, ans=0.0 2023-11-22 03:07:17,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1770940.0, ans=0.125 2023-11-22 03:07:24,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265650 2023-11-22 03:07:38,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1771006.6666666667, ans=0.125 2023-11-22 03:07:38,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1771006.6666666667, ans=0.0 2023-11-22 03:07:49,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1771073.3333333333, ans=0.0 2023-11-22 03:07:54,445 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1150, loss[loss=0.0774, simple_loss=0.1045, pruned_loss=0.01685, audio_tagging_loss=0.008289, over 15195.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09393, pruned_loss=0.01541, audio_tagging_loss=0.009484, over 3034209.47 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 8.0 2023-11-22 03:07:55,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1771140.0, ans=0.125 2023-11-22 03:07:58,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1771140.0, ans=0.07 2023-11-22 03:08:02,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1771140.0, ans=0.1 2023-11-22 03:08:05,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1771140.0, ans=0.125 2023-11-22 03:08:09,867 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.76 vs. limit=15.0 2023-11-22 03:08:28,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1771273.3333333333, ans=0.125 2023-11-22 03:08:29,850 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265700 2023-11-22 03:08:43,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1771340.0, ans=0.125 2023-11-22 03:08:59,945 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1200, loss[loss=0.07443, simple_loss=0.09963, pruned_loss=0.01817, audio_tagging_loss=0.006444, over 15357.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09417, pruned_loss=0.01533, audio_tagging_loss=0.009402, over 3032245.09 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 03:09:21,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0 2023-11-22 03:09:21,609 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.059e+01 8.626e+01 9.376e+01 1.132e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-22 03:09:26,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0 2023-11-22 03:09:31,527 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.41 vs. limit=22.5 2023-11-22 03:09:33,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1771606.6666666667, ans=0.025 2023-11-22 03:09:34,692 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265750 2023-11-22 03:09:47,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.22 vs. limit=6.0 2023-11-22 03:09:53,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1771740.0, ans=0.0 2023-11-22 03:10:04,576 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1250, loss[loss=0.08228, simple_loss=0.1035, pruned_loss=0.02137, audio_tagging_loss=0.009153, over 15597.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09415, pruned_loss=0.01549, audio_tagging_loss=0.009347, over 3033271.67 frames. ], batch size: 60, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 03:10:06,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1771806.6666666667, ans=0.125 2023-11-22 03:10:11,486 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.30 vs. limit=15.0 2023-11-22 03:10:16,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1771873.3333333333, ans=0.0 2023-11-22 03:10:19,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1771873.3333333333, ans=0.1 2023-11-22 03:10:20,754 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.21 vs. limit=15.0 2023-11-22 03:10:22,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1771873.3333333333, ans=0.125 2023-11-22 03:10:25,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1771873.3333333333, ans=0.0 2023-11-22 03:10:39,928 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265800 2023-11-22 03:10:50,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1772006.6666666667, ans=0.125 2023-11-22 03:11:08,262 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.75 vs. limit=15.0 2023-11-22 03:11:10,110 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1300, loss[loss=0.05361, simple_loss=0.06093, pruned_loss=0.01335, audio_tagging_loss=0.009796, over 15864.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09286, pruned_loss=0.01536, audio_tagging_loss=0.009393, over 3033223.22 frames. ], batch size: 61, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 03:11:32,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.75 vs. limit=12.0 2023-11-22 03:11:32,798 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.081e+01 8.746e+01 9.574e+01 1.188e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 03:11:35,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.07 vs. limit=12.0 2023-11-22 03:11:36,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1772273.3333333333, ans=0.95 2023-11-22 03:11:45,777 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265850 2023-11-22 03:11:46,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1772273.3333333333, ans=0.09899494936611666 2023-11-22 03:11:46,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=15.59 vs. limit=15.0 2023-11-22 03:11:48,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1772340.0, ans=0.125 2023-11-22 03:11:59,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1772340.0, ans=0.5 2023-11-22 03:12:10,152 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.03 vs. limit=6.0 2023-11-22 03:12:14,934 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1350, loss[loss=0.07006, simple_loss=0.08312, pruned_loss=0.01684, audio_tagging_loss=0.01166, over 15399.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09388, pruned_loss=0.01558, audio_tagging_loss=0.009394, over 3036788.24 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0 2023-11-22 03:12:19,699 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.42 vs. limit=22.5 2023-11-22 03:12:21,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1772473.3333333333, ans=0.2 2023-11-22 03:12:38,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1772540.0, ans=0.2 2023-11-22 03:12:42,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1772606.6666666667, ans=0.0 2023-11-22 03:12:44,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.16 vs. limit=15.0 2023-11-22 03:12:49,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265900 2023-11-22 03:12:53,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1772673.3333333333, ans=0.2 2023-11-22 03:12:57,916 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.08 vs. limit=22.5 2023-11-22 03:13:01,485 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:13:04,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1772673.3333333333, ans=0.125 2023-11-22 03:13:04,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1772673.3333333333, ans=0.125 2023-11-22 03:13:14,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1772740.0, ans=0.0 2023-11-22 03:13:15,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1772740.0, ans=0.125 2023-11-22 03:13:19,327 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1400, loss[loss=0.09354, simple_loss=0.1256, pruned_loss=0.0216, audio_tagging_loss=0.009119, over 15752.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09443, pruned_loss=0.01562, audio_tagging_loss=0.009444, over 3040673.97 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:13:31,983 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.68 vs. limit=15.0 2023-11-22 03:13:35,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1772873.3333333333, ans=0.07 2023-11-22 03:13:41,400 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.105e+01 8.866e+01 9.820e+01 1.190e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-22 03:13:42,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.44 vs. limit=15.0 2023-11-22 03:13:54,536 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 265950 2023-11-22 03:13:58,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1773006.6666666667, ans=0.0 2023-11-22 03:14:09,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1773073.3333333333, ans=0.125 2023-11-22 03:14:11,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1773073.3333333333, ans=10.0 2023-11-22 03:14:15,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.56 vs. limit=15.0 2023-11-22 03:14:23,824 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1450, loss[loss=0.09049, simple_loss=0.1211, pruned_loss=0.02246, audio_tagging_loss=0.00748, over 15542.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09423, pruned_loss=0.0155, audio_tagging_loss=0.009487, over 3044050.39 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:14:24,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1773140.0, ans=0.0 2023-11-22 03:14:46,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1773206.6666666667, ans=0.125 2023-11-22 03:14:56,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1773273.3333333333, ans=0.125 2023-11-22 03:14:58,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266000 2023-11-22 03:15:03,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1773340.0, ans=0.125 2023-11-22 03:15:09,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1773340.0, ans=0.125 2023-11-22 03:15:16,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1773406.6666666667, ans=0.125 2023-11-22 03:15:27,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1773473.3333333333, ans=0.0 2023-11-22 03:15:28,959 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1500, loss[loss=0.06417, simple_loss=0.07909, pruned_loss=0.01403, audio_tagging_loss=0.0106, over 15031.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.0943, pruned_loss=0.01553, audio_tagging_loss=0.009497, over 3038949.71 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:15:41,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1773540.0, ans=0.2 2023-11-22 03:15:42,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1773540.0, ans=0.125 2023-11-22 03:15:48,452 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:15:51,116 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.300e+01 8.986e+01 9.912e+01 1.319e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 03:15:51,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1773540.0, ans=0.2 2023-11-22 03:16:00,693 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.97 vs. limit=15.0 2023-11-22 03:16:04,213 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266050 2023-11-22 03:16:07,243 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.27 vs. limit=22.5 2023-11-22 03:16:11,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1773673.3333333333, ans=0.125 2023-11-22 03:16:34,336 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1550, loss[loss=0.09283, simple_loss=0.1214, pruned_loss=0.02301, audio_tagging_loss=0.00911, over 15043.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09483, pruned_loss=0.01578, audio_tagging_loss=0.009475, over 3040508.38 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:16:43,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1773806.6666666667, ans=0.1 2023-11-22 03:17:01,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.26 vs. limit=15.0 2023-11-22 03:17:09,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266100 2023-11-22 03:17:11,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1774006.6666666667, ans=0.125 2023-11-22 03:17:14,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1774006.6666666667, ans=0.2 2023-11-22 03:17:20,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1774006.6666666667, ans=0.125 2023-11-22 03:17:26,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1774073.3333333333, ans=0.125 2023-11-22 03:17:38,581 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1600, loss[loss=0.06758, simple_loss=0.0872, pruned_loss=0.01333, audio_tagging_loss=0.01066, over 14986.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09513, pruned_loss=0.01582, audio_tagging_loss=0.009519, over 3045583.14 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:17:41,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1774140.0, ans=0.125 2023-11-22 03:17:42,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1774140.0, ans=0.125 2023-11-22 03:17:44,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.71 vs. limit=15.0 2023-11-22 03:17:49,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1774140.0, ans=0.1 2023-11-22 03:17:54,680 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0 2023-11-22 03:17:57,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1774206.6666666667, ans=0.125 2023-11-22 03:18:00,648 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.210e+01 8.853e+01 9.521e+01 1.233e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-22 03:18:12,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1774273.3333333333, ans=0.2 2023-11-22 03:18:13,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266150 2023-11-22 03:18:20,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1774340.0, ans=0.125 2023-11-22 03:18:31,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.42 vs. limit=15.0 2023-11-22 03:18:42,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1774473.3333333333, ans=0.2 2023-11-22 03:18:43,455 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1650, loss[loss=0.06355, simple_loss=0.08795, pruned_loss=0.01057, audio_tagging_loss=0.00901, over 15567.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.0943, pruned_loss=0.01559, audio_tagging_loss=0.009518, over 3049372.90 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:18:44,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=12.0 2023-11-22 03:19:18,926 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266200 2023-11-22 03:19:33,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1774673.3333333333, ans=0.1 2023-11-22 03:19:39,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1774740.0, ans=0.0 2023-11-22 03:19:48,685 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1700, loss[loss=0.0479, simple_loss=0.06168, pruned_loss=0.006213, audio_tagging_loss=0.01084, over 15910.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.0944, pruned_loss=0.01564, audio_tagging_loss=0.009663, over 3055795.42 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:19:48,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1774806.6666666667, ans=0.0 2023-11-22 03:19:52,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1774806.6666666667, ans=0.2 2023-11-22 03:19:59,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1774806.6666666667, ans=0.0 2023-11-22 03:20:00,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1774806.6666666667, ans=0.1 2023-11-22 03:20:11,471 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.074e+01 8.655e+01 9.245e+01 1.270e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-22 03:20:24,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266250 2023-11-22 03:20:34,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1775006.6666666667, ans=0.1 2023-11-22 03:20:37,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1775006.6666666667, ans=0.125 2023-11-22 03:20:38,499 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:20:54,028 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1750, loss[loss=0.06338, simple_loss=0.07692, pruned_loss=0.01519, audio_tagging_loss=0.009731, over 15331.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09311, pruned_loss=0.01552, audio_tagging_loss=0.009612, over 3057515.26 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:21:03,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1775140.0, ans=0.2 2023-11-22 03:21:27,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1775273.3333333333, ans=0.125 2023-11-22 03:21:28,890 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266300 2023-11-22 03:21:30,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1775273.3333333333, ans=0.125 2023-11-22 03:21:43,933 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.19 vs. limit=22.5 2023-11-22 03:21:58,587 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1800, loss[loss=0.06691, simple_loss=0.08934, pruned_loss=0.01208, audio_tagging_loss=0.01016, over 16886.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09409, pruned_loss=0.01568, audio_tagging_loss=0.009522, over 3057429.39 frames. ], batch size: 62, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:22:09,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1775473.3333333333, ans=0.125 2023-11-22 03:22:21,024 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 7.973e+01 8.651e+01 9.483e+01 1.357e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-22 03:22:23,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1775606.6666666667, ans=0.125 2023-11-22 03:22:25,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1775606.6666666667, ans=0.125 2023-11-22 03:22:33,252 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266350 2023-11-22 03:22:43,845 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:22:53,005 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.21 vs. limit=15.0 2023-11-22 03:22:55,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1775740.0, ans=0.0 2023-11-22 03:22:57,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1775740.0, ans=0.125 2023-11-22 03:23:02,241 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1850, loss[loss=0.09237, simple_loss=0.1177, pruned_loss=0.02466, audio_tagging_loss=0.008868, over 15916.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09413, pruned_loss=0.01563, audio_tagging_loss=0.009367, over 3049197.04 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:23:13,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1775873.3333333333, ans=0.125 2023-11-22 03:23:14,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0 2023-11-22 03:23:23,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1775873.3333333333, ans=0.125 2023-11-22 03:23:37,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266400 2023-11-22 03:23:47,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=12.0 2023-11-22 03:23:48,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1776006.6666666667, ans=0.125 2023-11-22 03:23:53,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.00 vs. limit=10.0 2023-11-22 03:23:54,197 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:24:06,608 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1900, loss[loss=0.05277, simple_loss=0.06748, pruned_loss=0.009609, audio_tagging_loss=0.009418, over 14847.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09327, pruned_loss=0.01539, audio_tagging_loss=0.009382, over 3053193.63 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:24:17,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1776140.0, ans=0.0 2023-11-22 03:24:30,463 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.182e+01 8.329e+01 9.047e+01 9.606e+01 1.236e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-22 03:24:34,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1776273.3333333333, ans=0.2 2023-11-22 03:24:39,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1776273.3333333333, ans=0.125 2023-11-22 03:24:41,894 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266450 2023-11-22 03:25:01,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1776406.6666666667, ans=0.0 2023-11-22 03:25:03,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1776406.6666666667, ans=0.125 2023-11-22 03:25:09,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1776406.6666666667, ans=0.125 2023-11-22 03:25:09,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1776406.6666666667, ans=0.125 2023-11-22 03:25:11,706 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 1950, loss[loss=0.04881, simple_loss=0.05659, pruned_loss=0.009795, audio_tagging_loss=0.01072, over 15210.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09284, pruned_loss=0.0152, audio_tagging_loss=0.009365, over 3049999.31 frames. ], batch size: 60, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:25:13,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1776473.3333333333, ans=0.0 2023-11-22 03:25:36,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1776606.6666666667, ans=0.1 2023-11-22 03:25:46,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266500 2023-11-22 03:25:46,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1776606.6666666667, ans=0.025 2023-11-22 03:25:50,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1776673.3333333333, ans=0.1 2023-11-22 03:26:16,499 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2000, loss[loss=0.06989, simple_loss=0.09111, pruned_loss=0.01494, audio_tagging_loss=0.009395, over 13963.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09257, pruned_loss=0.01517, audio_tagging_loss=0.009366, over 3041565.06 frames. ], batch size: 53, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:26:38,386 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:26:39,139 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.529e+01 7.854e+01 8.461e+01 9.076e+01 1.159e+02, threshold=1.692e+02, percent-clipped=0.0 2023-11-22 03:26:51,394 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266550 2023-11-22 03:26:51,949 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.90 vs. limit=15.0 2023-11-22 03:26:52,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1776940.0, ans=0.0 2023-11-22 03:26:53,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1777006.6666666667, ans=0.1 2023-11-22 03:27:19,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=1777073.3333333333, ans=0.05 2023-11-22 03:27:21,311 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2050, loss[loss=0.06906, simple_loss=0.09618, pruned_loss=0.01097, audio_tagging_loss=0.009999, over 14847.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09278, pruned_loss=0.01524, audio_tagging_loss=0.009273, over 3041980.88 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:27:33,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1777206.6666666667, ans=0.1 2023-11-22 03:27:48,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1777273.3333333333, ans=0.125 2023-11-22 03:27:56,236 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266600 2023-11-22 03:28:12,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1777406.6666666667, ans=0.125 2023-11-22 03:28:27,081 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2100, loss[loss=0.06591, simple_loss=0.08414, pruned_loss=0.01674, audio_tagging_loss=0.007104, over 15089.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.0925, pruned_loss=0.01523, audio_tagging_loss=0.009304, over 3043259.70 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:28:33,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1777473.3333333333, ans=0.2 2023-11-22 03:28:49,896 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.317e+01 8.958e+01 9.611e+01 1.215e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-22 03:29:00,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1777606.6666666667, ans=0.125 2023-11-22 03:29:01,621 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266650 2023-11-22 03:29:05,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1777673.3333333333, ans=0.125 2023-11-22 03:29:07,883 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:29:11,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1777673.3333333333, ans=0.1 2023-11-22 03:29:22,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1777740.0, ans=0.125 2023-11-22 03:29:23,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1777740.0, ans=0.1 2023-11-22 03:29:26,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1777740.0, ans=0.125 2023-11-22 03:29:31,459 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2150, loss[loss=0.08421, simple_loss=0.118, pruned_loss=0.01917, audio_tagging_loss=0.006044, over 15496.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09312, pruned_loss=0.01532, audio_tagging_loss=0.009299, over 3045458.17 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:29:44,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.68 vs. limit=15.0 2023-11-22 03:30:07,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266700 2023-11-22 03:30:10,776 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:30:14,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1778006.6666666667, ans=0.0 2023-11-22 03:30:36,535 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2200, loss[loss=0.08974, simple_loss=0.1195, pruned_loss=0.0209, audio_tagging_loss=0.00908, over 14911.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09393, pruned_loss=0.01546, audio_tagging_loss=0.009319, over 3051695.27 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:31:01,200 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.015e+01 8.446e+01 9.352e+01 1.016e+02 2.574e+02, threshold=1.870e+02, percent-clipped=1.0 2023-11-22 03:31:11,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1778273.3333333333, ans=0.125 2023-11-22 03:31:12,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266750 2023-11-22 03:31:21,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.whiten.whitening_limit, batch_count=1778340.0, ans=12.0 2023-11-22 03:31:41,910 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2250, loss[loss=0.08974, simple_loss=0.1231, pruned_loss=0.02037, audio_tagging_loss=0.00781, over 14901.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09407, pruned_loss=0.01557, audio_tagging_loss=0.009348, over 3054053.57 frames. ], batch size: 54, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:32:08,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1778606.6666666667, ans=0.125 2023-11-22 03:32:14,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1778606.6666666667, ans=0.125 2023-11-22 03:32:17,335 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266800 2023-11-22 03:32:20,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1778673.3333333333, ans=0.125 2023-11-22 03:32:47,679 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2300, loss[loss=0.05944, simple_loss=0.07468, pruned_loss=0.01355, audio_tagging_loss=0.008553, over 14795.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09404, pruned_loss=0.01547, audio_tagging_loss=0.00934, over 3048794.53 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:32:51,994 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=12.0 2023-11-22 03:33:02,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1778873.3333333333, ans=0.0 2023-11-22 03:33:08,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1778873.3333333333, ans=0.125 2023-11-22 03:33:08,809 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.91 vs. limit=6.0 2023-11-22 03:33:11,803 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.550e+01 8.208e+01 8.733e+01 9.600e+01 1.238e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 03:33:22,326 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266850 2023-11-22 03:33:26,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1779006.6666666667, ans=0.07 2023-11-22 03:33:43,871 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:33:52,705 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2350, loss[loss=0.05957, simple_loss=0.08075, pruned_loss=0.01015, audio_tagging_loss=0.009043, over 14720.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09358, pruned_loss=0.01548, audio_tagging_loss=0.009444, over 3052135.05 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:34:20,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1779273.3333333333, ans=0.125 2023-11-22 03:34:26,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1779273.3333333333, ans=0.07 2023-11-22 03:34:28,356 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266900 2023-11-22 03:34:39,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1779340.0, ans=0.0 2023-11-22 03:34:57,473 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2400, loss[loss=0.09626, simple_loss=0.142, pruned_loss=0.01793, audio_tagging_loss=0.007324, over 15597.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.0942, pruned_loss=0.0156, audio_tagging_loss=0.009576, over 3053248.21 frames. ], batch size: 53, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:35:00,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1779473.3333333333, ans=0.1 2023-11-22 03:35:22,843 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 7.925e+01 8.589e+01 9.306e+01 1.063e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-22 03:35:24,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1779606.6666666667, ans=0.125 2023-11-22 03:35:33,471 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 266950 2023-11-22 03:35:41,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1779673.3333333333, ans=0.125 2023-11-22 03:36:03,420 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2450, loss[loss=0.06933, simple_loss=0.09059, pruned_loss=0.01307, audio_tagging_loss=0.01097, over 14710.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09363, pruned_loss=0.01539, audio_tagging_loss=0.009649, over 3046480.91 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:36:03,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1779806.6666666667, ans=0.125 2023-11-22 03:36:10,561 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:36:15,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1779873.3333333333, ans=0.07 2023-11-22 03:36:18,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1779873.3333333333, ans=6.0 2023-11-22 03:36:19,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1779873.3333333333, ans=0.1 2023-11-22 03:36:39,249 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267000 2023-11-22 03:36:44,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1780006.6666666667, ans=0.0 2023-11-22 03:36:48,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1780006.6666666667, ans=0.0 2023-11-22 03:36:54,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1780006.6666666667, ans=0.1 2023-11-22 03:37:04,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1780073.3333333333, ans=0.125 2023-11-22 03:37:09,370 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2500, loss[loss=0.08833, simple_loss=0.116, pruned_loss=0.02017, audio_tagging_loss=0.01016, over 14203.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09384, pruned_loss=0.01532, audio_tagging_loss=0.009643, over 3038355.64 frames. ], batch size: 53, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:37:19,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.29 vs. limit=22.5 2023-11-22 03:37:19,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1780140.0, ans=0.0 2023-11-22 03:37:34,048 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.102e+01 8.757e+01 9.446e+01 1.143e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-22 03:37:45,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267050 2023-11-22 03:38:08,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=15.12 vs. limit=15.0 2023-11-22 03:38:08,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1780406.6666666667, ans=0.125 2023-11-22 03:38:13,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1780473.3333333333, ans=0.125 2023-11-22 03:38:14,959 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2550, loss[loss=0.07435, simple_loss=0.1095, pruned_loss=0.01349, audio_tagging_loss=0.006127, over 16033.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09464, pruned_loss=0.01546, audio_tagging_loss=0.009556, over 3043457.85 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:38:16,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1780473.3333333333, ans=0.1 2023-11-22 03:38:28,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1780540.0, ans=0.025 2023-11-22 03:38:29,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1780540.0, ans=0.1 2023-11-22 03:38:34,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1780540.0, ans=10.0 2023-11-22 03:38:45,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1780606.6666666667, ans=0.07 2023-11-22 03:38:48,669 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.84 vs. limit=22.5 2023-11-22 03:38:50,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267100 2023-11-22 03:38:50,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1780606.6666666667, ans=0.125 2023-11-22 03:38:51,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.70 vs. limit=10.0 2023-11-22 03:39:15,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1780740.0, ans=0.0 2023-11-22 03:39:17,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.99 vs. limit=22.5 2023-11-22 03:39:20,355 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2600, loss[loss=0.05561, simple_loss=0.0733, pruned_loss=0.0108, audio_tagging_loss=0.008162, over 15814.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09368, pruned_loss=0.01519, audio_tagging_loss=0.009484, over 3035214.13 frames. ], batch size: 60, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:39:45,351 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.768e+01 8.100e+01 8.795e+01 9.749e+01 1.306e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 03:39:46,059 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.02 vs. limit=15.0 2023-11-22 03:39:55,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267150 2023-11-22 03:40:21,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1781073.3333333333, ans=0.1 2023-11-22 03:40:25,444 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2650, loss[loss=0.09979, simple_loss=0.1273, pruned_loss=0.02587, audio_tagging_loss=0.01027, over 18108.00 frames. ], tot_loss[loss=0.07153, simple_loss=0.09364, pruned_loss=0.01527, audio_tagging_loss=0.009442, over 3036368.45 frames. ], batch size: 67, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:40:28,381 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.06 vs. limit=15.0 2023-11-22 03:40:38,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1781206.6666666667, ans=0.125 2023-11-22 03:40:41,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1781206.6666666667, ans=0.0 2023-11-22 03:40:41,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.55 vs. limit=6.0 2023-11-22 03:40:50,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1781273.3333333333, ans=0.125 2023-11-22 03:41:00,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267200 2023-11-22 03:41:14,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1781340.0, ans=0.125 2023-11-22 03:41:24,065 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.42 vs. limit=15.0 2023-11-22 03:41:27,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1781406.6666666667, ans=0.05 2023-11-22 03:41:28,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1781406.6666666667, ans=0.035 2023-11-22 03:41:30,824 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2700, loss[loss=0.06856, simple_loss=0.08195, pruned_loss=0.01439, audio_tagging_loss=0.01319, over 16262.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09348, pruned_loss=0.01533, audio_tagging_loss=0.009384, over 3032726.38 frames. ], batch size: 63, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:41:32,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1781473.3333333333, ans=0.2 2023-11-22 03:41:36,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1781473.3333333333, ans=0.0 2023-11-22 03:41:55,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.82 vs. limit=15.0 2023-11-22 03:41:57,229 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.172e+01 8.019e+01 8.519e+01 9.203e+01 1.202e+02, threshold=1.704e+02, percent-clipped=0.0 2023-11-22 03:42:06,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267250 2023-11-22 03:42:21,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1781673.3333333333, ans=0.0 2023-11-22 03:42:22,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.11 vs. limit=10.0 2023-11-22 03:42:23,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1781740.0, ans=0.125 2023-11-22 03:42:34,494 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0 2023-11-22 03:42:36,498 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2750, loss[loss=0.07412, simple_loss=0.09841, pruned_loss=0.01632, audio_tagging_loss=0.008598, over 16092.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09244, pruned_loss=0.0151, audio_tagging_loss=0.009383, over 3040462.56 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:42:59,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1781873.3333333333, ans=0.04949747468305833 2023-11-22 03:43:03,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1781940.0, ans=0.125 2023-11-22 03:43:04,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1781940.0, ans=0.125 2023-11-22 03:43:11,830 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267300 2023-11-22 03:43:13,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1781940.0, ans=0.125 2023-11-22 03:43:18,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1782006.6666666667, ans=0.07 2023-11-22 03:43:32,617 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:43:39,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1782073.3333333333, ans=0.0 2023-11-22 03:43:41,234 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2800, loss[loss=0.0681, simple_loss=0.08816, pruned_loss=0.01686, audio_tagging_loss=0.007157, over 15695.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09229, pruned_loss=0.01508, audio_tagging_loss=0.009353, over 3036182.38 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 32.0 2023-11-22 03:43:55,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1782206.6666666667, ans=0.2 2023-11-22 03:43:59,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1782206.6666666667, ans=0.1 2023-11-22 03:44:06,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1782273.3333333333, ans=0.125 2023-11-22 03:44:07,787 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.173e+01 8.676e+01 9.379e+01 1.430e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-22 03:44:16,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267350 2023-11-22 03:44:26,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1782340.0, ans=0.125 2023-11-22 03:44:32,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.37 vs. limit=22.5 2023-11-22 03:44:47,359 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2850, loss[loss=0.08233, simple_loss=0.1077, pruned_loss=0.01868, audio_tagging_loss=0.009817, over 16422.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09297, pruned_loss=0.01517, audio_tagging_loss=0.009293, over 3034063.00 frames. ], batch size: 60, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:44:57,479 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.50 vs. limit=15.0 2023-11-22 03:45:08,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1782540.0, ans=0.125 2023-11-22 03:45:17,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1782606.6666666667, ans=0.125 2023-11-22 03:45:22,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267400 2023-11-22 03:45:36,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1782673.3333333333, ans=0.1 2023-11-22 03:45:52,363 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2900, loss[loss=0.09137, simple_loss=0.1192, pruned_loss=0.02338, audio_tagging_loss=0.008397, over 16137.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09535, pruned_loss=0.01561, audio_tagging_loss=0.009245, over 3039141.01 frames. ], batch size: 61, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:46:00,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1782806.6666666667, ans=0.1 2023-11-22 03:46:08,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1782873.3333333333, ans=0.0 2023-11-22 03:46:19,358 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.173e+01 8.778e+01 9.450e+01 1.328e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 03:46:27,436 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267450 2023-11-22 03:46:28,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1782940.0, ans=0.125 2023-11-22 03:46:31,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1783006.6666666667, ans=0.125 2023-11-22 03:46:52,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1783073.3333333333, ans=0.04949747468305833 2023-11-22 03:46:56,575 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 2950, loss[loss=0.08312, simple_loss=0.1118, pruned_loss=0.01991, audio_tagging_loss=0.007318, over 16151.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09501, pruned_loss=0.01558, audio_tagging_loss=0.009431, over 3036884.37 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:47:03,147 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.35 vs. limit=15.0 2023-11-22 03:47:21,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1783273.3333333333, ans=0.09899494936611666 2023-11-22 03:47:28,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.20 vs. limit=22.5 2023-11-22 03:47:31,597 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267500 2023-11-22 03:47:34,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1783340.0, ans=0.125 2023-11-22 03:47:42,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1783340.0, ans=0.0 2023-11-22 03:47:57,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1783406.6666666667, ans=0.0 2023-11-22 03:47:58,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1783406.6666666667, ans=0.09899494936611666 2023-11-22 03:48:01,457 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3000, loss[loss=0.05031, simple_loss=0.06056, pruned_loss=0.01122, audio_tagging_loss=0.008809, over 17157.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09464, pruned_loss=0.01565, audio_tagging_loss=0.009488, over 3032474.57 frames. ], batch size: 68, lr: 2.99e-03, grad_scale: 16.0 2023-11-22 03:48:01,458 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 03:48:23,590 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.7773, 1.3992, 3.5447, 3.1696, 3.0174, 3.2832, 2.8231, 3.2791], device='cuda:3') 2023-11-22 03:48:28,367 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.0361, 3.0030, 3.4170, 2.9630, 3.7790, 3.7531, 3.3289, 3.1943], device='cuda:3') 2023-11-22 03:48:41,392 INFO [train_asr.py:1253] (3/4) Epoch 23, validation: loss=0.05946, simple_loss=0.05181, pruned_loss=0.005129, audio_tagging_loss=0.02843, over 4681554.00 frames. 2023-11-22 03:48:41,393 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 03:49:02,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1783540.0, ans=0.0 2023-11-22 03:49:07,729 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.252e+01 8.878e+01 9.473e+01 1.263e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-22 03:49:12,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1783606.6666666667, ans=0.2 2023-11-22 03:49:15,790 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267550 2023-11-22 03:49:16,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1783606.6666666667, ans=10.0 2023-11-22 03:49:44,976 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3050, loss[loss=0.07462, simple_loss=0.1014, pruned_loss=0.01471, audio_tagging_loss=0.009204, over 15786.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09478, pruned_loss=0.01576, audio_tagging_loss=0.009547, over 3030983.98 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 8.0 2023-11-22 03:49:59,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1783873.3333333333, ans=0.05 2023-11-22 03:49:59,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.72 vs. limit=15.0 2023-11-22 03:50:20,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267600 2023-11-22 03:50:23,291 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:50:23,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1784006.6666666667, ans=0.125 2023-11-22 03:50:32,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1784006.6666666667, ans=0.125 2023-11-22 03:50:48,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1784140.0, ans=0.1 2023-11-22 03:50:49,184 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3100, loss[loss=0.06298, simple_loss=0.07795, pruned_loss=0.0121, audio_tagging_loss=0.0119, over 15218.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.0945, pruned_loss=0.01568, audio_tagging_loss=0.009582, over 3027663.22 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 8.0 2023-11-22 03:51:14,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1784273.3333333333, ans=0.125 2023-11-22 03:51:17,603 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.039e+01 8.127e+01 8.791e+01 9.439e+01 1.386e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 03:51:23,757 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267650 2023-11-22 03:51:35,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1784340.0, ans=0.1 2023-11-22 03:51:44,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1784406.6666666667, ans=0.0 2023-11-22 03:51:53,227 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3150, loss[loss=0.08885, simple_loss=0.1208, pruned_loss=0.0199, audio_tagging_loss=0.00852, over 14819.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09586, pruned_loss=0.01588, audio_tagging_loss=0.009554, over 3028812.93 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 8.0 2023-11-22 03:51:53,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1784473.3333333333, ans=0.125 2023-11-22 03:52:13,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1784540.0, ans=0.125 2023-11-22 03:52:17,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1784606.6666666667, ans=0.125 2023-11-22 03:52:27,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267700 2023-11-22 03:52:57,570 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3200, loss[loss=0.0612, simple_loss=0.08032, pruned_loss=0.01109, audio_tagging_loss=0.00995, over 15875.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09569, pruned_loss=0.01591, audio_tagging_loss=0.00965, over 3029401.34 frames. ], batch size: 61, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 03:53:01,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1784806.6666666667, ans=0.05 2023-11-22 03:53:14,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1784873.3333333333, ans=0.2 2023-11-22 03:53:26,398 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.278e+01 8.640e+01 9.610e+01 1.233e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-22 03:53:33,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267750 2023-11-22 03:53:33,847 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.14 vs. limit=10.0 2023-11-22 03:54:01,748 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3250, loss[loss=0.06311, simple_loss=0.07678, pruned_loss=0.0127, audio_tagging_loss=0.01202, over 14514.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09485, pruned_loss=0.01584, audio_tagging_loss=0.00973, over 3033997.80 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 03:54:05,856 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0 2023-11-22 03:54:10,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1785140.0, ans=0.125 2023-11-22 03:54:18,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1785206.6666666667, ans=0.025 2023-11-22 03:54:18,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1785206.6666666667, ans=0.0 2023-11-22 03:54:35,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267800 2023-11-22 03:54:54,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1785406.6666666667, ans=0.125 2023-11-22 03:55:06,135 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3300, loss[loss=0.08589, simple_loss=0.1132, pruned_loss=0.01965, audio_tagging_loss=0.00964, over 15752.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09442, pruned_loss=0.01572, audio_tagging_loss=0.009839, over 3040872.50 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 03:55:18,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1785540.0, ans=0.2 2023-11-22 03:55:25,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1785540.0, ans=0.2 2023-11-22 03:55:27,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1785540.0, ans=0.0 2023-11-22 03:55:34,164 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.246e+01 8.906e+01 9.569e+01 1.172e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-22 03:55:40,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267850 2023-11-22 03:55:43,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1785673.3333333333, ans=0.125 2023-11-22 03:55:45,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1785673.3333333333, ans=0.125 2023-11-22 03:55:46,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1785673.3333333333, ans=0.05 2023-11-22 03:56:04,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1785740.0, ans=0.125 2023-11-22 03:56:09,856 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3350, loss[loss=0.06737, simple_loss=0.08561, pruned_loss=0.01662, audio_tagging_loss=0.007945, over 14752.00 frames. ], tot_loss[loss=0.07319, simple_loss=0.09501, pruned_loss=0.01595, audio_tagging_loss=0.009739, over 3038498.43 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 03:56:16,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1785806.6666666667, ans=0.125 2023-11-22 03:56:18,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.07 vs. limit=22.5 2023-11-22 03:56:44,845 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267900 2023-11-22 03:57:11,613 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:57:13,702 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3400, loss[loss=0.07189, simple_loss=0.09628, pruned_loss=0.01527, audio_tagging_loss=0.008477, over 16241.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09616, pruned_loss=0.01608, audio_tagging_loss=0.009546, over 3045227.60 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 03:57:20,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1786140.0, ans=0.0 2023-11-22 03:57:23,783 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:57:32,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1786206.6666666667, ans=0.09899494936611666 2023-11-22 03:57:42,474 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.335e+01 8.871e+01 9.703e+01 1.398e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 03:57:48,661 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 267950 2023-11-22 03:57:48,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1786273.3333333333, ans=0.2 2023-11-22 03:57:54,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1786340.0, ans=0.025 2023-11-22 03:57:57,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1786340.0, ans=0.125 2023-11-22 03:58:00,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1786340.0, ans=0.0 2023-11-22 03:58:00,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1786340.0, ans=0.0 2023-11-22 03:58:06,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786406.6666666667, ans=0.1 2023-11-22 03:58:10,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1786406.6666666667, ans=0.1 2023-11-22 03:58:11,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1786406.6666666667, ans=0.025 2023-11-22 03:58:18,136 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3450, loss[loss=0.07926, simple_loss=0.09358, pruned_loss=0.02058, audio_tagging_loss=0.01189, over 15509.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09598, pruned_loss=0.01609, audio_tagging_loss=0.009441, over 3047403.09 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 03:58:29,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1786473.3333333333, ans=0.125 2023-11-22 03:58:41,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1786540.0, ans=0.125 2023-11-22 03:58:41,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1786540.0, ans=0.125 2023-11-22 03:58:46,870 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 03:58:53,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268000 2023-11-22 03:59:00,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1786673.3333333333, ans=0.05 2023-11-22 03:59:08,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1786673.3333333333, ans=0.0 2023-11-22 03:59:14,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.39 vs. limit=15.0 2023-11-22 03:59:22,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1786740.0, ans=0.0 2023-11-22 03:59:25,612 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3500, loss[loss=0.07679, simple_loss=0.1022, pruned_loss=0.01631, audio_tagging_loss=0.009388, over 15789.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09539, pruned_loss=0.01596, audio_tagging_loss=0.009382, over 3047433.62 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 03:59:28,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786806.6666666667, ans=0.1 2023-11-22 03:59:29,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786806.6666666667, ans=0.1 2023-11-22 03:59:43,978 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.85 vs. limit=15.0 2023-11-22 03:59:54,062 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.643e+01 7.936e+01 8.648e+01 9.422e+01 1.502e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-22 03:59:58,408 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 03:59:59,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268050 2023-11-22 04:00:01,405 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.92 vs. limit=12.0 2023-11-22 04:00:15,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.83 vs. limit=22.5 2023-11-22 04:00:29,147 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3550, loss[loss=0.06578, simple_loss=0.08072, pruned_loss=0.01609, audio_tagging_loss=0.009324, over 15009.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09525, pruned_loss=0.01587, audio_tagging_loss=0.00937, over 3048372.66 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:00:40,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1787206.6666666667, ans=0.125 2023-11-22 04:00:53,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1787273.3333333333, ans=0.0 2023-11-22 04:01:03,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268100 2023-11-22 04:01:04,424 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.68 vs. limit=15.0 2023-11-22 04:01:12,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=1787340.0, ans=0.1 2023-11-22 04:01:12,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=1787340.0, ans=0.2 2023-11-22 04:01:13,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1787340.0, ans=0.125 2023-11-22 04:01:16,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1787340.0, ans=0.125 2023-11-22 04:01:24,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=1787406.6666666667, ans=0.05 2023-11-22 04:01:25,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1787406.6666666667, ans=0.2 2023-11-22 04:01:29,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1787406.6666666667, ans=0.125 2023-11-22 04:01:31,968 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3600, loss[loss=0.07282, simple_loss=0.08954, pruned_loss=0.01785, audio_tagging_loss=0.01021, over 14340.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09455, pruned_loss=0.01573, audio_tagging_loss=0.009368, over 3046652.60 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:01:35,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1787473.3333333333, ans=0.0 2023-11-22 04:01:39,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1787473.3333333333, ans=0.125 2023-11-22 04:01:43,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1787473.3333333333, ans=0.125 2023-11-22 04:01:59,855 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 04:02:01,927 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.327e+01 8.799e+01 9.625e+01 1.533e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 04:02:07,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268150 2023-11-22 04:02:26,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1787740.0, ans=0.125 2023-11-22 04:02:37,499 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3650, loss[loss=0.04062, simple_loss=0.04684, pruned_loss=0.006859, audio_tagging_loss=0.01034, over 15969.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09511, pruned_loss=0.01571, audio_tagging_loss=0.009264, over 3057855.91 frames. ], batch size: 64, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:02:37,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1787806.6666666667, ans=0.125 2023-11-22 04:02:42,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1787806.6666666667, ans=0.2 2023-11-22 04:02:46,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.00 vs. limit=22.5 2023-11-22 04:03:07,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1787940.0, ans=0.2 2023-11-22 04:03:11,593 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268200 2023-11-22 04:03:19,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1788006.6666666667, ans=0.125 2023-11-22 04:03:30,211 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.94 vs. limit=22.5 2023-11-22 04:03:40,415 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3700, loss[loss=0.05951, simple_loss=0.07503, pruned_loss=0.01335, audio_tagging_loss=0.008648, over 15245.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09509, pruned_loss=0.0157, audio_tagging_loss=0.009173, over 3058967.61 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:03:50,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1788140.0, ans=0.0 2023-11-22 04:03:52,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1788206.6666666667, ans=0.125 2023-11-22 04:04:09,834 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.401e+01 8.835e+01 9.477e+01 1.225e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 04:04:15,633 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268250 2023-11-22 04:04:19,906 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.08 vs. limit=15.0 2023-11-22 04:04:32,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1788406.6666666667, ans=0.0 2023-11-22 04:04:43,993 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3750, loss[loss=0.0812, simple_loss=0.1115, pruned_loss=0.01657, audio_tagging_loss=0.008891, over 15507.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09551, pruned_loss=0.01577, audio_tagging_loss=0.009202, over 3053215.94 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:04:53,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1788473.3333333333, ans=0.1 2023-11-22 04:05:13,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1788606.6666666667, ans=0.1 2023-11-22 04:05:18,524 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268300 2023-11-22 04:05:28,621 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 04:05:45,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1788740.0, ans=0.0 2023-11-22 04:05:48,207 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3800, loss[loss=0.07173, simple_loss=0.08649, pruned_loss=0.01737, audio_tagging_loss=0.01112, over 14184.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09608, pruned_loss=0.01594, audio_tagging_loss=0.009213, over 3052597.27 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:05:54,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1788806.6666666667, ans=0.1 2023-11-22 04:06:18,646 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.152e+01 8.764e+01 9.512e+01 1.350e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-22 04:06:22,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268350 2023-11-22 04:06:25,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1789006.6666666667, ans=0.1 2023-11-22 04:06:37,723 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.73 vs. limit=15.0 2023-11-22 04:06:52,275 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3850, loss[loss=0.06891, simple_loss=0.09058, pruned_loss=0.01326, audio_tagging_loss=0.01036, over 14179.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09554, pruned_loss=0.01587, audio_tagging_loss=0.009287, over 3050594.74 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:06:52,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1789140.0, ans=0.0 2023-11-22 04:06:55,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1789140.0, ans=0.125 2023-11-22 04:06:58,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1789140.0, ans=0.125 2023-11-22 04:07:09,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1789206.6666666667, ans=0.0 2023-11-22 04:07:19,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1789273.3333333333, ans=0.0 2023-11-22 04:07:27,379 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268400 2023-11-22 04:07:32,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1789340.0, ans=0.125 2023-11-22 04:07:34,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1789340.0, ans=0.2 2023-11-22 04:07:35,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1789340.0, ans=0.1 2023-11-22 04:07:57,361 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3900, loss[loss=0.06208, simple_loss=0.06527, pruned_loss=0.01302, audio_tagging_loss=0.01643, over 14012.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09513, pruned_loss=0.01589, audio_tagging_loss=0.009367, over 3050368.57 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:08:07,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1789473.3333333333, ans=0.2 2023-11-22 04:08:20,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.41 vs. limit=15.0 2023-11-22 04:08:28,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 8.044e+01 8.875e+01 9.539e+01 1.176e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 04:08:32,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268450 2023-11-22 04:08:39,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1789673.3333333333, ans=0.025 2023-11-22 04:08:40,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff2.min_abs, batch_count=1789673.3333333333, ans=0.1 2023-11-22 04:08:41,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1789673.3333333333, ans=0.035 2023-11-22 04:08:46,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.67 vs. limit=10.0 2023-11-22 04:08:50,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1789740.0, ans=0.125 2023-11-22 04:09:01,975 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 3950, loss[loss=0.0721, simple_loss=0.09584, pruned_loss=0.01277, audio_tagging_loss=0.01141, over 14493.00 frames. ], tot_loss[loss=0.07369, simple_loss=0.09646, pruned_loss=0.01609, audio_tagging_loss=0.009361, over 3047322.10 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:09:03,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1789806.6666666667, ans=0.0 2023-11-22 04:09:35,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268500 2023-11-22 04:09:36,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1789940.0, ans=0.125 2023-11-22 04:09:37,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1789940.0, ans=0.0 2023-11-22 04:09:37,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1789940.0, ans=0.95 2023-11-22 04:09:45,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1790006.6666666667, ans=0.2 2023-11-22 04:09:59,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1790073.3333333333, ans=0.125 2023-11-22 04:10:04,737 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4000, loss[loss=0.06724, simple_loss=0.08457, pruned_loss=0.01305, audio_tagging_loss=0.0119, over 15459.00 frames. ], tot_loss[loss=0.07391, simple_loss=0.09644, pruned_loss=0.01623, audio_tagging_loss=0.009459, over 3045892.46 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:10:14,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1790140.0, ans=0.125 2023-11-22 04:10:14,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1790140.0, ans=0.07 2023-11-22 04:10:29,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1790273.3333333333, ans=0.125 2023-11-22 04:10:36,041 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.201e+01 8.416e+01 8.758e+01 9.509e+01 1.185e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-22 04:10:37,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1790273.3333333333, ans=0.0 2023-11-22 04:10:40,299 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268550 2023-11-22 04:11:09,604 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4050, loss[loss=0.05999, simple_loss=0.07424, pruned_loss=0.01368, audio_tagging_loss=0.009189, over 14438.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09704, pruned_loss=0.01636, audio_tagging_loss=0.009403, over 3045232.56 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:11:12,192 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 04:11:31,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1790540.0, ans=0.125 2023-11-22 04:11:43,693 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268600 2023-11-22 04:11:51,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1790673.3333333333, ans=10.0 2023-11-22 04:12:10,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1790740.0, ans=0.0 2023-11-22 04:12:13,286 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4100, loss[loss=0.05177, simple_loss=0.06854, pruned_loss=0.008603, audio_tagging_loss=0.008895, over 15971.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09662, pruned_loss=0.01621, audio_tagging_loss=0.009389, over 3045552.47 frames. ], batch size: 61, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:12:35,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1790873.3333333333, ans=0.125 2023-11-22 04:12:44,098 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.196e+01 8.304e+01 9.033e+01 9.578e+01 1.202e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-22 04:12:47,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268650 2023-11-22 04:13:14,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.21 vs. limit=15.0 2023-11-22 04:13:17,310 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4150, loss[loss=0.05518, simple_loss=0.0746, pruned_loss=0.009577, audio_tagging_loss=0.008308, over 14423.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09506, pruned_loss=0.01585, audio_tagging_loss=0.009336, over 3042349.57 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:13:30,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1791206.6666666667, ans=0.2 2023-11-22 04:13:33,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1791206.6666666667, ans=0.125 2023-11-22 04:13:38,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1791206.6666666667, ans=0.1 2023-11-22 04:13:48,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1791273.3333333333, ans=0.125 2023-11-22 04:13:52,785 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268700 2023-11-22 04:14:02,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1791340.0, ans=0.125 2023-11-22 04:14:03,575 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 04:14:19,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1791406.6666666667, ans=0.125 2023-11-22 04:14:19,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1791406.6666666667, ans=0.125 2023-11-22 04:14:21,875 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4200, loss[loss=0.07371, simple_loss=0.09605, pruned_loss=0.01748, audio_tagging_loss=0.008207, over 14421.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09603, pruned_loss=0.01602, audio_tagging_loss=0.009248, over 3037685.95 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:14:31,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1791473.3333333333, ans=0.125 2023-11-22 04:14:31,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1791473.3333333333, ans=0.1 2023-11-22 04:14:52,145 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.143e+01 8.798e+01 9.723e+01 1.732e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 04:14:53,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.whiten.whitening_limit, batch_count=1791606.6666666667, ans=12.0 2023-11-22 04:14:54,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1791606.6666666667, ans=0.1 2023-11-22 04:14:55,984 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268750 2023-11-22 04:15:06,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1791673.3333333333, ans=0.025 2023-11-22 04:15:07,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1791673.3333333333, ans=0.2 2023-11-22 04:15:15,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1791740.0, ans=6.0 2023-11-22 04:15:25,326 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.44 vs. limit=15.0 2023-11-22 04:15:25,781 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4250, loss[loss=0.06322, simple_loss=0.07089, pruned_loss=0.01581, audio_tagging_loss=0.01196, over 15140.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09581, pruned_loss=0.0157, audio_tagging_loss=0.009227, over 3037691.54 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:15:27,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=15.0 2023-11-22 04:15:28,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1791806.6666666667, ans=0.0 2023-11-22 04:15:35,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1791806.6666666667, ans=0.125 2023-11-22 04:15:55,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1791940.0, ans=0.0 2023-11-22 04:16:00,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268800 2023-11-22 04:16:08,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1792006.6666666667, ans=0.125 2023-11-22 04:16:27,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1792073.3333333333, ans=0.0 2023-11-22 04:16:30,660 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4300, loss[loss=0.0961, simple_loss=0.1272, pruned_loss=0.02576, audio_tagging_loss=0.006737, over 16720.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09791, pruned_loss=0.0161, audio_tagging_loss=0.009039, over 3044992.50 frames. ], batch size: 62, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:16:32,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.76 vs. limit=15.0 2023-11-22 04:16:58,056 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 04:17:01,432 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.380e+01 9.006e+01 9.846e+01 1.214e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-22 04:17:05,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268850 2023-11-22 04:17:06,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1792273.3333333333, ans=0.125 2023-11-22 04:17:35,222 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4350, loss[loss=0.07441, simple_loss=0.09692, pruned_loss=0.01828, audio_tagging_loss=0.007667, over 15321.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09725, pruned_loss=0.01592, audio_tagging_loss=0.009019, over 3036610.63 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:17:52,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1792540.0, ans=0.0 2023-11-22 04:17:58,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1792540.0, ans=0.2 2023-11-22 04:18:05,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1792606.6666666667, ans=0.125 2023-11-22 04:18:10,627 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268900 2023-11-22 04:18:22,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.74 vs. limit=12.0 2023-11-22 04:18:39,824 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4400, loss[loss=0.06526, simple_loss=0.08837, pruned_loss=0.01254, audio_tagging_loss=0.008539, over 15125.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09574, pruned_loss=0.01559, audio_tagging_loss=0.009114, over 3046047.61 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 32.0 2023-11-22 04:19:10,793 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 7.984e+01 8.605e+01 9.395e+01 1.319e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-22 04:19:14,612 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 268950 2023-11-22 04:19:33,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1793073.3333333333, ans=0.125 2023-11-22 04:19:36,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1793073.3333333333, ans=0.0 2023-11-22 04:19:37,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1793073.3333333333, ans=0.95 2023-11-22 04:19:45,070 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4450, loss[loss=0.05959, simple_loss=0.07657, pruned_loss=0.01281, audio_tagging_loss=0.008489, over 14082.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09539, pruned_loss=0.01556, audio_tagging_loss=0.009124, over 3042537.53 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 32.0 2023-11-22 04:19:51,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1793140.0, ans=0.1 2023-11-22 04:19:53,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1793140.0, ans=0.1 2023-11-22 04:19:58,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1793206.6666666667, ans=0.125 2023-11-22 04:19:58,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1793206.6666666667, ans=0.1 2023-11-22 04:20:19,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269000 2023-11-22 04:20:21,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0 2023-11-22 04:20:27,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1793340.0, ans=0.0 2023-11-22 04:20:28,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1793340.0, ans=0.0 2023-11-22 04:20:49,442 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4500, loss[loss=0.06156, simple_loss=0.07591, pruned_loss=0.01235, audio_tagging_loss=0.01126, over 17432.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09493, pruned_loss=0.01559, audio_tagging_loss=0.009159, over 3046160.62 frames. ], batch size: 66, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:21:08,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1793540.0, ans=0.125 2023-11-22 04:21:19,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1793606.6666666667, ans=0.2 2023-11-22 04:21:23,189 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.138e+01 8.864e+01 9.705e+01 1.342e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-22 04:21:24,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269050 2023-11-22 04:21:33,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1793673.3333333333, ans=0.0 2023-11-22 04:21:44,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1793740.0, ans=0.125 2023-11-22 04:21:51,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1793806.6666666667, ans=0.0 2023-11-22 04:21:52,983 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4550, loss[loss=0.05839, simple_loss=0.07783, pruned_loss=0.008749, audio_tagging_loss=0.01073, over 16044.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09424, pruned_loss=0.01549, audio_tagging_loss=0.009189, over 3040957.31 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:22:04,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1793806.6666666667, ans=0.1 2023-11-22 04:22:28,114 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269100 2023-11-22 04:22:30,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1794006.6666666667, ans=0.0 2023-11-22 04:22:41,456 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 04:22:57,695 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4600, loss[loss=0.06026, simple_loss=0.07298, pruned_loss=0.01418, audio_tagging_loss=0.009593, over 15375.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09452, pruned_loss=0.01564, audio_tagging_loss=0.009256, over 3046377.52 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:23:09,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1794206.6666666667, ans=0.0 2023-11-22 04:23:10,765 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.04 vs. limit=15.0 2023-11-22 04:23:19,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1794206.6666666667, ans=0.1 2023-11-22 04:23:19,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.20 vs. limit=15.0 2023-11-22 04:23:30,460 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.306e+01 8.737e+01 9.544e+01 1.284e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 04:23:31,788 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269150 2023-11-22 04:24:01,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1794473.3333333333, ans=0.125 2023-11-22 04:24:02,027 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4650, loss[loss=0.07402, simple_loss=0.1032, pruned_loss=0.01351, audio_tagging_loss=0.008929, over 14722.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09459, pruned_loss=0.01548, audio_tagging_loss=0.009365, over 3043013.08 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:24:11,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1794473.3333333333, ans=0.07 2023-11-22 04:24:13,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1794540.0, ans=0.125 2023-11-22 04:24:16,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1794540.0, ans=0.1 2023-11-22 04:24:23,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1794540.0, ans=0.125 2023-11-22 04:24:23,497 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.66 vs. limit=6.0 2023-11-22 04:24:25,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1794540.0, ans=0.95 2023-11-22 04:24:29,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0 2023-11-22 04:24:31,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1794606.6666666667, ans=0.0 2023-11-22 04:24:37,457 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269200 2023-11-22 04:24:37,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1794606.6666666667, ans=0.1 2023-11-22 04:25:06,243 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4700, loss[loss=0.06159, simple_loss=0.07781, pruned_loss=0.01314, audio_tagging_loss=0.009544, over 14108.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09422, pruned_loss=0.01538, audio_tagging_loss=0.009479, over 3038871.08 frames. ], batch size: 52, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:25:06,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.60 vs. limit=6.0 2023-11-22 04:25:31,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1794940.0, ans=0.1 2023-11-22 04:25:39,201 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.243e+01 8.677e+01 9.667e+01 1.211e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-22 04:25:40,538 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269250 2023-11-22 04:26:09,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.34 vs. limit=10.0 2023-11-22 04:26:11,055 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4750, loss[loss=0.04886, simple_loss=0.05817, pruned_loss=0.00755, audio_tagging_loss=0.01222, over 14725.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09368, pruned_loss=0.01529, audio_tagging_loss=0.009629, over 3033779.53 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0 2023-11-22 04:26:12,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1795140.0, ans=0.1 2023-11-22 04:26:13,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1795140.0, ans=0.125 2023-11-22 04:26:19,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1795140.0, ans=0.2 2023-11-22 04:26:25,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1795206.6666666667, ans=0.125 2023-11-22 04:26:35,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.49 vs. limit=15.0 2023-11-22 04:26:41,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1795273.3333333333, ans=0.125 2023-11-22 04:26:44,540 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269300 2023-11-22 04:26:55,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1795340.0, ans=0.1 2023-11-22 04:26:59,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1795340.0, ans=0.125 2023-11-22 04:27:11,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1795406.6666666667, ans=0.125 2023-11-22 04:27:14,475 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4800, loss[loss=0.08292, simple_loss=0.1129, pruned_loss=0.01608, audio_tagging_loss=0.01039, over 15921.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.0932, pruned_loss=0.01524, audio_tagging_loss=0.009662, over 3044969.01 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:27:18,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1795473.3333333333, ans=0.0 2023-11-22 04:27:27,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1795540.0, ans=0.2 2023-11-22 04:27:47,742 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.253e+01 9.179e+01 1.004e+02 1.459e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-22 04:27:49,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269350 2023-11-22 04:27:57,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1795673.3333333333, ans=0.0 2023-11-22 04:28:17,846 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4850, loss[loss=0.06382, simple_loss=0.07912, pruned_loss=0.01218, audio_tagging_loss=0.01209, over 14524.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09255, pruned_loss=0.01509, audio_tagging_loss=0.009733, over 3047025.35 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:28:39,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1795873.3333333333, ans=0.125 2023-11-22 04:28:52,326 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269400 2023-11-22 04:29:21,635 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4900, loss[loss=0.0716, simple_loss=0.09022, pruned_loss=0.01923, audio_tagging_loss=0.007254, over 15212.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09397, pruned_loss=0.01546, audio_tagging_loss=0.009601, over 3044651.11 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:29:25,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1796140.0, ans=0.125 2023-11-22 04:29:28,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1796140.0, ans=0.5 2023-11-22 04:29:33,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1796140.0, ans=0.07 2023-11-22 04:29:34,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1796206.6666666667, ans=0.1 2023-11-22 04:29:55,274 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 7.981e+01 8.746e+01 9.394e+01 1.160e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 04:29:56,614 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269450 2023-11-22 04:30:09,895 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 04:30:11,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1796340.0, ans=0.0 2023-11-22 04:30:26,658 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 4950, loss[loss=0.1061, simple_loss=0.139, pruned_loss=0.02893, audio_tagging_loss=0.007682, over 16024.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09444, pruned_loss=0.01558, audio_tagging_loss=0.009444, over 3041654.66 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0 2023-11-22 04:30:58,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1796606.6666666667, ans=0.1 2023-11-22 04:31:01,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269500 2023-11-22 04:31:10,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1796673.3333333333, ans=0.1 2023-11-22 04:31:19,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1796740.0, ans=0.125 2023-11-22 04:31:21,105 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.23 vs. limit=15.0 2023-11-22 04:31:30,697 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5000, loss[loss=0.07681, simple_loss=0.09589, pruned_loss=0.02005, audio_tagging_loss=0.00882, over 15512.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09364, pruned_loss=0.01525, audio_tagging_loss=0.009406, over 3046705.06 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:31:31,364 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.17 vs. limit=15.0 2023-11-22 04:31:44,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1796873.3333333333, ans=0.0 2023-11-22 04:32:04,489 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.014e+01 8.549e+01 9.215e+01 1.274e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-22 04:32:05,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269550 2023-11-22 04:32:12,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0 2023-11-22 04:32:13,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1797006.6666666667, ans=0.1 2023-11-22 04:32:16,505 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 04:32:33,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1797073.3333333333, ans=0.125 2023-11-22 04:32:33,535 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.81 vs. limit=15.0 2023-11-22 04:32:35,460 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5050, loss[loss=0.06736, simple_loss=0.07934, pruned_loss=0.01903, audio_tagging_loss=0.008661, over 14537.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09255, pruned_loss=0.01509, audio_tagging_loss=0.009396, over 3044943.17 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:32:48,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1797206.6666666667, ans=0.1 2023-11-22 04:33:00,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1797273.3333333333, ans=0.1 2023-11-22 04:33:03,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1797273.3333333333, ans=0.2 2023-11-22 04:33:10,965 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269600 2023-11-22 04:33:24,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1797340.0, ans=0.125 2023-11-22 04:33:35,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1797406.6666666667, ans=0.2 2023-11-22 04:33:40,606 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5100, loss[loss=0.06613, simple_loss=0.08351, pruned_loss=0.0139, audio_tagging_loss=0.01047, over 15960.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09187, pruned_loss=0.01492, audio_tagging_loss=0.009381, over 3043318.48 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:33:47,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1797473.3333333333, ans=0.0 2023-11-22 04:34:14,415 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.728e+01 7.951e+01 8.483e+01 9.214e+01 1.214e+02, threshold=1.697e+02, percent-clipped=0.0 2023-11-22 04:34:15,792 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269650 2023-11-22 04:34:38,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1797740.0, ans=0.125 2023-11-22 04:34:44,040 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.35 vs. limit=22.5 2023-11-22 04:34:45,426 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.57 vs. limit=12.0 2023-11-22 04:34:45,989 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5150, loss[loss=0.07921, simple_loss=0.1048, pruned_loss=0.01785, audio_tagging_loss=0.008968, over 15612.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09211, pruned_loss=0.01494, audio_tagging_loss=0.00941, over 3050194.14 frames. ], batch size: 59, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:34:46,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1797806.6666666667, ans=0.2 2023-11-22 04:34:49,145 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0 2023-11-22 04:34:51,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1797806.6666666667, ans=0.125 2023-11-22 04:34:56,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1797806.6666666667, ans=0.1 2023-11-22 04:35:06,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1797873.3333333333, ans=0.05 2023-11-22 04:35:17,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1797940.0, ans=0.95 2023-11-22 04:35:20,607 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269700 2023-11-22 04:35:20,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1797940.0, ans=0.125 2023-11-22 04:35:38,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1798073.3333333333, ans=0.2 2023-11-22 04:35:50,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1798140.0, ans=0.125 2023-11-22 04:35:51,090 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5200, loss[loss=0.06006, simple_loss=0.06844, pruned_loss=0.01401, audio_tagging_loss=0.01182, over 14582.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09251, pruned_loss=0.01505, audio_tagging_loss=0.009409, over 3039815.15 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 04:35:52,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1798140.0, ans=0.0 2023-11-22 04:36:24,909 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.234e+01 8.647e+01 9.394e+01 1.220e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-22 04:36:26,265 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269750 2023-11-22 04:36:55,991 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5250, loss[loss=0.09253, simple_loss=0.1269, pruned_loss=0.02166, audio_tagging_loss=0.007411, over 14494.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09359, pruned_loss=0.01514, audio_tagging_loss=0.009357, over 3038901.41 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 04:37:02,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1798473.3333333333, ans=0.125 2023-11-22 04:37:15,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1798540.0, ans=15.0 2023-11-22 04:37:24,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1798606.6666666667, ans=0.0 2023-11-22 04:37:27,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1798606.6666666667, ans=0.125 2023-11-22 04:37:30,763 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269800 2023-11-22 04:37:35,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.81 vs. limit=15.0 2023-11-22 04:37:50,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.06 vs. limit=15.0 2023-11-22 04:37:51,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1798740.0, ans=0.1 2023-11-22 04:37:57,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=1798740.0, ans=15.0 2023-11-22 04:38:00,460 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5300, loss[loss=0.0652, simple_loss=0.08288, pruned_loss=0.01178, audio_tagging_loss=0.01198, over 15190.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09452, pruned_loss=0.01551, audio_tagging_loss=0.009365, over 3039755.64 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 04:38:18,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1798873.3333333333, ans=0.0 2023-11-22 04:38:34,770 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.362e+01 8.825e+01 9.313e+01 1.444e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-22 04:38:34,914 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269850 2023-11-22 04:38:55,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1799073.3333333333, ans=0.125 2023-11-22 04:39:04,229 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5350, loss[loss=0.06161, simple_loss=0.08318, pruned_loss=0.01199, audio_tagging_loss=0.008027, over 15375.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09448, pruned_loss=0.01542, audio_tagging_loss=0.009302, over 3037262.59 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:39:39,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269900 2023-11-22 04:39:58,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1799406.6666666667, ans=0.2 2023-11-22 04:40:09,604 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5400, loss[loss=0.06844, simple_loss=0.0906, pruned_loss=0.009308, audio_tagging_loss=0.01383, over 15345.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09505, pruned_loss=0.01557, audio_tagging_loss=0.009257, over 3036218.95 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:40:09,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1799473.3333333333, ans=0.0 2023-11-22 04:40:17,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1799473.3333333333, ans=0.1 2023-11-22 04:40:22,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.51 vs. limit=12.0 2023-11-22 04:40:31,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1799540.0, ans=0.125 2023-11-22 04:40:37,230 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.05 vs. limit=15.0 2023-11-22 04:40:40,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=22.5 2023-11-22 04:40:43,901 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.520e+01 8.148e+01 8.900e+01 9.649e+01 1.296e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-22 04:40:44,696 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 269950 2023-11-22 04:40:59,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_na.min_abs, batch_count=1799673.3333333333, ans=0.02 2023-11-22 04:41:01,911 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.50 vs. limit=15.0 2023-11-22 04:41:02,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1799740.0, ans=0.0 2023-11-22 04:41:06,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1799740.0, ans=0.2 2023-11-22 04:41:11,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1799740.0, ans=0.125 2023-11-22 04:41:13,903 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5450, loss[loss=0.08019, simple_loss=0.1004, pruned_loss=0.01842, audio_tagging_loss=0.01157, over 15658.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09556, pruned_loss=0.01571, audio_tagging_loss=0.009307, over 3032981.38 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:41:46,285 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 04:41:49,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270000 2023-11-22 04:41:56,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1800006.6666666667, ans=0.0 2023-11-22 04:41:59,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1800006.6666666667, ans=0.125 2023-11-22 04:42:02,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1800006.6666666667, ans=0.1 2023-11-22 04:42:14,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1800073.3333333333, ans=0.0 2023-11-22 04:42:17,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0 2023-11-22 04:42:18,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1800140.0, ans=0.125 2023-11-22 04:42:19,713 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5500, loss[loss=0.06001, simple_loss=0.07297, pruned_loss=0.01054, audio_tagging_loss=0.01298, over 15625.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.09511, pruned_loss=0.01548, audio_tagging_loss=0.009354, over 3038005.94 frames. ], batch size: 61, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:42:29,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1800140.0, ans=0.125 2023-11-22 04:42:30,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1800140.0, ans=0.95 2023-11-22 04:42:53,708 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.241e+01 8.879e+01 9.729e+01 1.307e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-22 04:42:53,850 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270050 2023-11-22 04:43:05,508 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.23 vs. limit=15.0 2023-11-22 04:43:24,372 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5550, loss[loss=0.07981, simple_loss=0.1036, pruned_loss=0.02074, audio_tagging_loss=0.007261, over 15273.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09444, pruned_loss=0.01546, audio_tagging_loss=0.009416, over 3044130.79 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:43:59,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270100 2023-11-22 04:44:20,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1800740.0, ans=0.0 2023-11-22 04:44:28,577 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5600, loss[loss=0.09469, simple_loss=0.1296, pruned_loss=0.02233, audio_tagging_loss=0.007565, over 15081.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09417, pruned_loss=0.0153, audio_tagging_loss=0.009507, over 3047219.37 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 04:44:44,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1800873.3333333333, ans=0.0 2023-11-22 04:44:58,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1800940.0, ans=0.2 2023-11-22 04:45:04,396 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.061e+01 8.801e+01 9.492e+01 1.565e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 04:45:04,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270150 2023-11-22 04:45:15,356 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 04:45:33,026 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5650, loss[loss=0.05495, simple_loss=0.07029, pruned_loss=0.009832, audio_tagging_loss=0.009973, over 14935.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.09474, pruned_loss=0.0155, audio_tagging_loss=0.009599, over 3056289.19 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:45:50,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1801206.6666666667, ans=0.125 2023-11-22 04:46:03,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1801273.3333333333, ans=0.0 2023-11-22 04:46:07,662 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270200 2023-11-22 04:46:37,706 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5700, loss[loss=0.07831, simple_loss=0.1122, pruned_loss=0.01311, audio_tagging_loss=0.00912, over 15146.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09417, pruned_loss=0.01529, audio_tagging_loss=0.009674, over 3049904.48 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:46:49,743 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.08 vs. limit=15.0 2023-11-22 04:46:53,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1801540.0, ans=0.09899494936611666 2023-11-22 04:46:58,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1801540.0, ans=0.1 2023-11-22 04:47:11,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1801606.6666666667, ans=0.2 2023-11-22 04:47:12,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270250 2023-11-22 04:47:12,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1801606.6666666667, ans=0.1 2023-11-22 04:47:13,227 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.286e+01 8.958e+01 9.545e+01 1.493e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-22 04:47:14,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1801673.3333333333, ans=0.125 2023-11-22 04:47:21,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1801673.3333333333, ans=0.125 2023-11-22 04:47:38,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1801740.0, ans=0.1 2023-11-22 04:47:41,458 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5750, loss[loss=0.06104, simple_loss=0.07183, pruned_loss=0.014, audio_tagging_loss=0.01113, over 14037.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09445, pruned_loss=0.0155, audio_tagging_loss=0.009488, over 3047467.87 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:47:45,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1801806.6666666667, ans=0.0 2023-11-22 04:47:48,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1801806.6666666667, ans=0.0 2023-11-22 04:47:49,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1801806.6666666667, ans=0.125 2023-11-22 04:48:09,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1801940.0, ans=10.0 2023-11-22 04:48:11,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1801940.0, ans=0.0 2023-11-22 04:48:15,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1801940.0, ans=0.2 2023-11-22 04:48:16,412 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270300 2023-11-22 04:48:19,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1802006.6666666667, ans=0.0 2023-11-22 04:48:45,859 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5800, loss[loss=0.06958, simple_loss=0.08819, pruned_loss=0.01549, audio_tagging_loss=0.01, over 16096.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09297, pruned_loss=0.01523, audio_tagging_loss=0.009383, over 3047436.99 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 8.0 2023-11-22 04:48:49,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1802140.0, ans=0.125 2023-11-22 04:48:54,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.23 vs. limit=22.5 2023-11-22 04:49:13,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1802273.3333333333, ans=0.125 2023-11-22 04:49:17,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1802273.3333333333, ans=0.125 2023-11-22 04:49:21,056 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270350 2023-11-22 04:49:23,354 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.122e+01 8.117e+01 8.783e+01 9.431e+01 1.385e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 04:49:32,005 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.40 vs. limit=22.5 2023-11-22 04:49:33,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1802340.0, ans=0.0 2023-11-22 04:49:50,746 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5850, loss[loss=0.04968, simple_loss=0.06388, pruned_loss=0.009206, audio_tagging_loss=0.008539, over 15700.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09273, pruned_loss=0.01508, audio_tagging_loss=0.009322, over 3044194.05 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 8.0 2023-11-22 04:49:59,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.27 vs. limit=10.0 2023-11-22 04:50:12,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1802540.0, ans=0.2 2023-11-22 04:50:23,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1802606.6666666667, ans=0.125 2023-11-22 04:50:24,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1802606.6666666667, ans=0.125 2023-11-22 04:50:25,346 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270400 2023-11-22 04:50:55,536 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5900, loss[loss=0.05393, simple_loss=0.06811, pruned_loss=0.01073, audio_tagging_loss=0.009147, over 14500.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09402, pruned_loss=0.0154, audio_tagging_loss=0.009251, over 3043151.99 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 8.0 2023-11-22 04:51:18,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1802873.3333333333, ans=0.125 2023-11-22 04:51:26,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.59 vs. limit=10.0 2023-11-22 04:51:30,450 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270450 2023-11-22 04:51:30,569 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 04:51:31,941 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 04:51:32,862 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.407e+01 8.248e+01 8.937e+01 9.600e+01 1.150e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-22 04:51:39,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1803006.6666666667, ans=0.125 2023-11-22 04:52:00,261 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 5950, loss[loss=0.07384, simple_loss=0.0934, pruned_loss=0.0156, audio_tagging_loss=0.01155, over 13290.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09442, pruned_loss=0.01544, audio_tagging_loss=0.009164, over 3037868.07 frames. ], batch size: 53, lr: 2.97e-03, grad_scale: 8.0 2023-11-22 04:52:05,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1803140.0, ans=0.125 2023-11-22 04:52:22,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1803206.6666666667, ans=0.0 2023-11-22 04:52:23,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1803206.6666666667, ans=0.2 2023-11-22 04:52:36,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270500 2023-11-22 04:52:36,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1803273.3333333333, ans=0.07 2023-11-22 04:52:41,515 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.45 vs. limit=15.0 2023-11-22 04:52:53,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1803406.6666666667, ans=0.0 2023-11-22 04:52:54,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1803406.6666666667, ans=0.125 2023-11-22 04:53:05,910 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6000, loss[loss=0.0679, simple_loss=0.09238, pruned_loss=0.01056, audio_tagging_loss=0.01116, over 15351.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.0945, pruned_loss=0.01538, audio_tagging_loss=0.009042, over 3040176.86 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:53:05,911 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 04:53:45,805 INFO [train_asr.py:1253] (3/4) Epoch 23, validation: loss=0.05955, simple_loss=0.05175, pruned_loss=0.005139, audio_tagging_loss=0.02853, over 4681554.00 frames. 2023-11-22 04:53:45,805 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 04:54:12,374 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.76 vs. limit=22.5 2023-11-22 04:54:20,409 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270550 2023-11-22 04:54:23,982 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.322e+01 8.050e+01 8.627e+01 9.368e+01 1.674e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-22 04:54:33,798 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 04:54:38,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1803740.0, ans=0.125 2023-11-22 04:54:50,468 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6050, loss[loss=0.06508, simple_loss=0.0959, pruned_loss=0.01044, audio_tagging_loss=0.006687, over 14732.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09464, pruned_loss=0.01557, audio_tagging_loss=0.009065, over 3047398.89 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:54:53,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1803806.6666666667, ans=0.125 2023-11-22 04:54:53,657 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.65 vs. limit=15.0 2023-11-22 04:54:59,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1803806.6666666667, ans=0.125 2023-11-22 04:55:24,554 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270600 2023-11-22 04:55:26,394 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.69 vs. limit=12.0 2023-11-22 04:55:30,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.66 vs. limit=6.0 2023-11-22 04:55:38,054 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.81 vs. limit=22.5 2023-11-22 04:55:54,020 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6100, loss[loss=0.06695, simple_loss=0.09379, pruned_loss=0.009367, audio_tagging_loss=0.01069, over 15030.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.0945, pruned_loss=0.01562, audio_tagging_loss=0.009136, over 3044288.12 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:56:05,889 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.94 vs. limit=15.0 2023-11-22 04:56:21,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.79 vs. limit=15.0 2023-11-22 04:56:29,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270650 2023-11-22 04:56:31,738 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.256e+01 8.259e+01 8.763e+01 9.618e+01 1.406e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-22 04:56:50,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1804406.6666666667, ans=0.1 2023-11-22 04:56:59,519 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6150, loss[loss=0.07844, simple_loss=0.104, pruned_loss=0.01709, audio_tagging_loss=0.009377, over 15291.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09432, pruned_loss=0.01552, audio_tagging_loss=0.009219, over 3049649.87 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:57:03,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1804473.3333333333, ans=0.125 2023-11-22 04:57:03,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.04 vs. limit=22.5 2023-11-22 04:57:09,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1804473.3333333333, ans=0.2 2023-11-22 04:57:22,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1804540.0, ans=0.0 2023-11-22 04:57:25,154 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.04 vs. limit=15.0 2023-11-22 04:57:30,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1804606.6666666667, ans=0.04949747468305833 2023-11-22 04:57:34,167 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270700 2023-11-22 04:57:55,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1804740.0, ans=0.0 2023-11-22 04:58:03,976 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6200, loss[loss=0.0631, simple_loss=0.08019, pruned_loss=0.0129, audio_tagging_loss=0.01011, over 14801.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09331, pruned_loss=0.01536, audio_tagging_loss=0.009319, over 3048473.70 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:58:20,893 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.81 vs. limit=15.0 2023-11-22 04:58:28,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.33 vs. limit=22.5 2023-11-22 04:58:38,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270750 2023-11-22 04:58:40,808 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.162e+01 9.001e+01 9.547e+01 1.182e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-22 04:58:51,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1805006.6666666667, ans=0.125 2023-11-22 04:58:59,896 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.69 vs. limit=22.5 2023-11-22 04:59:04,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.40 vs. limit=22.5 2023-11-22 04:59:07,553 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6250, loss[loss=0.06878, simple_loss=0.09007, pruned_loss=0.01457, audio_tagging_loss=0.009174, over 15321.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09446, pruned_loss=0.01558, audio_tagging_loss=0.009313, over 3044098.39 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 04:59:07,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1805140.0, ans=0.0 2023-11-22 04:59:11,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0 2023-11-22 04:59:22,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1805206.6666666667, ans=0.125 2023-11-22 04:59:32,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=22.5 2023-11-22 04:59:42,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270800 2023-11-22 05:00:09,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1805406.6666666667, ans=0.0 2023-11-22 05:00:11,874 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6300, loss[loss=0.09921, simple_loss=0.1361, pruned_loss=0.02396, audio_tagging_loss=0.007227, over 15540.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09459, pruned_loss=0.0156, audio_tagging_loss=0.009443, over 3042282.62 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 05:00:37,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.12 vs. limit=15.0 2023-11-22 05:00:46,738 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270850 2023-11-22 05:00:46,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1805606.6666666667, ans=0.125 2023-11-22 05:00:49,021 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.345e+01 8.951e+01 9.677e+01 1.248e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-22 05:00:51,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1805673.3333333333, ans=0.1 2023-11-22 05:01:08,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1805740.0, ans=0.125 2023-11-22 05:01:09,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.53 vs. limit=15.0 2023-11-22 05:01:16,578 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6350, loss[loss=0.07394, simple_loss=0.1017, pruned_loss=0.01645, audio_tagging_loss=0.006637, over 15569.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09439, pruned_loss=0.01552, audio_tagging_loss=0.00951, over 3043745.50 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 05:01:16,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1805806.6666666667, ans=0.1 2023-11-22 05:01:33,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1805873.3333333333, ans=0.1 2023-11-22 05:01:38,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.81 vs. limit=15.0 2023-11-22 05:01:46,522 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.28 vs. limit=22.5 2023-11-22 05:01:50,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1805940.0, ans=0.125 2023-11-22 05:01:50,964 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270900 2023-11-22 05:01:51,734 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.55 vs. limit=10.0 2023-11-22 05:02:06,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1806073.3333333333, ans=0.0 2023-11-22 05:02:08,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1806073.3333333333, ans=0.125 2023-11-22 05:02:17,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1806073.3333333333, ans=0.0 2023-11-22 05:02:20,579 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6400, loss[loss=0.06119, simple_loss=0.07885, pruned_loss=0.01078, audio_tagging_loss=0.01099, over 15379.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.0936, pruned_loss=0.01536, audio_tagging_loss=0.009679, over 3041915.52 frames. ], batch size: 59, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 05:02:30,834 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.35 vs. limit=10.0 2023-11-22 05:02:34,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1806206.6666666667, ans=0.1 2023-11-22 05:02:44,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1806273.3333333333, ans=0.0 2023-11-22 05:02:54,882 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 270950 2023-11-22 05:02:57,219 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.214e+01 8.231e+01 8.800e+01 9.655e+01 1.370e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 05:03:03,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.51 vs. limit=15.0 2023-11-22 05:03:21,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.36 vs. limit=22.5 2023-11-22 05:03:23,825 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6450, loss[loss=0.08339, simple_loss=0.1136, pruned_loss=0.01851, audio_tagging_loss=0.008071, over 14498.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.0943, pruned_loss=0.01535, audio_tagging_loss=0.009633, over 3045018.29 frames. ], batch size: 53, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 05:03:28,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1806473.3333333333, ans=0.2 2023-11-22 05:03:31,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.52 vs. limit=15.0 2023-11-22 05:03:41,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1806540.0, ans=0.125 2023-11-22 05:03:58,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271000 2023-11-22 05:04:26,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1806740.0, ans=0.125 2023-11-22 05:04:28,838 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6500, loss[loss=0.0945, simple_loss=0.1233, pruned_loss=0.02479, audio_tagging_loss=0.008083, over 15916.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.0931, pruned_loss=0.01513, audio_tagging_loss=0.009566, over 3046073.46 frames. ], batch size: 59, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 05:05:02,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271050 2023-11-22 05:05:05,187 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.277e+01 8.344e+01 8.837e+01 9.717e+01 1.139e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 05:05:31,935 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.78 vs. limit=6.0 2023-11-22 05:05:32,578 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6550, loss[loss=0.07511, simple_loss=0.09618, pruned_loss=0.01874, audio_tagging_loss=0.008282, over 14569.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09431, pruned_loss=0.0155, audio_tagging_loss=0.009405, over 3051501.75 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 05:05:36,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1807140.0, ans=0.125 2023-11-22 05:05:38,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1807140.0, ans=0.0 2023-11-22 05:06:06,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1807273.3333333333, ans=0.125 2023-11-22 05:06:07,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271100 2023-11-22 05:06:36,250 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6600, loss[loss=0.09083, simple_loss=0.1281, pruned_loss=0.01866, audio_tagging_loss=0.008116, over 14449.00 frames. ], tot_loss[loss=0.07262, simple_loss=0.09528, pruned_loss=0.01568, audio_tagging_loss=0.009302, over 3041761.24 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 05:07:02,065 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.64 vs. limit=15.0 2023-11-22 05:07:11,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271150 2023-11-22 05:07:13,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1807606.6666666667, ans=0.125 2023-11-22 05:07:15,163 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.298e+01 8.931e+01 9.754e+01 1.164e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 05:07:38,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1807806.6666666667, ans=0.1 2023-11-22 05:07:40,528 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6650, loss[loss=0.06781, simple_loss=0.09134, pruned_loss=0.01195, audio_tagging_loss=0.01019, over 16415.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09474, pruned_loss=0.01554, audio_tagging_loss=0.009323, over 3045128.01 frames. ], batch size: 61, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 05:08:12,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1807940.0, ans=0.025 2023-11-22 05:08:15,362 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271200 2023-11-22 05:08:37,381 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:08:37,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1808073.3333333333, ans=0.0 2023-11-22 05:08:45,695 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6700, loss[loss=0.08215, simple_loss=0.11, pruned_loss=0.01635, audio_tagging_loss=0.01082, over 16951.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09442, pruned_loss=0.01542, audio_tagging_loss=0.009238, over 3047889.56 frames. ], batch size: 64, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 05:08:55,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1808140.0, ans=0.125 2023-11-22 05:09:10,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1808273.3333333333, ans=0.2 2023-11-22 05:09:20,499 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271250 2023-11-22 05:09:24,049 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.573e+01 8.261e+01 8.925e+01 9.693e+01 1.242e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-22 05:09:49,335 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.59 vs. limit=12.0 2023-11-22 05:09:50,111 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6750, loss[loss=0.06942, simple_loss=0.09632, pruned_loss=0.01314, audio_tagging_loss=0.008117, over 14961.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09451, pruned_loss=0.01549, audio_tagging_loss=0.009267, over 3045538.99 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0 2023-11-22 05:09:50,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1808473.3333333333, ans=22.5 2023-11-22 05:10:02,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1808540.0, ans=0.0 2023-11-22 05:10:25,939 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271300 2023-11-22 05:10:32,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1808673.3333333333, ans=0.125 2023-11-22 05:10:33,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1808673.3333333333, ans=0.0 2023-11-22 05:10:44,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1808740.0, ans=0.0 2023-11-22 05:10:54,799 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6800, loss[loss=0.08141, simple_loss=0.1103, pruned_loss=0.0161, audio_tagging_loss=0.01014, over 15130.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09475, pruned_loss=0.01551, audio_tagging_loss=0.009251, over 3048886.43 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 32.0 2023-11-22 05:11:26,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1808940.0, ans=0.1 2023-11-22 05:11:30,406 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271350 2023-11-22 05:11:30,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1808940.0, ans=0.125 2023-11-22 05:11:34,842 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.372e+01 8.044e+01 8.828e+01 9.747e+01 1.352e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 05:11:36,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1809006.6666666667, ans=0.2 2023-11-22 05:12:00,281 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6850, loss[loss=0.06082, simple_loss=0.07593, pruned_loss=0.01309, audio_tagging_loss=0.009764, over 15024.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09474, pruned_loss=0.01547, audio_tagging_loss=0.009215, over 3045183.86 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:12:00,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1809140.0, ans=0.2 2023-11-22 05:12:09,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1809140.0, ans=0.125 2023-11-22 05:12:24,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1809206.6666666667, ans=0.0 2023-11-22 05:12:31,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1809273.3333333333, ans=0.125 2023-11-22 05:12:35,805 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271400 2023-11-22 05:12:51,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1809406.6666666667, ans=0.1 2023-11-22 05:13:06,056 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6900, loss[loss=0.05588, simple_loss=0.06167, pruned_loss=0.01149, audio_tagging_loss=0.01356, over 17099.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.09403, pruned_loss=0.01537, audio_tagging_loss=0.009266, over 3038099.93 frames. ], batch size: 66, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:13:06,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1809473.3333333333, ans=0.125 2023-11-22 05:13:21,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1809540.0, ans=0.0 2023-11-22 05:13:40,992 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271450 2023-11-22 05:13:46,997 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.045e+01 8.623e+01 9.194e+01 1.221e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-22 05:13:48,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1809673.3333333333, ans=0.1 2023-11-22 05:13:55,782 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 05:14:04,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1809740.0, ans=0.125 2023-11-22 05:14:10,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1809806.6666666667, ans=0.125 2023-11-22 05:14:11,289 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 6950, loss[loss=0.07227, simple_loss=0.09466, pruned_loss=0.01344, audio_tagging_loss=0.0115, over 14396.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09475, pruned_loss=0.01546, audio_tagging_loss=0.009229, over 3036403.22 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:14:18,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1809806.6666666667, ans=0.2 2023-11-22 05:14:29,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1809873.3333333333, ans=0.125 2023-11-22 05:14:31,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1809873.3333333333, ans=0.125 2023-11-22 05:14:46,463 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271500 2023-11-22 05:15:01,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1810073.3333333333, ans=0.125 2023-11-22 05:15:09,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1810073.3333333333, ans=0.1 2023-11-22 05:15:15,669 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7000, loss[loss=0.09425, simple_loss=0.1212, pruned_loss=0.0253, audio_tagging_loss=0.008367, over 15168.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09512, pruned_loss=0.01559, audio_tagging_loss=0.009264, over 3038449.18 frames. ], batch size: 60, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:15:31,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1810206.6666666667, ans=0.1 2023-11-22 05:15:41,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1810273.3333333333, ans=0.125 2023-11-22 05:15:48,267 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:15:48,745 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.02 vs. limit=22.5 2023-11-22 05:15:50,547 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271550 2023-11-22 05:15:55,914 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.160e+01 8.872e+01 9.665e+01 1.181e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 05:15:57,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1810340.0, ans=0.0 2023-11-22 05:16:01,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1810340.0, ans=0.0 2023-11-22 05:16:08,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1810406.6666666667, ans=0.0 2023-11-22 05:16:14,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1810406.6666666667, ans=0.0 2023-11-22 05:16:20,964 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7050, loss[loss=0.09825, simple_loss=0.1277, pruned_loss=0.02324, audio_tagging_loss=0.01116, over 15058.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09569, pruned_loss=0.01569, audio_tagging_loss=0.009353, over 3041688.18 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:16:44,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1810540.0, ans=0.0 2023-11-22 05:16:53,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1810606.6666666667, ans=0.0 2023-11-22 05:16:53,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.24 vs. limit=6.0 2023-11-22 05:16:55,771 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271600 2023-11-22 05:17:23,887 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:17:26,017 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7100, loss[loss=0.07448, simple_loss=0.09114, pruned_loss=0.01665, audio_tagging_loss=0.01226, over 15077.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09487, pruned_loss=0.01553, audio_tagging_loss=0.009436, over 3036666.85 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:17:37,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.79 vs. limit=15.0 2023-11-22 05:17:44,493 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.69 vs. limit=8.0 2023-11-22 05:17:47,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=1810873.3333333333, ans=0.02 2023-11-22 05:17:51,874 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.64 vs. limit=12.0 2023-11-22 05:17:59,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271650 2023-11-22 05:18:01,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1810940.0, ans=0.125 2023-11-22 05:18:05,325 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.205e+01 9.069e+01 1.012e+02 2.750e+02, threshold=1.814e+02, percent-clipped=1.0 2023-11-22 05:18:08,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1811006.6666666667, ans=0.125 2023-11-22 05:18:30,057 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7150, loss[loss=0.07241, simple_loss=0.09896, pruned_loss=0.01401, audio_tagging_loss=0.008929, over 15274.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09543, pruned_loss=0.01555, audio_tagging_loss=0.009408, over 3034681.31 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:18:39,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1811140.0, ans=0.125 2023-11-22 05:18:45,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1811206.6666666667, ans=0.125 2023-11-22 05:18:49,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.29 vs. limit=6.0 2023-11-22 05:18:50,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1811206.6666666667, ans=0.1 2023-11-22 05:19:05,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271700 2023-11-22 05:19:06,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1811273.3333333333, ans=0.125 2023-11-22 05:19:10,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1811340.0, ans=15.0 2023-11-22 05:19:34,518 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7200, loss[loss=0.06972, simple_loss=0.09092, pruned_loss=0.01532, audio_tagging_loss=0.008942, over 14750.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09502, pruned_loss=0.01553, audio_tagging_loss=0.009525, over 3031930.39 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:19:39,826 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:19:40,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1811473.3333333333, ans=0.0 2023-11-22 05:19:55,401 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.69 vs. limit=6.0 2023-11-22 05:20:09,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271750 2023-11-22 05:20:14,758 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.124e+01 9.002e+01 9.849e+01 1.230e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-22 05:20:18,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1811673.3333333333, ans=0.125 2023-11-22 05:20:34,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1811740.0, ans=0.0 2023-11-22 05:20:36,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1811740.0, ans=0.0 2023-11-22 05:20:38,219 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.75 vs. limit=10.0 2023-11-22 05:20:40,019 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7250, loss[loss=0.07569, simple_loss=0.09673, pruned_loss=0.01726, audio_tagging_loss=0.01007, over 14694.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09476, pruned_loss=0.01547, audio_tagging_loss=0.00957, over 3033614.92 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:20:43,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1811806.6666666667, ans=0.2 2023-11-22 05:20:46,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1811806.6666666667, ans=0.0 2023-11-22 05:21:08,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1811940.0, ans=0.125 2023-11-22 05:21:12,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1811940.0, ans=0.125 2023-11-22 05:21:14,415 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271800 2023-11-22 05:21:15,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1811940.0, ans=0.125 2023-11-22 05:21:16,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1811940.0, ans=0.125 2023-11-22 05:21:29,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.43 vs. limit=15.0 2023-11-22 05:21:34,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.80 vs. limit=22.5 2023-11-22 05:21:35,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1812073.3333333333, ans=0.1 2023-11-22 05:21:44,757 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7300, loss[loss=0.0756, simple_loss=0.09567, pruned_loss=0.0158, audio_tagging_loss=0.01196, over 15525.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09419, pruned_loss=0.01538, audio_tagging_loss=0.009548, over 3038348.61 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:21:57,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1812206.6666666667, ans=0.0 2023-11-22 05:22:19,981 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271850 2023-11-22 05:22:25,967 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.604e+01 8.310e+01 9.010e+01 9.721e+01 2.858e+02, threshold=1.802e+02, percent-clipped=1.0 2023-11-22 05:22:37,219 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.88 vs. limit=15.0 2023-11-22 05:22:38,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0 2023-11-22 05:22:49,425 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7350, loss[loss=0.08315, simple_loss=0.1163, pruned_loss=0.01607, audio_tagging_loss=0.008934, over 15763.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09353, pruned_loss=0.01517, audio_tagging_loss=0.009433, over 3039807.75 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:23:09,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1812540.0, ans=0.125 2023-11-22 05:23:14,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.99 vs. limit=15.0 2023-11-22 05:23:21,970 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.64 vs. limit=6.0 2023-11-22 05:23:25,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271900 2023-11-22 05:23:29,003 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:23:33,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1812673.3333333333, ans=0.07 2023-11-22 05:23:39,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1812673.3333333333, ans=0.125 2023-11-22 05:23:40,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1812740.0, ans=0.09899494936611666 2023-11-22 05:23:45,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1812740.0, ans=0.1 2023-11-22 05:23:54,483 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7400, loss[loss=0.04862, simple_loss=0.06095, pruned_loss=0.009008, audio_tagging_loss=0.009136, over 14239.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09221, pruned_loss=0.01484, audio_tagging_loss=0.009438, over 3036777.95 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:23:54,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1812806.6666666667, ans=0.1 2023-11-22 05:24:01,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1812806.6666666667, ans=0.1 2023-11-22 05:24:17,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1812873.3333333333, ans=0.0 2023-11-22 05:24:17,654 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.54 vs. limit=10.0 2023-11-22 05:24:30,096 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 271950 2023-11-22 05:24:30,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1812940.0, ans=0.125 2023-11-22 05:24:36,087 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 7.965e+01 8.642e+01 9.211e+01 1.272e+02, threshold=1.728e+02, percent-clipped=0.0 2023-11-22 05:24:38,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1813006.6666666667, ans=0.125 2023-11-22 05:24:46,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1813073.3333333333, ans=0.0 2023-11-22 05:24:59,492 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.35 vs. limit=15.0 2023-11-22 05:25:00,090 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7450, loss[loss=0.06437, simple_loss=0.08806, pruned_loss=0.0138, audio_tagging_loss=0.006536, over 14053.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09306, pruned_loss=0.01489, audio_tagging_loss=0.009369, over 3037567.56 frames. ], batch size: 53, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:25:20,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1813206.6666666667, ans=0.125 2023-11-22 05:25:23,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1813206.6666666667, ans=0.0 2023-11-22 05:25:23,831 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.47 vs. limit=10.0 2023-11-22 05:25:24,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1813273.3333333333, ans=0.125 2023-11-22 05:25:34,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272000 2023-11-22 05:25:34,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1813273.3333333333, ans=0.125 2023-11-22 05:25:35,088 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.00 vs. limit=12.0 2023-11-22 05:26:08,227 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7500, loss[loss=0.07818, simple_loss=0.1047, pruned_loss=0.01783, audio_tagging_loss=0.008008, over 15896.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09359, pruned_loss=0.01514, audio_tagging_loss=0.009324, over 3035034.69 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:26:25,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1813540.0, ans=0.125 2023-11-22 05:26:29,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1813540.0, ans=0.125 2023-11-22 05:26:31,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1813540.0, ans=0.125 2023-11-22 05:26:43,635 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272050 2023-11-22 05:26:49,564 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.448e+01 8.968e+01 9.582e+01 2.186e+02, threshold=1.794e+02, percent-clipped=1.0 2023-11-22 05:26:52,417 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:26:57,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1813673.3333333333, ans=0.125 2023-11-22 05:26:58,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.46 vs. limit=22.5 2023-11-22 05:26:59,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1813740.0, ans=0.1 2023-11-22 05:27:13,282 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7550, loss[loss=0.06068, simple_loss=0.08081, pruned_loss=0.01128, audio_tagging_loss=0.008997, over 16458.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09557, pruned_loss=0.01569, audio_tagging_loss=0.00922, over 3049321.79 frames. ], batch size: 63, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:27:13,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1813806.6666666667, ans=0.125 2023-11-22 05:27:23,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1813806.6666666667, ans=0.2 2023-11-22 05:27:46,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.61 vs. limit=15.0 2023-11-22 05:27:48,114 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272100 2023-11-22 05:27:52,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1814006.6666666667, ans=0.125 2023-11-22 05:28:02,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1814006.6666666667, ans=0.0 2023-11-22 05:28:18,021 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7600, loss[loss=0.05134, simple_loss=0.06287, pruned_loss=0.007184, audio_tagging_loss=0.01272, over 15555.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09414, pruned_loss=0.01541, audio_tagging_loss=0.009317, over 3045131.18 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:28:20,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1814140.0, ans=0.0 2023-11-22 05:28:24,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1814140.0, ans=0.0 2023-11-22 05:28:47,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1814273.3333333333, ans=0.0 2023-11-22 05:28:51,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272150 2023-11-22 05:28:57,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1814340.0, ans=0.0 2023-11-22 05:28:59,136 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.478e+01 8.103e+01 8.710e+01 9.390e+01 1.351e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-22 05:29:00,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1814340.0, ans=0.1 2023-11-22 05:29:13,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1814406.6666666667, ans=0.2 2023-11-22 05:29:20,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.81 vs. limit=15.0 2023-11-22 05:29:20,749 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7650, loss[loss=0.04941, simple_loss=0.06573, pruned_loss=0.007801, audio_tagging_loss=0.008744, over 14985.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.0939, pruned_loss=0.01531, audio_tagging_loss=0.009258, over 3053957.94 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:29:27,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1814473.3333333333, ans=0.5 2023-11-22 05:29:44,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1814540.0, ans=0.0 2023-11-22 05:29:56,425 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272200 2023-11-22 05:30:06,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1814673.3333333333, ans=0.0 2023-11-22 05:30:25,811 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7700, loss[loss=0.07044, simple_loss=0.1006, pruned_loss=0.01176, audio_tagging_loss=0.008374, over 14962.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09425, pruned_loss=0.01543, audio_tagging_loss=0.009252, over 3050271.70 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:30:32,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1814806.6666666667, ans=0.0 2023-11-22 05:30:38,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1814873.3333333333, ans=0.125 2023-11-22 05:30:51,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1814940.0, ans=0.0 2023-11-22 05:31:01,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272250 2023-11-22 05:31:09,430 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 7.911e+01 8.690e+01 9.336e+01 1.339e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-22 05:31:14,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1815006.6666666667, ans=0.0 2023-11-22 05:31:25,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1815073.3333333333, ans=0.025 2023-11-22 05:31:31,236 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7750, loss[loss=0.09807, simple_loss=0.125, pruned_loss=0.02496, audio_tagging_loss=0.0106, over 15666.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09466, pruned_loss=0.01564, audio_tagging_loss=0.009321, over 3044651.47 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:31:42,967 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.18 vs. limit=15.0 2023-11-22 05:31:50,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.62 vs. limit=15.0 2023-11-22 05:31:52,688 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.24 vs. limit=5.0 2023-11-22 05:32:06,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272300 2023-11-22 05:32:09,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1815340.0, ans=0.0 2023-11-22 05:32:36,668 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7800, loss[loss=0.07332, simple_loss=0.08632, pruned_loss=0.01856, audio_tagging_loss=0.0116, over 14434.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09454, pruned_loss=0.0157, audio_tagging_loss=0.009381, over 3043852.41 frames. ], batch size: 53, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:32:57,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1815540.0, ans=0.1 2023-11-22 05:33:11,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272350 2023-11-22 05:33:16,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1815673.3333333333, ans=0.125 2023-11-22 05:33:19,304 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.265e+01 8.937e+01 9.454e+01 1.226e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-22 05:33:19,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1815673.3333333333, ans=0.125 2023-11-22 05:33:41,777 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7850, loss[loss=0.05779, simple_loss=0.07864, pruned_loss=0.00944, audio_tagging_loss=0.009033, over 15083.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09463, pruned_loss=0.0157, audio_tagging_loss=0.009353, over 3042797.89 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:33:47,235 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0 2023-11-22 05:33:58,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1815873.3333333333, ans=0.05 2023-11-22 05:34:01,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1815873.3333333333, ans=0.0 2023-11-22 05:34:06,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1815940.0, ans=0.2 2023-11-22 05:34:11,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1815940.0, ans=0.125 2023-11-22 05:34:17,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272400 2023-11-22 05:34:32,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.11 vs. limit=15.0 2023-11-22 05:34:47,294 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7900, loss[loss=0.05705, simple_loss=0.07298, pruned_loss=0.01092, audio_tagging_loss=0.009641, over 14999.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09502, pruned_loss=0.01564, audio_tagging_loss=0.009336, over 3038297.32 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:34:53,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1816140.0, ans=0.0 2023-11-22 05:35:02,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1816206.6666666667, ans=0.125 2023-11-22 05:35:16,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1816273.3333333333, ans=0.05 2023-11-22 05:35:22,017 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272450 2023-11-22 05:35:22,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.25 vs. limit=10.0 2023-11-22 05:35:30,025 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.475e+01 8.108e+01 8.831e+01 9.576e+01 1.163e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 05:35:36,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1816340.0, ans=0.0 2023-11-22 05:35:37,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1816406.6666666667, ans=0.125 2023-11-22 05:35:51,650 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 7950, loss[loss=0.05604, simple_loss=0.06713, pruned_loss=0.01097, audio_tagging_loss=0.01151, over 14918.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09418, pruned_loss=0.01558, audio_tagging_loss=0.009629, over 3035302.38 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:36:07,932 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 05:36:11,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1816540.0, ans=0.125 2023-11-22 05:36:27,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272500 2023-11-22 05:36:57,554 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8000, loss[loss=0.06753, simple_loss=0.082, pruned_loss=0.01449, audio_tagging_loss=0.01204, over 14383.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.09313, pruned_loss=0.01535, audio_tagging_loss=0.009758, over 3038324.45 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:37:01,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.82 vs. limit=15.0 2023-11-22 05:37:03,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1816806.6666666667, ans=0.2 2023-11-22 05:37:14,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1816873.3333333333, ans=0.125 2023-11-22 05:37:27,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1816940.0, ans=0.0 2023-11-22 05:37:32,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272550 2023-11-22 05:37:33,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.09 vs. limit=15.0 2023-11-22 05:37:36,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1817006.6666666667, ans=0.0 2023-11-22 05:37:40,494 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.372e+01 8.294e+01 8.813e+01 9.754e+01 1.240e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-22 05:38:02,640 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8050, loss[loss=0.05965, simple_loss=0.07087, pruned_loss=0.01334, audio_tagging_loss=0.01088, over 14843.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09381, pruned_loss=0.01542, audio_tagging_loss=0.009708, over 3035680.11 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:38:20,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1817206.6666666667, ans=0.125 2023-11-22 05:38:21,906 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.77 vs. limit=15.0 2023-11-22 05:38:32,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.27 vs. limit=22.5 2023-11-22 05:38:37,257 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272600 2023-11-22 05:38:38,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1817273.3333333333, ans=0.125 2023-11-22 05:39:06,916 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8100, loss[loss=0.06667, simple_loss=0.08089, pruned_loss=0.01537, audio_tagging_loss=0.01086, over 14691.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09321, pruned_loss=0.01534, audio_tagging_loss=0.009715, over 3032542.94 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:39:16,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1817473.3333333333, ans=0.2 2023-11-22 05:39:22,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1817540.0, ans=0.2 2023-11-22 05:39:31,962 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.51 vs. limit=15.0 2023-11-22 05:39:41,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272650 2023-11-22 05:39:50,205 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.59 vs. limit=15.0 2023-11-22 05:39:50,703 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.525e+01 8.265e+01 8.788e+01 9.527e+01 1.186e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 05:40:07,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1817740.0, ans=0.125 2023-11-22 05:40:11,844 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8150, loss[loss=0.08369, simple_loss=0.112, pruned_loss=0.01977, audio_tagging_loss=0.007918, over 16051.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09356, pruned_loss=0.0155, audio_tagging_loss=0.009548, over 3034150.07 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:40:21,012 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.92 vs. limit=10.0 2023-11-22 05:40:31,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1817873.3333333333, ans=0.125 2023-11-22 05:40:44,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1817940.0, ans=0.2 2023-11-22 05:40:46,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272700 2023-11-22 05:40:51,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1818006.6666666667, ans=0.05 2023-11-22 05:40:55,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1818006.6666666667, ans=0.0 2023-11-22 05:41:14,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1818073.3333333333, ans=0.125 2023-11-22 05:41:15,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1818140.0, ans=0.125 2023-11-22 05:41:16,702 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8200, loss[loss=0.07298, simple_loss=0.09478, pruned_loss=0.01686, audio_tagging_loss=0.008729, over 15321.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09367, pruned_loss=0.01541, audio_tagging_loss=0.009365, over 3031702.29 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:41:16,766 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 05:41:23,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1818140.0, ans=0.1 2023-11-22 05:41:32,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1818206.6666666667, ans=0.125 2023-11-22 05:41:36,425 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.54 vs. limit=15.0 2023-11-22 05:41:37,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.51 vs. limit=15.0 2023-11-22 05:41:39,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1818206.6666666667, ans=0.1 2023-11-22 05:41:39,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1818206.6666666667, ans=0.07 2023-11-22 05:41:50,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272750 2023-11-22 05:42:00,278 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.465e+01 8.322e+01 8.892e+01 9.603e+01 1.403e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 05:42:21,083 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8250, loss[loss=0.07344, simple_loss=0.08803, pruned_loss=0.01783, audio_tagging_loss=0.01159, over 14768.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09358, pruned_loss=0.01559, audio_tagging_loss=0.009387, over 3031255.87 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:42:29,135 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.40 vs. limit=15.0 2023-11-22 05:42:32,984 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.65 vs. limit=6.0 2023-11-22 05:42:45,135 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.31 vs. limit=15.0 2023-11-22 05:42:53,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1818606.6666666667, ans=0.125 2023-11-22 05:42:56,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272800 2023-11-22 05:43:21,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1818740.0, ans=0.0 2023-11-22 05:43:25,813 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8300, loss[loss=0.08174, simple_loss=0.1083, pruned_loss=0.01825, audio_tagging_loss=0.009342, over 16921.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09316, pruned_loss=0.01545, audio_tagging_loss=0.009363, over 3035334.45 frames. ], batch size: 62, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:43:43,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1818873.3333333333, ans=0.125 2023-11-22 05:44:01,260 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272850 2023-11-22 05:44:09,829 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.221e+01 8.844e+01 9.406e+01 1.147e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 05:44:10,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1819006.6666666667, ans=0.125 2023-11-22 05:44:31,033 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8350, loss[loss=0.0825, simple_loss=0.1118, pruned_loss=0.01979, audio_tagging_loss=0.006806, over 15331.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09389, pruned_loss=0.01554, audio_tagging_loss=0.009379, over 3042242.67 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0 2023-11-22 05:44:31,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1819140.0, ans=0.0 2023-11-22 05:44:43,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.48 vs. limit=15.0 2023-11-22 05:45:04,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272900 2023-11-22 05:45:06,682 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.70 vs. limit=15.0 2023-11-22 05:45:14,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1819340.0, ans=0.125 2023-11-22 05:45:18,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1819340.0, ans=0.1 2023-11-22 05:45:19,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1819340.0, ans=0.0 2023-11-22 05:45:28,476 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:45:34,870 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8400, loss[loss=0.06455, simple_loss=0.07828, pruned_loss=0.01526, audio_tagging_loss=0.01015, over 14151.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09346, pruned_loss=0.01534, audio_tagging_loss=0.009377, over 3046516.92 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:45:42,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1819473.3333333333, ans=0.125 2023-11-22 05:45:47,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1819540.0, ans=0.2 2023-11-22 05:46:09,202 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 272950 2023-11-22 05:46:13,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1819673.3333333333, ans=0.95 2023-11-22 05:46:17,479 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.034e+01 8.663e+01 9.140e+01 1.267e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-22 05:46:21,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1819673.3333333333, ans=0.1 2023-11-22 05:46:25,655 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:46:30,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1819740.0, ans=0.125 2023-11-22 05:46:36,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1819806.6666666667, ans=0.125 2023-11-22 05:46:37,825 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8450, loss[loss=0.07964, simple_loss=0.1071, pruned_loss=0.01828, audio_tagging_loss=0.007792, over 15583.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09348, pruned_loss=0.01537, audio_tagging_loss=0.009464, over 3050579.41 frames. ], batch size: 60, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:47:12,781 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273000 2023-11-22 05:47:14,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1819940.0, ans=0.2 2023-11-22 05:47:22,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1820006.6666666667, ans=0.125 2023-11-22 05:47:25,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1820006.6666666667, ans=0.2 2023-11-22 05:47:34,970 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:47:41,852 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8500, loss[loss=0.08021, simple_loss=0.102, pruned_loss=0.01938, audio_tagging_loss=0.009848, over 15232.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09397, pruned_loss=0.01534, audio_tagging_loss=0.009519, over 3056986.72 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:48:02,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1820206.6666666667, ans=0.5 2023-11-22 05:48:16,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273050 2023-11-22 05:48:25,404 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.253e+01 8.845e+01 9.555e+01 1.261e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 05:48:26,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1820340.0, ans=0.025 2023-11-22 05:48:29,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1820340.0, ans=0.125 2023-11-22 05:48:34,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1820406.6666666667, ans=0.125 2023-11-22 05:48:44,329 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.48 vs. limit=15.0 2023-11-22 05:48:46,826 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8550, loss[loss=0.07169, simple_loss=0.09793, pruned_loss=0.01535, audio_tagging_loss=0.007373, over 15132.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09449, pruned_loss=0.01548, audio_tagging_loss=0.009525, over 3050784.99 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:48:47,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1820473.3333333333, ans=0.0 2023-11-22 05:48:54,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1820473.3333333333, ans=0.025 2023-11-22 05:49:09,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1820540.0, ans=0.125 2023-11-22 05:49:21,330 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273100 2023-11-22 05:49:36,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1820673.3333333333, ans=0.2 2023-11-22 05:49:49,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1820806.6666666667, ans=0.125 2023-11-22 05:49:50,452 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8600, loss[loss=0.0721, simple_loss=0.1005, pruned_loss=0.01274, audio_tagging_loss=0.009104, over 15444.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09483, pruned_loss=0.01547, audio_tagging_loss=0.009508, over 3046850.84 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 32.0 2023-11-22 05:49:58,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.69 vs. limit=15.0 2023-11-22 05:50:01,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1820806.6666666667, ans=0.1 2023-11-22 05:50:04,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1820873.3333333333, ans=0.125 2023-11-22 05:50:25,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273150 2023-11-22 05:50:28,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1821006.6666666667, ans=0.0 2023-11-22 05:50:35,538 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.265e+01 8.649e+01 9.295e+01 1.205e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-22 05:50:38,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1821006.6666666667, ans=0.1 2023-11-22 05:50:54,632 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8650, loss[loss=0.08357, simple_loss=0.1104, pruned_loss=0.02015, audio_tagging_loss=0.008216, over 15421.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09516, pruned_loss=0.01543, audio_tagging_loss=0.009466, over 3047294.74 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 05:51:00,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1821140.0, ans=0.0 2023-11-22 05:51:29,183 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273200 2023-11-22 05:51:38,232 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.27 vs. limit=5.0 2023-11-22 05:51:48,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1821406.6666666667, ans=0.125 2023-11-22 05:51:59,110 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8700, loss[loss=0.07674, simple_loss=0.1046, pruned_loss=0.01629, audio_tagging_loss=0.008147, over 15176.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09487, pruned_loss=0.0153, audio_tagging_loss=0.009552, over 3044131.04 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 05:52:08,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1821473.3333333333, ans=0.125 2023-11-22 05:52:14,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1821540.0, ans=0.0 2023-11-22 05:52:33,724 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273250 2023-11-22 05:52:43,985 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 7.986e+01 8.723e+01 9.547e+01 1.201e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-22 05:53:03,549 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8750, loss[loss=0.07908, simple_loss=0.1115, pruned_loss=0.01553, audio_tagging_loss=0.007802, over 15212.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09617, pruned_loss=0.01575, audio_tagging_loss=0.009581, over 3040389.71 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 05:53:36,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1821940.0, ans=0.2 2023-11-22 05:53:37,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1821940.0, ans=0.125 2023-11-22 05:53:38,800 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273300 2023-11-22 05:54:07,853 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8800, loss[loss=0.07794, simple_loss=0.08691, pruned_loss=0.02215, audio_tagging_loss=0.01233, over 16251.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09685, pruned_loss=0.016, audio_tagging_loss=0.009582, over 3042315.92 frames. ], batch size: 63, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 05:54:36,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1822273.3333333333, ans=0.025 2023-11-22 05:54:42,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273350 2023-11-22 05:54:42,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1822273.3333333333, ans=0.0 2023-11-22 05:54:52,332 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.080e+01 8.934e+01 9.573e+01 1.295e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-22 05:55:11,544 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8850, loss[loss=0.0762, simple_loss=0.09726, pruned_loss=0.01772, audio_tagging_loss=0.009851, over 14866.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09506, pruned_loss=0.01581, audio_tagging_loss=0.009717, over 3046565.00 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 05:55:23,824 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 05:55:26,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1822540.0, ans=0.1 2023-11-22 05:55:28,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.93 vs. limit=15.0 2023-11-22 05:55:36,202 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.36 vs. limit=15.0 2023-11-22 05:55:36,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1822606.6666666667, ans=0.125 2023-11-22 05:55:40,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1822606.6666666667, ans=0.2 2023-11-22 05:55:43,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1822606.6666666667, ans=0.125 2023-11-22 05:55:46,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273400 2023-11-22 05:56:13,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1822740.0, ans=0.0 2023-11-22 05:56:16,870 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8900, loss[loss=0.07571, simple_loss=0.09855, pruned_loss=0.01882, audio_tagging_loss=0.007613, over 13621.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09405, pruned_loss=0.01558, audio_tagging_loss=0.009683, over 3045171.88 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 05:56:19,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1822806.6666666667, ans=0.1 2023-11-22 05:56:39,719 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:56:51,054 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273450 2023-11-22 05:57:03,198 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.200e+01 8.739e+01 9.306e+01 1.295e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 05:57:10,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1823073.3333333333, ans=0.1 2023-11-22 05:57:18,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1823073.3333333333, ans=0.015 2023-11-22 05:57:20,953 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 8950, loss[loss=0.09728, simple_loss=0.1355, pruned_loss=0.02209, audio_tagging_loss=0.007442, over 14600.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09423, pruned_loss=0.01549, audio_tagging_loss=0.009538, over 3040027.85 frames. ], batch size: 52, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 05:57:48,277 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:57:56,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273500 2023-11-22 05:58:07,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1823340.0, ans=0.0 2023-11-22 05:58:15,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1823406.6666666667, ans=0.125 2023-11-22 05:58:17,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1823406.6666666667, ans=0.0 2023-11-22 05:58:25,506 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9000, loss[loss=0.07892, simple_loss=0.1074, pruned_loss=0.01721, audio_tagging_loss=0.008014, over 16152.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09473, pruned_loss=0.01561, audio_tagging_loss=0.009372, over 3047219.80 frames. ], batch size: 63, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 05:58:25,506 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 05:59:06,000 INFO [train_asr.py:1253] (3/4) Epoch 23, validation: loss=0.06035, simple_loss=0.05169, pruned_loss=0.005137, audio_tagging_loss=0.02937, over 4681554.00 frames. 2023-11-22 05:59:06,000 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 05:59:15,965 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.79 vs. limit=22.5 2023-11-22 05:59:25,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1823540.0, ans=0.2 2023-11-22 05:59:29,453 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 05:59:39,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1823606.6666666667, ans=0.125 2023-11-22 05:59:40,479 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273550 2023-11-22 05:59:52,551 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.165e+01 8.843e+01 9.497e+01 1.751e+02, threshold=1.769e+02, percent-clipped=1.0 2023-11-22 05:59:56,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1823740.0, ans=0.1 2023-11-22 06:00:10,312 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9050, loss[loss=0.05366, simple_loss=0.06956, pruned_loss=0.01129, audio_tagging_loss=0.00759, over 15896.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09524, pruned_loss=0.01568, audio_tagging_loss=0.009277, over 3061097.73 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:00:38,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1823940.0, ans=0.05 2023-11-22 06:00:45,204 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273600 2023-11-22 06:00:52,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1824006.6666666667, ans=0.0 2023-11-22 06:01:13,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1824140.0, ans=0.1 2023-11-22 06:01:14,752 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9100, loss[loss=0.08387, simple_loss=0.1106, pruned_loss=0.021, audio_tagging_loss=0.007588, over 15091.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.09453, pruned_loss=0.01549, audio_tagging_loss=0.00923, over 3052690.74 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:01:28,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1824206.6666666667, ans=0.125 2023-11-22 06:01:49,989 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273650 2023-11-22 06:01:51,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1824273.3333333333, ans=0.125 2023-11-22 06:02:01,427 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.073e+01 8.633e+01 9.456e+01 1.130e+02, threshold=1.727e+02, percent-clipped=0.0 2023-11-22 06:02:01,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1824340.0, ans=0.0 2023-11-22 06:02:13,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1824406.6666666667, ans=0.125 2023-11-22 06:02:19,624 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9150, loss[loss=0.06244, simple_loss=0.07194, pruned_loss=0.01431, audio_tagging_loss=0.01215, over 15727.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09413, pruned_loss=0.01545, audio_tagging_loss=0.009257, over 3054215.41 frames. ], batch size: 61, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:02:24,072 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.08 vs. limit=15.0 2023-11-22 06:02:46,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1824606.6666666667, ans=0.125 2023-11-22 06:02:47,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1824606.6666666667, ans=0.125 2023-11-22 06:02:51,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1824606.6666666667, ans=10.0 2023-11-22 06:02:54,854 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273700 2023-11-22 06:03:25,055 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9200, loss[loss=0.06287, simple_loss=0.07726, pruned_loss=0.008371, audio_tagging_loss=0.01586, over 14958.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09386, pruned_loss=0.01544, audio_tagging_loss=0.00934, over 3056475.16 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:03:29,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1824806.6666666667, ans=0.0 2023-11-22 06:03:39,872 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.72 vs. limit=22.5 2023-11-22 06:03:40,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1824873.3333333333, ans=0.0 2023-11-22 06:03:50,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1824940.0, ans=0.1 2023-11-22 06:03:59,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273750 2023-11-22 06:04:01,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1824940.0, ans=0.125 2023-11-22 06:04:03,977 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.50 vs. limit=15.0 2023-11-22 06:04:12,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.84 vs. limit=15.0 2023-11-22 06:04:12,808 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.119e+01 8.781e+01 9.554e+01 1.352e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 06:04:29,609 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9250, loss[loss=0.07664, simple_loss=0.09963, pruned_loss=0.01773, audio_tagging_loss=0.009088, over 15341.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09343, pruned_loss=0.0152, audio_tagging_loss=0.00933, over 3051453.38 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:04:51,039 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.84 vs. limit=12.0 2023-11-22 06:05:04,370 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=12.0 2023-11-22 06:05:05,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273800 2023-11-22 06:05:05,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1825273.3333333333, ans=0.1 2023-11-22 06:05:18,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1825340.0, ans=0.125 2023-11-22 06:05:35,261 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9300, loss[loss=0.07869, simple_loss=0.106, pruned_loss=0.01824, audio_tagging_loss=0.007469, over 14968.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09335, pruned_loss=0.01527, audio_tagging_loss=0.009318, over 3052313.04 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:06:09,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1825606.6666666667, ans=0.05 2023-11-22 06:06:11,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273850 2023-11-22 06:06:24,303 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.188e+01 8.737e+01 9.420e+01 1.693e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 06:06:40,820 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9350, loss[loss=0.07291, simple_loss=0.09419, pruned_loss=0.016, audio_tagging_loss=0.009817, over 14717.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09353, pruned_loss=0.01522, audio_tagging_loss=0.009311, over 3048629.22 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:06:49,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.81 vs. limit=10.0 2023-11-22 06:07:15,974 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273900 2023-11-22 06:07:17,991 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.63 vs. limit=22.5 2023-11-22 06:07:30,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1826006.6666666667, ans=0.125 2023-11-22 06:07:35,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1826073.3333333333, ans=0.0 2023-11-22 06:07:37,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1826073.3333333333, ans=0.09899494936611666 2023-11-22 06:07:45,843 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9400, loss[loss=0.06271, simple_loss=0.07536, pruned_loss=0.01364, audio_tagging_loss=0.01139, over 15399.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09348, pruned_loss=0.01527, audio_tagging_loss=0.009474, over 3053111.69 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:08:18,720 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.35 vs. limit=15.0 2023-11-22 06:08:20,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 273950 2023-11-22 06:08:34,425 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.127e+01 8.861e+01 9.548e+01 1.246e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-22 06:08:34,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.72 vs. limit=15.0 2023-11-22 06:08:38,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1826406.6666666667, ans=0.07 2023-11-22 06:08:42,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.78 vs. limit=6.0 2023-11-22 06:08:48,510 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:08:50,909 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9450, loss[loss=0.08253, simple_loss=0.1124, pruned_loss=0.01869, audio_tagging_loss=0.007626, over 16521.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09367, pruned_loss=0.01526, audio_tagging_loss=0.009517, over 3054182.82 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:09:10,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1826540.0, ans=0.04949747468305833 2023-11-22 06:09:25,879 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274000 2023-11-22 06:09:55,607 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9500, loss[loss=0.07644, simple_loss=0.1027, pruned_loss=0.01589, audio_tagging_loss=0.009214, over 14859.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09331, pruned_loss=0.01528, audio_tagging_loss=0.009578, over 3063252.43 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:09:59,928 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.15 vs. limit=22.5 2023-11-22 06:10:00,070 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.49 vs. limit=15.0 2023-11-22 06:10:02,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1826806.6666666667, ans=0.2 2023-11-22 06:10:31,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274050 2023-11-22 06:10:35,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1827006.6666666667, ans=0.0 2023-11-22 06:10:43,869 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.476e+01 8.082e+01 8.870e+01 9.586e+01 1.219e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 06:10:45,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1827006.6666666667, ans=0.125 2023-11-22 06:11:01,485 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9550, loss[loss=0.09375, simple_loss=0.1294, pruned_loss=0.0217, audio_tagging_loss=0.007353, over 14805.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.09418, pruned_loss=0.01534, audio_tagging_loss=0.009561, over 3058808.25 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:11:21,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1827206.6666666667, ans=0.125 2023-11-22 06:11:29,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1827273.3333333333, ans=0.09899494936611666 2023-11-22 06:11:34,274 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.21 vs. limit=15.0 2023-11-22 06:11:36,396 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274100 2023-11-22 06:11:42,894 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.33 vs. limit=15.0 2023-11-22 06:12:03,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1827406.6666666667, ans=0.1 2023-11-22 06:12:06,028 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9600, loss[loss=0.06517, simple_loss=0.08527, pruned_loss=0.01355, audio_tagging_loss=0.00898, over 14350.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09439, pruned_loss=0.01532, audio_tagging_loss=0.009613, over 3054488.84 frames. ], batch size: 53, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:12:08,825 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:12:38,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1827606.6666666667, ans=0.0 2023-11-22 06:12:40,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274150 2023-11-22 06:12:53,499 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.292e+01 9.125e+01 9.909e+01 1.275e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-22 06:12:53,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1827673.3333333333, ans=0.125 2023-11-22 06:13:09,794 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9650, loss[loss=0.09014, simple_loss=0.1294, pruned_loss=0.01737, audio_tagging_loss=0.008067, over 17025.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09497, pruned_loss=0.01577, audio_tagging_loss=0.009584, over 3051688.76 frames. ], batch size: 62, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:13:11,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1827806.6666666667, ans=0.2 2023-11-22 06:13:22,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1827873.3333333333, ans=0.125 2023-11-22 06:13:31,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.94 vs. limit=15.0 2023-11-22 06:13:37,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1827940.0, ans=0.0 2023-11-22 06:13:37,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1827940.0, ans=0.2 2023-11-22 06:13:38,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1827940.0, ans=0.2 2023-11-22 06:13:38,598 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:13:45,050 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274200 2023-11-22 06:13:46,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1827940.0, ans=0.5 2023-11-22 06:14:04,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1828073.3333333333, ans=0.125 2023-11-22 06:14:14,382 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9700, loss[loss=0.1121, simple_loss=0.156, pruned_loss=0.0295, audio_tagging_loss=0.004566, over 16279.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09623, pruned_loss=0.01589, audio_tagging_loss=0.009378, over 3065236.16 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:14:34,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0 2023-11-22 06:14:45,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1828273.3333333333, ans=0.0 2023-11-22 06:14:49,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274250 2023-11-22 06:14:58,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.59 vs. limit=10.0 2023-11-22 06:15:01,335 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.325e+01 8.777e+01 9.483e+01 1.567e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-22 06:15:13,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1828406.6666666667, ans=0.125 2023-11-22 06:15:19,172 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9750, loss[loss=0.08333, simple_loss=0.1022, pruned_loss=0.02156, audio_tagging_loss=0.01066, over 14944.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09567, pruned_loss=0.01573, audio_tagging_loss=0.009376, over 3067539.44 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:15:20,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1828473.3333333333, ans=0.125 2023-11-22 06:15:35,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1828540.0, ans=0.07 2023-11-22 06:15:39,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1828540.0, ans=0.1 2023-11-22 06:15:46,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1828606.6666666667, ans=0.0 2023-11-22 06:15:53,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274300 2023-11-22 06:15:56,799 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.11 vs. limit=10.0 2023-11-22 06:16:20,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0 2023-11-22 06:16:23,202 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9800, loss[loss=0.06887, simple_loss=0.08447, pruned_loss=0.01689, audio_tagging_loss=0.00974, over 13976.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09537, pruned_loss=0.0156, audio_tagging_loss=0.009355, over 3061525.04 frames. ], batch size: 53, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:16:50,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1828940.0, ans=0.125 2023-11-22 06:16:51,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1828940.0, ans=0.125 2023-11-22 06:16:58,262 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274350 2023-11-22 06:17:10,629 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.412e+01 8.125e+01 8.803e+01 9.670e+01 1.768e+02, threshold=1.761e+02, percent-clipped=1.0 2023-11-22 06:17:19,171 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.98 vs. limit=15.0 2023-11-22 06:17:19,779 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:17:19,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1829073.3333333333, ans=0.2 2023-11-22 06:17:21,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1829073.3333333333, ans=0.125 2023-11-22 06:17:27,060 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9850, loss[loss=0.08553, simple_loss=0.1184, pruned_loss=0.01763, audio_tagging_loss=0.008707, over 17014.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09546, pruned_loss=0.01552, audio_tagging_loss=0.009333, over 3059577.79 frames. ], batch size: 61, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:18:01,806 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274400 2023-11-22 06:18:04,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1829340.0, ans=0.125 2023-11-22 06:18:08,802 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.72 vs. limit=15.0 2023-11-22 06:18:14,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1829340.0, ans=0.2 2023-11-22 06:18:20,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1829406.6666666667, ans=0.0 2023-11-22 06:18:31,227 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9900, loss[loss=0.07679, simple_loss=0.1029, pruned_loss=0.01845, audio_tagging_loss=0.006903, over 15661.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09495, pruned_loss=0.01539, audio_tagging_loss=0.009357, over 3059865.01 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:18:55,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1829606.6666666667, ans=0.0 2023-11-22 06:19:05,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274450 2023-11-22 06:19:10,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.60 vs. limit=12.0 2023-11-22 06:19:17,951 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.069e+01 8.830e+01 9.400e+01 1.279e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 06:19:23,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1829740.0, ans=0.125 2023-11-22 06:19:23,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1829740.0, ans=0.125 2023-11-22 06:19:35,137 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 9950, loss[loss=0.07154, simple_loss=0.09613, pruned_loss=0.01357, audio_tagging_loss=0.0099, over 16945.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09471, pruned_loss=0.0152, audio_tagging_loss=0.009327, over 3057035.19 frames. ], batch size: 64, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:19:51,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1829873.3333333333, ans=0.0 2023-11-22 06:20:06,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1829940.0, ans=0.125 2023-11-22 06:20:09,486 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274500 2023-11-22 06:20:19,762 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:20:19,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1830006.6666666667, ans=0.0 2023-11-22 06:20:27,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.05 vs. limit=15.0 2023-11-22 06:20:39,464 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10000, loss[loss=0.1005, simple_loss=0.1334, pruned_loss=0.0257, audio_tagging_loss=0.008104, over 15798.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.0949, pruned_loss=0.01539, audio_tagging_loss=0.009318, over 3049856.52 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:20:45,431 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0 2023-11-22 06:20:56,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1830206.6666666667, ans=0.2 2023-11-22 06:21:14,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274550 2023-11-22 06:21:28,399 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.149e+01 8.699e+01 9.689e+01 1.229e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 06:21:43,770 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10050, loss[loss=0.07303, simple_loss=0.09219, pruned_loss=0.01863, audio_tagging_loss=0.008308, over 13954.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.09488, pruned_loss=0.01551, audio_tagging_loss=0.009444, over 3047165.78 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:21:59,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1830540.0, ans=0.0 2023-11-22 06:22:02,501 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:22:05,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1830540.0, ans=0.0 2023-11-22 06:22:11,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1830606.6666666667, ans=0.0 2023-11-22 06:22:18,439 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274600 2023-11-22 06:22:41,738 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:22:47,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1830806.6666666667, ans=0.0 2023-11-22 06:22:48,259 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10100, loss[loss=0.06692, simple_loss=0.08622, pruned_loss=0.0122, audio_tagging_loss=0.01161, over 15284.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09546, pruned_loss=0.01561, audio_tagging_loss=0.009444, over 3046302.54 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:23:05,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.22 vs. limit=15.0 2023-11-22 06:23:19,357 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.72 vs. limit=15.0 2023-11-22 06:23:19,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1830940.0, ans=0.125 2023-11-22 06:23:22,233 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274650 2023-11-22 06:23:36,109 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 8.114e+01 8.687e+01 9.390e+01 1.135e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-22 06:23:39,247 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:23:41,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=23.71 vs. limit=22.5 2023-11-22 06:23:41,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.78 vs. limit=22.5 2023-11-22 06:23:52,056 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10150, loss[loss=0.1059, simple_loss=0.1403, pruned_loss=0.02898, audio_tagging_loss=0.006793, over 15440.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09573, pruned_loss=0.01558, audio_tagging_loss=0.009384, over 3048041.46 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:23:52,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.71 vs. limit=15.0 2023-11-22 06:24:01,933 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.30 vs. limit=10.0 2023-11-22 06:24:03,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1831206.6666666667, ans=0.125 2023-11-22 06:24:06,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1831206.6666666667, ans=0.0 2023-11-22 06:24:10,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1831206.6666666667, ans=0.1 2023-11-22 06:24:21,558 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:24:22,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1831273.3333333333, ans=0.0 2023-11-22 06:24:26,632 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274700 2023-11-22 06:24:32,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1831340.0, ans=0.0 2023-11-22 06:24:56,404 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10200, loss[loss=0.06966, simple_loss=0.08658, pruned_loss=0.01755, audio_tagging_loss=0.008828, over 15471.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09504, pruned_loss=0.01553, audio_tagging_loss=0.009482, over 3042746.05 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:25:19,850 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:25:31,352 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274750 2023-11-22 06:25:41,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1831673.3333333333, ans=0.0 2023-11-22 06:25:45,209 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.234e+01 8.918e+01 9.479e+01 1.480e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 06:25:52,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1831740.0, ans=0.125 2023-11-22 06:25:59,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1831806.6666666667, ans=0.0 2023-11-22 06:26:00,408 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10250, loss[loss=0.08178, simple_loss=0.1031, pruned_loss=0.02229, audio_tagging_loss=0.007938, over 16072.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09398, pruned_loss=0.01537, audio_tagging_loss=0.009532, over 3043229.73 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:26:16,363 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:26:16,443 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:26:28,314 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.50 vs. limit=15.0 2023-11-22 06:26:32,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1831940.0, ans=0.0 2023-11-22 06:26:35,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274800 2023-11-22 06:26:52,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1832073.3333333333, ans=10.0 2023-11-22 06:27:05,159 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10300, loss[loss=0.05845, simple_loss=0.06703, pruned_loss=0.01258, audio_tagging_loss=0.01236, over 14878.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09318, pruned_loss=0.01516, audio_tagging_loss=0.009567, over 3052029.38 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:27:07,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1832140.0, ans=0.125 2023-11-22 06:27:17,700 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.08 vs. limit=15.0 2023-11-22 06:27:30,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0 2023-11-22 06:27:31,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1832273.3333333333, ans=0.125 2023-11-22 06:27:39,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274850 2023-11-22 06:27:39,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1832273.3333333333, ans=0.125 2023-11-22 06:27:43,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1832340.0, ans=0.0 2023-11-22 06:27:43,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1832340.0, ans=0.125 2023-11-22 06:27:50,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1832340.0, ans=0.1 2023-11-22 06:27:53,800 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.276e+01 9.014e+01 9.790e+01 1.332e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-22 06:27:56,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1832406.6666666667, ans=0.125 2023-11-22 06:28:04,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1832406.6666666667, ans=0.0 2023-11-22 06:28:08,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1832473.3333333333, ans=0.125 2023-11-22 06:28:09,399 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10350, loss[loss=0.07802, simple_loss=0.1024, pruned_loss=0.01771, audio_tagging_loss=0.009123, over 15491.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09409, pruned_loss=0.01535, audio_tagging_loss=0.009613, over 3048086.55 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0 2023-11-22 06:28:23,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1832540.0, ans=0.1 2023-11-22 06:28:44,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274900 2023-11-22 06:28:44,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1832606.6666666667, ans=0.025 2023-11-22 06:28:45,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1832606.6666666667, ans=0.5 2023-11-22 06:28:58,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1832673.3333333333, ans=0.125 2023-11-22 06:28:58,527 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.33 vs. limit=22.5 2023-11-22 06:29:12,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1832806.6666666667, ans=0.125 2023-11-22 06:29:12,650 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.20 vs. limit=12.0 2023-11-22 06:29:13,107 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10400, loss[loss=0.08101, simple_loss=0.1038, pruned_loss=0.01984, audio_tagging_loss=0.009289, over 13995.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09449, pruned_loss=0.01539, audio_tagging_loss=0.009613, over 3050084.94 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:29:23,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1832806.6666666667, ans=0.05 2023-11-22 06:29:47,781 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 274950 2023-11-22 06:29:58,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1833006.6666666667, ans=0.125 2023-11-22 06:30:01,650 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.321e+01 8.235e+01 8.763e+01 9.443e+01 1.759e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-22 06:30:01,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1833006.6666666667, ans=0.0 2023-11-22 06:30:10,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1833073.3333333333, ans=0.125 2023-11-22 06:30:17,052 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10450, loss[loss=0.07358, simple_loss=0.09697, pruned_loss=0.01405, audio_tagging_loss=0.01104, over 14309.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09404, pruned_loss=0.01529, audio_tagging_loss=0.009656, over 3051361.85 frames. ], batch size: 52, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:30:43,908 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.25 vs. limit=10.0 2023-11-22 06:30:51,975 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275000 2023-11-22 06:31:02,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1833340.0, ans=0.125 2023-11-22 06:31:06,774 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.51 vs. limit=12.0 2023-11-22 06:31:08,067 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.73 vs. limit=22.5 2023-11-22 06:31:18,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.51 vs. limit=22.5 2023-11-22 06:31:21,973 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10500, loss[loss=0.07675, simple_loss=0.1021, pruned_loss=0.01595, audio_tagging_loss=0.00975, over 14640.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.095, pruned_loss=0.01545, audio_tagging_loss=0.009524, over 3054852.39 frames. ], batch size: 53, lr: 2.95e-03, grad_scale: 32.0 2023-11-22 06:31:23,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1833473.3333333333, ans=0.0 2023-11-22 06:31:31,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1833473.3333333333, ans=0.0 2023-11-22 06:31:38,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.99 vs. limit=22.5 2023-11-22 06:31:45,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1833540.0, ans=0.2 2023-11-22 06:31:45,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1833540.0, ans=0.04949747468305833 2023-11-22 06:31:50,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1833606.6666666667, ans=0.125 2023-11-22 06:31:56,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275050 2023-11-22 06:32:07,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.06 vs. limit=15.0 2023-11-22 06:32:10,953 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.246e+01 8.781e+01 9.587e+01 1.309e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 06:32:13,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1833740.0, ans=0.1 2023-11-22 06:32:14,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.29 vs. limit=15.0 2023-11-22 06:32:26,270 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10550, loss[loss=0.07405, simple_loss=0.09592, pruned_loss=0.01615, audio_tagging_loss=0.009948, over 14335.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09454, pruned_loss=0.01531, audio_tagging_loss=0.009448, over 3054164.48 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:32:44,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1833873.3333333333, ans=0.125 2023-11-22 06:33:00,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275100 2023-11-22 06:33:03,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1834006.6666666667, ans=0.0 2023-11-22 06:33:29,418 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10600, loss[loss=0.06063, simple_loss=0.07451, pruned_loss=0.01248, audio_tagging_loss=0.0109, over 15547.00 frames. ], tot_loss[loss=0.07164, simple_loss=0.0941, pruned_loss=0.01518, audio_tagging_loss=0.009406, over 3048652.86 frames. ], batch size: 59, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:33:47,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1834206.6666666667, ans=0.0 2023-11-22 06:34:05,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275150 2023-11-22 06:34:19,497 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.081e+01 8.805e+01 9.218e+01 1.246e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-22 06:34:29,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1834406.6666666667, ans=0.125 2023-11-22 06:34:36,106 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10650, loss[loss=0.05581, simple_loss=0.07371, pruned_loss=0.008102, audio_tagging_loss=0.01085, over 15372.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09432, pruned_loss=0.01534, audio_tagging_loss=0.009334, over 3046422.19 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:34:40,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1834473.3333333333, ans=0.125 2023-11-22 06:34:48,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1834540.0, ans=0.125 2023-11-22 06:34:50,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1834540.0, ans=0.05 2023-11-22 06:34:56,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1834540.0, ans=0.125 2023-11-22 06:35:10,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275200 2023-11-22 06:35:23,734 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:35:27,865 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.24 vs. limit=12.0 2023-11-22 06:35:30,396 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.74 vs. limit=22.5 2023-11-22 06:35:35,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1834740.0, ans=0.125 2023-11-22 06:35:41,430 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10700, loss[loss=0.06445, simple_loss=0.07915, pruned_loss=0.01458, audio_tagging_loss=0.01029, over 15224.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09459, pruned_loss=0.01534, audio_tagging_loss=0.009357, over 3053431.13 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:35:52,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1834873.3333333333, ans=0.125 2023-11-22 06:35:56,927 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.86 vs. limit=22.5 2023-11-22 06:35:59,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0 2023-11-22 06:36:10,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1834940.0, ans=0.2 2023-11-22 06:36:13,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1834940.0, ans=0.2 2023-11-22 06:36:15,695 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275250 2023-11-22 06:36:17,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1834940.0, ans=0.2 2023-11-22 06:36:30,355 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.298e+01 8.136e+01 8.812e+01 9.444e+01 1.567e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 06:36:35,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1835073.3333333333, ans=0.2 2023-11-22 06:36:45,213 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10750, loss[loss=0.05829, simple_loss=0.08577, pruned_loss=0.006687, audio_tagging_loss=0.008712, over 16740.00 frames. ], tot_loss[loss=0.07184, simple_loss=0.09445, pruned_loss=0.01527, audio_tagging_loss=0.00935, over 3048921.67 frames. ], batch size: 63, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:36:56,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.79 vs. limit=22.5 2023-11-22 06:37:11,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1835273.3333333333, ans=0.0 2023-11-22 06:37:11,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1835273.3333333333, ans=0.2 2023-11-22 06:37:21,029 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275300 2023-11-22 06:37:31,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1835340.0, ans=0.125 2023-11-22 06:37:35,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1835406.6666666667, ans=0.2 2023-11-22 06:37:48,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1835473.3333333333, ans=0.0 2023-11-22 06:37:49,823 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10800, loss[loss=0.0626, simple_loss=0.08161, pruned_loss=0.01242, audio_tagging_loss=0.009378, over 15677.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09388, pruned_loss=0.01522, audio_tagging_loss=0.009367, over 3054468.59 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:38:14,874 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.86 vs. limit=22.5 2023-11-22 06:38:25,508 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275350 2023-11-22 06:38:41,119 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.004e+01 8.229e+01 8.855e+01 9.329e+01 1.142e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-22 06:38:52,977 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.42 vs. limit=15.0 2023-11-22 06:38:56,677 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10850, loss[loss=0.07148, simple_loss=0.1007, pruned_loss=0.0122, audio_tagging_loss=0.008924, over 15650.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09359, pruned_loss=0.01515, audio_tagging_loss=0.009424, over 3052206.02 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:39:05,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1835806.6666666667, ans=0.125 2023-11-22 06:39:06,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0 2023-11-22 06:39:07,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1835806.6666666667, ans=0.1 2023-11-22 06:39:23,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1835940.0, ans=0.125 2023-11-22 06:39:31,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275400 2023-11-22 06:39:45,340 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:39:47,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=15.0 2023-11-22 06:39:53,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1836073.3333333333, ans=0.125 2023-11-22 06:39:56,758 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:40:01,518 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10900, loss[loss=0.06703, simple_loss=0.08684, pruned_loss=0.0157, audio_tagging_loss=0.007907, over 15078.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09352, pruned_loss=0.01523, audio_tagging_loss=0.009525, over 3053924.90 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:40:05,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1836140.0, ans=0.125 2023-11-22 06:40:05,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1836140.0, ans=0.125 2023-11-22 06:40:14,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1836206.6666666667, ans=0.0 2023-11-22 06:40:28,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1836273.3333333333, ans=0.125 2023-11-22 06:40:37,341 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275450 2023-11-22 06:40:44,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1836340.0, ans=0.125 2023-11-22 06:40:51,643 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.279e+01 8.746e+01 9.747e+01 1.923e+02, threshold=1.749e+02, percent-clipped=1.0 2023-11-22 06:40:59,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.54 vs. limit=15.0 2023-11-22 06:41:05,880 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 10950, loss[loss=0.07161, simple_loss=0.0917, pruned_loss=0.01523, audio_tagging_loss=0.01053, over 14999.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09346, pruned_loss=0.01524, audio_tagging_loss=0.009548, over 3048021.72 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:41:15,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1836473.3333333333, ans=0.125 2023-11-22 06:41:31,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1836606.6666666667, ans=0.1 2023-11-22 06:41:40,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275500 2023-11-22 06:41:53,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1836673.3333333333, ans=0.2 2023-11-22 06:41:54,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.55 vs. limit=15.0 2023-11-22 06:42:06,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1836740.0, ans=0.0 2023-11-22 06:42:09,621 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11000, loss[loss=0.06879, simple_loss=0.09803, pruned_loss=0.01111, audio_tagging_loss=0.008674, over 14735.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09367, pruned_loss=0.01526, audio_tagging_loss=0.00946, over 3041520.00 frames. ], batch size: 59, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:42:19,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1836806.6666666667, ans=0.0 2023-11-22 06:42:20,676 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:42:44,020 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=22.5 2023-11-22 06:42:44,582 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275550 2023-11-22 06:42:53,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1837006.6666666667, ans=0.2 2023-11-22 06:43:00,978 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.356e+01 8.870e+01 9.515e+01 1.517e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 06:43:06,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1837073.3333333333, ans=0.035 2023-11-22 06:43:08,086 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.46 vs. limit=15.0 2023-11-22 06:43:14,626 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11050, loss[loss=0.06684, simple_loss=0.0868, pruned_loss=0.01373, audio_tagging_loss=0.009709, over 14950.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09408, pruned_loss=0.01529, audio_tagging_loss=0.009592, over 3043630.00 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:43:15,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1837140.0, ans=0.125 2023-11-22 06:43:16,692 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0 2023-11-22 06:43:36,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1837206.6666666667, ans=0.125 2023-11-22 06:43:38,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1837206.6666666667, ans=0.125 2023-11-22 06:43:49,003 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275600 2023-11-22 06:44:05,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1837406.6666666667, ans=0.125 2023-11-22 06:44:05,419 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.37 vs. limit=15.0 2023-11-22 06:44:09,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1837406.6666666667, ans=0.0 2023-11-22 06:44:18,967 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11100, loss[loss=0.08594, simple_loss=0.1099, pruned_loss=0.02269, audio_tagging_loss=0.008319, over 14347.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09471, pruned_loss=0.01546, audio_tagging_loss=0.009647, over 3045472.57 frames. ], batch size: 53, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:44:31,150 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.47 vs. limit=15.0 2023-11-22 06:44:34,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1837540.0, ans=0.5 2023-11-22 06:44:39,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1837540.0, ans=0.125 2023-11-22 06:44:50,726 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.34 vs. limit=15.0 2023-11-22 06:44:53,795 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275650 2023-11-22 06:45:11,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.66 vs. limit=22.5 2023-11-22 06:45:11,537 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.680e+01 8.196e+01 8.803e+01 9.419e+01 1.192e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-22 06:45:22,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1837806.6666666667, ans=0.0 2023-11-22 06:45:23,227 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11150, loss[loss=0.06882, simple_loss=0.09137, pruned_loss=0.0143, audio_tagging_loss=0.008836, over 14508.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09444, pruned_loss=0.01549, audio_tagging_loss=0.009776, over 3037001.50 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 8.0 2023-11-22 06:45:35,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1837873.3333333333, ans=0.125 2023-11-22 06:45:38,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1837873.3333333333, ans=0.1 2023-11-22 06:45:44,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1837873.3333333333, ans=0.1 2023-11-22 06:45:48,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1837940.0, ans=0.125 2023-11-22 06:45:58,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275700 2023-11-22 06:46:20,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1838073.3333333333, ans=0.125 2023-11-22 06:46:26,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1838140.0, ans=0.1 2023-11-22 06:46:27,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=11.14 vs. limit=12.0 2023-11-22 06:46:28,374 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11200, loss[loss=0.06049, simple_loss=0.07006, pruned_loss=0.01328, audio_tagging_loss=0.01219, over 14400.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09377, pruned_loss=0.01539, audio_tagging_loss=0.0098, over 3032528.76 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:46:45,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1838206.6666666667, ans=0.0 2023-11-22 06:46:58,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=1838273.3333333333, ans=0.02 2023-11-22 06:47:03,001 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275750 2023-11-22 06:47:20,588 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.15 vs. limit=12.0 2023-11-22 06:47:21,150 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.197e+01 8.985e+01 9.663e+01 1.096e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 06:47:32,685 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11250, loss[loss=0.08544, simple_loss=0.1198, pruned_loss=0.01782, audio_tagging_loss=0.007732, over 16021.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.09388, pruned_loss=0.01531, audio_tagging_loss=0.009748, over 3036451.99 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:47:34,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1838473.3333333333, ans=0.125 2023-11-22 06:47:37,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1838473.3333333333, ans=10.0 2023-11-22 06:47:39,614 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.39 vs. limit=22.5 2023-11-22 06:47:41,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1838473.3333333333, ans=0.125 2023-11-22 06:47:47,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1838540.0, ans=0.125 2023-11-22 06:48:00,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1838606.6666666667, ans=0.0 2023-11-22 06:48:04,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1838606.6666666667, ans=0.125 2023-11-22 06:48:07,775 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275800 2023-11-22 06:48:38,273 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11300, loss[loss=0.059, simple_loss=0.07197, pruned_loss=0.01204, audio_tagging_loss=0.01097, over 14760.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09429, pruned_loss=0.01548, audio_tagging_loss=0.00956, over 3032222.81 frames. ], batch size: 61, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:48:42,086 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:48:45,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1838806.6666666667, ans=0.015 2023-11-22 06:48:45,625 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:48:48,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1838806.6666666667, ans=0.125 2023-11-22 06:48:50,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1838873.3333333333, ans=0.125 2023-11-22 06:48:52,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1838873.3333333333, ans=0.0 2023-11-22 06:49:02,551 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.55 vs. limit=15.0 2023-11-22 06:49:08,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1838940.0, ans=0.1 2023-11-22 06:49:13,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275850 2023-11-22 06:49:21,477 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.20 vs. limit=15.0 2023-11-22 06:49:25,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1839006.6666666667, ans=0.0 2023-11-22 06:49:31,029 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 7.924e+01 8.561e+01 9.574e+01 1.338e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-22 06:49:42,776 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11350, loss[loss=0.07098, simple_loss=0.09792, pruned_loss=0.01352, audio_tagging_loss=0.008497, over 15060.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09439, pruned_loss=0.0154, audio_tagging_loss=0.009411, over 3036845.98 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:49:44,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.56 vs. limit=22.5 2023-11-22 06:50:05,047 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 06:50:17,562 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275900 2023-11-22 06:50:17,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1839273.3333333333, ans=0.125 2023-11-22 06:50:38,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1839406.6666666667, ans=0.1 2023-11-22 06:50:42,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1839406.6666666667, ans=0.125 2023-11-22 06:50:47,671 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11400, loss[loss=0.07996, simple_loss=0.1028, pruned_loss=0.01925, audio_tagging_loss=0.009289, over 14269.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09511, pruned_loss=0.01532, audio_tagging_loss=0.009286, over 3035522.72 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:50:49,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1839473.3333333333, ans=0.0 2023-11-22 06:50:50,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1839473.3333333333, ans=0.125 2023-11-22 06:51:18,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1839606.6666666667, ans=0.125 2023-11-22 06:51:22,127 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 275950 2023-11-22 06:51:30,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1839673.3333333333, ans=0.0 2023-11-22 06:51:40,305 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.209e+01 8.167e+01 8.873e+01 9.595e+01 1.109e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 06:51:49,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1839740.0, ans=0.2 2023-11-22 06:51:51,934 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11450, loss[loss=0.07328, simple_loss=0.09623, pruned_loss=0.0176, audio_tagging_loss=0.00757, over 14461.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09512, pruned_loss=0.01544, audio_tagging_loss=0.009334, over 3036440.47 frames. ], batch size: 54, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:51:52,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1839806.6666666667, ans=0.0 2023-11-22 06:52:15,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1839873.3333333333, ans=0.05 2023-11-22 06:52:26,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1839940.0, ans=0.07 2023-11-22 06:52:27,313 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276000 2023-11-22 06:52:33,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1840006.6666666667, ans=0.07 2023-11-22 06:52:34,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.41 vs. limit=15.0 2023-11-22 06:52:59,780 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11500, loss[loss=0.0673, simple_loss=0.09237, pruned_loss=0.01157, audio_tagging_loss=0.009546, over 15432.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09474, pruned_loss=0.01538, audio_tagging_loss=0.009278, over 3040730.44 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:53:11,637 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.34 vs. limit=15.0 2023-11-22 06:53:19,927 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.44 vs. limit=6.0 2023-11-22 06:53:28,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1840273.3333333333, ans=0.1 2023-11-22 06:53:30,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1840273.3333333333, ans=0.1 2023-11-22 06:53:34,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276050 2023-11-22 06:53:35,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=15.0 2023-11-22 06:53:38,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1840340.0, ans=0.125 2023-11-22 06:53:52,667 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.162e+01 8.710e+01 9.515e+01 1.226e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-22 06:53:54,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1840406.6666666667, ans=0.1 2023-11-22 06:54:03,889 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11550, loss[loss=0.06263, simple_loss=0.07689, pruned_loss=0.01174, audio_tagging_loss=0.01244, over 14927.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09524, pruned_loss=0.01546, audio_tagging_loss=0.009239, over 3039624.82 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:54:39,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.80 vs. limit=12.0 2023-11-22 06:54:39,931 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276100 2023-11-22 06:54:41,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1840606.6666666667, ans=0.125 2023-11-22 06:54:44,936 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 06:54:51,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1840673.3333333333, ans=0.125 2023-11-22 06:54:58,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1840740.0, ans=0.0 2023-11-22 06:55:03,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1840740.0, ans=0.1 2023-11-22 06:55:09,737 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11600, loss[loss=0.07184, simple_loss=0.09721, pruned_loss=0.01427, audio_tagging_loss=0.008966, over 13841.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09436, pruned_loss=0.0153, audio_tagging_loss=0.009251, over 3036762.34 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:55:14,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1840806.6666666667, ans=0.125 2023-11-22 06:55:16,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1840806.6666666667, ans=0.0 2023-11-22 06:55:16,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1840806.6666666667, ans=0.125 2023-11-22 06:55:32,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1840873.3333333333, ans=0.0 2023-11-22 06:55:32,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1840873.3333333333, ans=0.0 2023-11-22 06:55:36,349 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.51 vs. limit=22.5 2023-11-22 06:55:42,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.52 vs. limit=15.0 2023-11-22 06:55:44,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276150 2023-11-22 06:55:55,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1841006.6666666667, ans=0.125 2023-11-22 06:56:01,634 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.62 vs. limit=22.5 2023-11-22 06:56:03,215 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.255e+01 8.370e+01 8.963e+01 9.598e+01 1.244e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-22 06:56:15,229 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11650, loss[loss=0.05266, simple_loss=0.0676, pruned_loss=0.009688, audio_tagging_loss=0.009171, over 16332.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.0945, pruned_loss=0.01533, audio_tagging_loss=0.009289, over 3039982.57 frames. ], batch size: 65, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:56:34,491 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.64 vs. limit=22.5 2023-11-22 06:56:50,391 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276200 2023-11-22 06:56:50,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1841273.3333333333, ans=0.05 2023-11-22 06:56:51,059 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.26 vs. limit=22.5 2023-11-22 06:56:53,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1841340.0, ans=0.125 2023-11-22 06:56:55,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1841340.0, ans=0.125 2023-11-22 06:56:55,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1841340.0, ans=0.125 2023-11-22 06:57:00,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1841340.0, ans=0.125 2023-11-22 06:57:09,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1841406.6666666667, ans=0.1 2023-11-22 06:57:09,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1841406.6666666667, ans=0.125 2023-11-22 06:57:19,043 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11700, loss[loss=0.06958, simple_loss=0.09096, pruned_loss=0.01478, audio_tagging_loss=0.009324, over 14725.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09408, pruned_loss=0.01518, audio_tagging_loss=0.009226, over 3039487.98 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:57:34,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1841540.0, ans=0.04949747468305833 2023-11-22 06:57:43,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1841540.0, ans=0.0 2023-11-22 06:57:45,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1841606.6666666667, ans=0.125 2023-11-22 06:57:54,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276250 2023-11-22 06:58:11,820 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.445e+01 9.108e+01 1.005e+02 1.343e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-22 06:58:14,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1841740.0, ans=0.0 2023-11-22 06:58:24,068 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11750, loss[loss=0.07466, simple_loss=0.09318, pruned_loss=0.01536, audio_tagging_loss=0.0127, over 16461.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09334, pruned_loss=0.01515, audio_tagging_loss=0.009422, over 3039720.02 frames. ], batch size: 60, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 06:58:27,125 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0 2023-11-22 06:58:36,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1841873.3333333333, ans=0.125 2023-11-22 06:58:38,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1841873.3333333333, ans=0.0 2023-11-22 06:58:39,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1841873.3333333333, ans=0.125 2023-11-22 06:58:58,288 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276300 2023-11-22 06:59:02,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1842006.6666666667, ans=0.2 2023-11-22 06:59:20,498 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.41 vs. limit=22.5 2023-11-22 06:59:28,425 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11800, loss[loss=0.06761, simple_loss=0.09939, pruned_loss=0.01091, audio_tagging_loss=0.007009, over 15076.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09395, pruned_loss=0.01514, audio_tagging_loss=0.009339, over 3037463.00 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 06:59:39,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1842140.0, ans=0.1 2023-11-22 06:59:57,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1842273.3333333333, ans=0.125 2023-11-22 07:00:03,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276350 2023-11-22 07:00:21,957 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.506e-03 2023-11-22 07:00:24,082 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.151e+01 8.673e+01 9.167e+01 1.144e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-22 07:00:29,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1842406.6666666667, ans=0.125 2023-11-22 07:00:34,106 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11850, loss[loss=0.08886, simple_loss=0.1228, pruned_loss=0.01783, audio_tagging_loss=0.009648, over 15496.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09364, pruned_loss=0.01501, audio_tagging_loss=0.009394, over 3039772.16 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 07:00:39,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1842473.3333333333, ans=0.125 2023-11-22 07:00:49,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1842540.0, ans=0.0 2023-11-22 07:00:52,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1842540.0, ans=0.1 2023-11-22 07:01:08,907 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276400 2023-11-22 07:01:12,466 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:01:16,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1842673.3333333333, ans=0.0 2023-11-22 07:01:24,141 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0 2023-11-22 07:01:36,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1842740.0, ans=0.125 2023-11-22 07:01:38,739 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11900, loss[loss=0.1077, simple_loss=0.1464, pruned_loss=0.0286, audio_tagging_loss=0.005922, over 15632.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.09391, pruned_loss=0.015, audio_tagging_loss=0.009538, over 3043320.35 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 07:01:51,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1842873.3333333333, ans=0.2 2023-11-22 07:01:55,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1842873.3333333333, ans=0.0 2023-11-22 07:02:09,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1842940.0, ans=0.0 2023-11-22 07:02:14,123 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276450 2023-11-22 07:02:33,097 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 7.969e+01 8.417e+01 9.242e+01 1.141e+02, threshold=1.683e+02, percent-clipped=0.0 2023-11-22 07:02:37,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.58 vs. limit=15.0 2023-11-22 07:02:41,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1843073.3333333333, ans=0.125 2023-11-22 07:02:43,984 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 11950, loss[loss=0.07271, simple_loss=0.1003, pruned_loss=0.01404, audio_tagging_loss=0.008546, over 15492.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09365, pruned_loss=0.01481, audio_tagging_loss=0.009497, over 3044893.61 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0 2023-11-22 07:03:11,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1843273.3333333333, ans=0.1 2023-11-22 07:03:15,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1843273.3333333333, ans=0.125 2023-11-22 07:03:18,552 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276500 2023-11-22 07:03:21,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1843340.0, ans=0.1 2023-11-22 07:03:36,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1843406.6666666667, ans=0.125 2023-11-22 07:03:46,900 INFO [train_asr.py:1221] (3/4) Epoch 23, batch 12000, loss[loss=0.07087, simple_loss=0.1025, pruned_loss=0.01067, audio_tagging_loss=0.008935, over 15977.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09376, pruned_loss=0.01494, audio_tagging_loss=0.009631, over 3043346.95 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 32.0 2023-11-22 07:03:46,901 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 07:04:27,925 INFO [train_asr.py:1253] (3/4) Epoch 23, validation: loss=0.05966, simple_loss=0.05174, pruned_loss=0.005186, audio_tagging_loss=0.02861, over 4681554.00 frames. 2023-11-22 07:04:27,926 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 07:04:46,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1843540.0, ans=0.0 2023-11-22 07:04:54,456 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.92 vs. limit=15.0 2023-11-22 07:05:34,202 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 0, loss[loss=0.08667, simple_loss=0.09266, pruned_loss=0.0154, audio_tagging_loss=0.02493, over 15422.00 frames. ], tot_loss[loss=0.08667, simple_loss=0.09266, pruned_loss=0.0154, audio_tagging_loss=0.02493, over 15422.00 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:05:34,203 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 07:05:56,464 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.7815, 1.3991, 3.5968, 3.1797, 3.0118, 3.3401, 3.0380, 3.4084], device='cuda:3') 2023-11-22 07:05:59,299 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6136, 3.6386, 3.8151, 3.4026], device='cuda:3') 2023-11-22 07:06:06,479 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2418, 4.1789, 4.4079, 4.4364], device='cuda:3') 2023-11-22 07:06:09,676 INFO [train_asr.py:1253] (3/4) Epoch 24, validation: loss=0.05907, simple_loss=0.05179, pruned_loss=0.005258, audio_tagging_loss=0.02792, over 4681554.00 frames. 2023-11-22 07:06:09,677 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 07:06:12,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1843633.3333333333, ans=0.125 2023-11-22 07:06:13,338 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276550 2023-11-22 07:06:14,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1843633.3333333333, ans=0.1 2023-11-22 07:06:18,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1843633.3333333333, ans=0.0 2023-11-22 07:06:24,568 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.49 vs. limit=15.0 2023-11-22 07:06:32,162 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.350e+01 8.889e+01 9.652e+01 1.254e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 07:06:39,103 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.36 vs. limit=12.0 2023-11-22 07:06:45,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1843766.6666666667, ans=0.1 2023-11-22 07:06:58,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.36 vs. limit=15.0 2023-11-22 07:07:07,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1843900.0, ans=0.0 2023-11-22 07:07:10,732 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.37 vs. limit=22.5 2023-11-22 07:07:13,679 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 50, loss[loss=0.06962, simple_loss=0.07948, pruned_loss=0.01061, audio_tagging_loss=0.01927, over 15196.00 frames. ], tot_loss[loss=0.08209, simple_loss=0.09746, pruned_loss=0.01539, audio_tagging_loss=0.01798, over 689314.70 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:07:16,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1843966.6666666667, ans=0.125 2023-11-22 07:07:17,390 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276600 2023-11-22 07:08:04,710 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.64 vs. limit=15.0 2023-11-22 07:08:13,323 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.61 vs. limit=22.5 2023-11-22 07:08:17,339 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 100, loss[loss=0.07613, simple_loss=0.1036, pruned_loss=0.01367, audio_tagging_loss=0.01067, over 15578.00 frames. ], tot_loss[loss=0.07921, simple_loss=0.09415, pruned_loss=0.01476, audio_tagging_loss=0.01738, over 1208266.66 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:08:21,163 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276650 2023-11-22 07:08:35,142 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.90 vs. limit=15.0 2023-11-22 07:08:35,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1844366.6666666667, ans=0.0 2023-11-22 07:08:40,999 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.235e+01 8.891e+01 9.359e+01 9.974e+01 1.363e+02, threshold=1.872e+02, percent-clipped=0.0 2023-11-22 07:09:22,240 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 150, loss[loss=0.06629, simple_loss=0.08845, pruned_loss=0.01228, audio_tagging_loss=0.009785, over 15118.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09398, pruned_loss=0.01482, audio_tagging_loss=0.01547, over 1614839.62 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:09:26,055 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276700 2023-11-22 07:09:38,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1844700.0, ans=0.1 2023-11-22 07:09:47,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1844766.6666666667, ans=0.125 2023-11-22 07:09:59,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1844833.3333333333, ans=0.0 2023-11-22 07:10:01,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.49 vs. limit=6.0 2023-11-22 07:10:02,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1844833.3333333333, ans=0.025 2023-11-22 07:10:02,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1844833.3333333333, ans=0.125 2023-11-22 07:10:16,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1844900.0, ans=0.0 2023-11-22 07:10:27,708 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 200, loss[loss=0.05828, simple_loss=0.07898, pruned_loss=0.01258, audio_tagging_loss=0.006212, over 14455.00 frames. ], tot_loss[loss=0.07571, simple_loss=0.09387, pruned_loss=0.01509, audio_tagging_loss=0.01369, over 1924367.94 frames. ], batch size: 52, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:10:31,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276750 2023-11-22 07:10:51,669 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.378e+01 9.056e+01 9.976e+01 1.254e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-22 07:10:58,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1845100.0, ans=0.125 2023-11-22 07:11:19,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1845233.3333333333, ans=0.125 2023-11-22 07:11:21,117 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.14 vs. limit=15.0 2023-11-22 07:11:31,357 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 250, loss[loss=0.05093, simple_loss=0.06416, pruned_loss=0.006389, audio_tagging_loss=0.01246, over 15428.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.0939, pruned_loss=0.01517, audio_tagging_loss=0.01235, over 2173608.05 frames. ], batch size: 60, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:11:31,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1845300.0, ans=0.0 2023-11-22 07:11:33,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0 2023-11-22 07:11:35,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276800 2023-11-22 07:11:41,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1845300.0, ans=0.1 2023-11-22 07:11:43,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1845366.6666666667, ans=0.0 2023-11-22 07:11:43,900 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0 2023-11-22 07:11:59,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1845433.3333333333, ans=0.05 2023-11-22 07:12:01,531 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:12:08,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1845433.3333333333, ans=0.2 2023-11-22 07:12:36,691 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 300, loss[loss=0.07209, simple_loss=0.09542, pruned_loss=0.01343, audio_tagging_loss=0.01095, over 15159.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09456, pruned_loss=0.01528, audio_tagging_loss=0.01145, over 2376340.76 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:12:40,519 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276850 2023-11-22 07:12:40,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1845633.3333333333, ans=0.125 2023-11-22 07:12:42,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1845633.3333333333, ans=0.125 2023-11-22 07:12:51,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1845700.0, ans=0.2 2023-11-22 07:13:00,922 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.553e+01 8.200e+01 8.925e+01 9.618e+01 1.197e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-22 07:13:02,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1845766.6666666667, ans=0.0 2023-11-22 07:13:08,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1845766.6666666667, ans=0.125 2023-11-22 07:13:09,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1845766.6666666667, ans=0.0 2023-11-22 07:13:20,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1845833.3333333333, ans=0.0 2023-11-22 07:13:32,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1845900.0, ans=15.0 2023-11-22 07:13:35,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.35 vs. limit=15.0 2023-11-22 07:13:39,904 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 350, loss[loss=0.06861, simple_loss=0.09036, pruned_loss=0.0116, audio_tagging_loss=0.01183, over 15347.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09621, pruned_loss=0.01545, audio_tagging_loss=0.01084, over 2524258.61 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:13:42,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1845966.6666666667, ans=0.125 2023-11-22 07:13:44,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276900 2023-11-22 07:13:46,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1845966.6666666667, ans=0.5 2023-11-22 07:13:51,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1845966.6666666667, ans=0.0 2023-11-22 07:14:03,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1846033.3333333333, ans=0.0 2023-11-22 07:14:44,727 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 400, loss[loss=0.07591, simple_loss=0.101, pruned_loss=0.01759, audio_tagging_loss=0.007846, over 14961.00 frames. ], tot_loss[loss=0.07343, simple_loss=0.09502, pruned_loss=0.01542, audio_tagging_loss=0.0105, over 2631396.07 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:14:48,658 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 276950 2023-11-22 07:15:00,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1846366.6666666667, ans=0.125 2023-11-22 07:15:09,169 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.931e+01 8.582e+01 9.377e+01 1.207e+02, threshold=1.716e+02, percent-clipped=0.0 2023-11-22 07:15:11,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1846433.3333333333, ans=0.1 2023-11-22 07:15:21,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1846433.3333333333, ans=0.0 2023-11-22 07:15:30,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1846500.0, ans=0.125 2023-11-22 07:15:49,045 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 450, loss[loss=0.08315, simple_loss=0.1136, pruned_loss=0.01976, audio_tagging_loss=0.006578, over 15336.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09538, pruned_loss=0.01547, audio_tagging_loss=0.01017, over 2722711.80 frames. ], batch size: 53, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:15:53,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277000 2023-11-22 07:16:11,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1846700.0, ans=0.025 2023-11-22 07:16:12,262 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.98 vs. limit=15.0 2023-11-22 07:16:40,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.42 vs. limit=22.5 2023-11-22 07:16:46,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1846900.0, ans=0.125 2023-11-22 07:16:53,438 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 500, loss[loss=0.08074, simple_loss=0.1032, pruned_loss=0.02153, audio_tagging_loss=0.00759, over 14832.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09461, pruned_loss=0.01555, audio_tagging_loss=0.01001, over 2788815.75 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:16:55,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1846966.6666666667, ans=0.125 2023-11-22 07:16:57,947 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277050 2023-11-22 07:17:17,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1847033.3333333333, ans=0.0 2023-11-22 07:17:18,052 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.239e+01 8.246e+01 8.888e+01 9.960e+01 1.317e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 07:17:27,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=1847100.0, ans=0.1 2023-11-22 07:17:31,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1847166.6666666667, ans=0.125 2023-11-22 07:17:43,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1847166.6666666667, ans=0.2 2023-11-22 07:17:58,808 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 550, loss[loss=0.06906, simple_loss=0.09047, pruned_loss=0.01386, audio_tagging_loss=0.009966, over 16423.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09463, pruned_loss=0.01558, audio_tagging_loss=0.009876, over 2854357.19 frames. ], batch size: 62, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:18:02,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277100 2023-11-22 07:18:30,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1847433.3333333333, ans=0.2 2023-11-22 07:18:33,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1847433.3333333333, ans=0.125 2023-11-22 07:18:38,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1847500.0, ans=0.125 2023-11-22 07:18:42,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1847500.0, ans=0.125 2023-11-22 07:18:48,972 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:18:50,688 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.66 vs. limit=22.5 2023-11-22 07:18:50,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1847566.6666666667, ans=22.5 2023-11-22 07:18:58,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1847566.6666666667, ans=0.2 2023-11-22 07:19:02,829 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 600, loss[loss=0.0759, simple_loss=0.1083, pruned_loss=0.0146, audio_tagging_loss=0.007149, over 15981.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09473, pruned_loss=0.01564, audio_tagging_loss=0.009804, over 2888457.68 frames. ], batch size: 59, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:19:07,278 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277150 2023-11-22 07:19:08,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1847633.3333333333, ans=0.2 2023-11-22 07:19:11,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1847633.3333333333, ans=0.125 2023-11-22 07:19:13,921 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.38 vs. limit=6.0 2023-11-22 07:19:20,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1847700.0, ans=0.07 2023-11-22 07:19:27,309 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.641e+01 8.220e+01 8.869e+01 9.857e+01 1.255e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 07:19:28,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1847766.6666666667, ans=0.0 2023-11-22 07:19:56,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1847900.0, ans=0.125 2023-11-22 07:20:07,405 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 650, loss[loss=0.05985, simple_loss=0.07819, pruned_loss=0.01299, audio_tagging_loss=0.007766, over 15536.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09495, pruned_loss=0.01566, audio_tagging_loss=0.009732, over 2914233.26 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:20:11,154 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277200 2023-11-22 07:20:22,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1848033.3333333333, ans=0.0 2023-11-22 07:20:30,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1848033.3333333333, ans=0.125 2023-11-22 07:20:31,279 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.73 vs. limit=15.0 2023-11-22 07:20:58,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1848233.3333333333, ans=0.0 2023-11-22 07:21:04,107 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.48 vs. limit=15.0 2023-11-22 07:21:06,638 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.08 vs. limit=10.0 2023-11-22 07:21:11,169 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 700, loss[loss=0.066, simple_loss=0.0904, pruned_loss=0.01453, audio_tagging_loss=0.006265, over 15101.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09595, pruned_loss=0.01564, audio_tagging_loss=0.009612, over 2940951.58 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:21:13,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1848300.0, ans=0.125 2023-11-22 07:21:15,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277250 2023-11-22 07:21:25,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1848366.6666666667, ans=0.125 2023-11-22 07:21:37,773 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.366e+01 8.189e+01 8.629e+01 9.381e+01 1.193e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-22 07:22:15,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1848633.3333333333, ans=0.1 2023-11-22 07:22:16,043 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 750, loss[loss=0.07111, simple_loss=0.1003, pruned_loss=0.01064, audio_tagging_loss=0.01029, over 15050.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09669, pruned_loss=0.01564, audio_tagging_loss=0.009472, over 2967266.27 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:22:20,489 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277300 2023-11-22 07:22:25,371 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.20 vs. limit=15.0 2023-11-22 07:22:28,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1848700.0, ans=0.2 2023-11-22 07:22:33,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1848700.0, ans=0.1 2023-11-22 07:22:41,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1848766.6666666667, ans=0.05 2023-11-22 07:22:48,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1848766.6666666667, ans=0.0 2023-11-22 07:22:52,821 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:22:56,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1848833.3333333333, ans=15.0 2023-11-22 07:23:01,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1848833.3333333333, ans=0.1 2023-11-22 07:23:09,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1848900.0, ans=0.125 2023-11-22 07:23:21,092 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 800, loss[loss=0.07291, simple_loss=0.09579, pruned_loss=0.01435, audio_tagging_loss=0.01067, over 15633.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09608, pruned_loss=0.01548, audio_tagging_loss=0.009543, over 2985304.00 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:23:24,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277350 2023-11-22 07:23:27,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1848966.6666666667, ans=0.09899494936611666 2023-11-22 07:23:28,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1848966.6666666667, ans=0.125 2023-11-22 07:23:38,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1849033.3333333333, ans=0.2 2023-11-22 07:23:45,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.74 vs. limit=15.0 2023-11-22 07:23:46,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.163e+01 8.225e+01 8.846e+01 9.560e+01 1.170e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 07:23:51,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1849100.0, ans=0.125 2023-11-22 07:23:53,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1849100.0, ans=0.125 2023-11-22 07:24:00,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1849166.6666666667, ans=0.0 2023-11-22 07:24:11,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1849233.3333333333, ans=0.125 2023-11-22 07:24:24,998 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 850, loss[loss=0.0974, simple_loss=0.1281, pruned_loss=0.02573, audio_tagging_loss=0.00761, over 15407.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09534, pruned_loss=0.01546, audio_tagging_loss=0.009641, over 3002253.59 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:24:28,695 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277400 2023-11-22 07:24:36,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1849366.6666666667, ans=0.125 2023-11-22 07:25:00,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1849433.3333333333, ans=0.025 2023-11-22 07:25:28,885 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 900, loss[loss=0.08268, simple_loss=0.1028, pruned_loss=0.0195, audio_tagging_loss=0.01178, over 15116.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09496, pruned_loss=0.01531, audio_tagging_loss=0.009771, over 3007934.44 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:25:33,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277450 2023-11-22 07:25:56,246 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.215e+01 8.871e+01 9.558e+01 1.650e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 07:26:33,302 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 950, loss[loss=0.08033, simple_loss=0.1106, pruned_loss=0.01562, audio_tagging_loss=0.00941, over 15713.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09551, pruned_loss=0.01533, audio_tagging_loss=0.009577, over 3018321.36 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:26:36,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1849966.6666666667, ans=0.0 2023-11-22 07:26:37,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277500 2023-11-22 07:26:42,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1849966.6666666667, ans=0.0 2023-11-22 07:26:47,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1850033.3333333333, ans=0.125 2023-11-22 07:26:54,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.37 vs. limit=10.0 2023-11-22 07:27:11,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1850166.6666666667, ans=0.125 2023-11-22 07:27:36,857 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1000, loss[loss=0.05944, simple_loss=0.07193, pruned_loss=0.01411, audio_tagging_loss=0.009367, over 15708.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.09562, pruned_loss=0.01539, audio_tagging_loss=0.009318, over 3021505.33 frames. ], batch size: 61, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:27:40,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277550 2023-11-22 07:27:44,850 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.53 vs. limit=15.0 2023-11-22 07:27:54,165 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:27:54,588 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.36 vs. limit=22.5 2023-11-22 07:27:57,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.74 vs. limit=22.5 2023-11-22 07:28:04,354 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.008e+01 8.714e+01 9.298e+01 1.223e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-22 07:28:04,477 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 07:28:08,765 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.68 vs. limit=15.0 2023-11-22 07:28:24,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1850500.0, ans=0.125 2023-11-22 07:28:24,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1850500.0, ans=0.04949747468305833 2023-11-22 07:28:26,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1850500.0, ans=0.125 2023-11-22 07:28:28,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1850566.6666666667, ans=0.125 2023-11-22 07:28:40,593 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1050, loss[loss=0.08172, simple_loss=0.1107, pruned_loss=0.01819, audio_tagging_loss=0.008177, over 15047.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09577, pruned_loss=0.01563, audio_tagging_loss=0.009286, over 3032712.87 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:28:40,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1850633.3333333333, ans=0.2 2023-11-22 07:28:45,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277600 2023-11-22 07:29:16,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1850766.6666666667, ans=0.125 2023-11-22 07:29:20,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1850833.3333333333, ans=0.125 2023-11-22 07:29:20,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.45 vs. limit=15.0 2023-11-22 07:29:46,041 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1100, loss[loss=0.08234, simple_loss=0.1151, pruned_loss=0.01652, audio_tagging_loss=0.008255, over 15366.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09466, pruned_loss=0.01537, audio_tagging_loss=0.009367, over 3031647.77 frames. ], batch size: 53, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:29:49,650 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 07:29:49,738 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277650 2023-11-22 07:29:50,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1850966.6666666667, ans=0.0 2023-11-22 07:30:11,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1851100.0, ans=0.05 2023-11-22 07:30:11,992 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.181e+01 8.784e+01 9.508e+01 1.137e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 07:30:18,849 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.58 vs. limit=15.0 2023-11-22 07:30:50,560 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1150, loss[loss=0.05579, simple_loss=0.0691, pruned_loss=0.008824, audio_tagging_loss=0.01241, over 14378.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09498, pruned_loss=0.0155, audio_tagging_loss=0.009239, over 3035237.73 frames. ], batch size: 53, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:30:54,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277700 2023-11-22 07:31:10,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1851366.6666666667, ans=0.95 2023-11-22 07:31:35,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.33 vs. limit=15.0 2023-11-22 07:31:54,999 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1200, loss[loss=0.08031, simple_loss=0.1145, pruned_loss=0.01601, audio_tagging_loss=0.007071, over 15119.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09573, pruned_loss=0.01566, audio_tagging_loss=0.009166, over 3034871.39 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:31:58,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.20 vs. limit=15.0 2023-11-22 07:31:58,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277750 2023-11-22 07:32:19,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1851700.0, ans=0.1 2023-11-22 07:32:23,093 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.243e+01 8.919e+01 9.746e+01 1.203e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 07:32:39,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.70 vs. limit=15.0 2023-11-22 07:32:52,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1851900.0, ans=0.125 2023-11-22 07:33:00,930 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1250, loss[loss=0.06617, simple_loss=0.07832, pruned_loss=0.01395, audio_tagging_loss=0.01306, over 16547.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.0948, pruned_loss=0.01544, audio_tagging_loss=0.009234, over 3044435.05 frames. ], batch size: 63, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:33:04,925 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277800 2023-11-22 07:33:10,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=12.0 2023-11-22 07:34:06,752 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1300, loss[loss=0.07307, simple_loss=0.09394, pruned_loss=0.01779, audio_tagging_loss=0.008313, over 14732.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.09525, pruned_loss=0.01557, audio_tagging_loss=0.00919, over 3035315.11 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:34:09,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1852300.0, ans=0.125 2023-11-22 07:34:10,650 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277850 2023-11-22 07:34:20,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1852366.6666666667, ans=0.0 2023-11-22 07:34:25,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1852366.6666666667, ans=0.125 2023-11-22 07:34:33,409 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.077e+01 8.549e+01 9.237e+01 1.175e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-22 07:34:41,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1852433.3333333333, ans=0.125 2023-11-22 07:34:52,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1852500.0, ans=0.1 2023-11-22 07:35:11,085 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1350, loss[loss=0.07951, simple_loss=0.1028, pruned_loss=0.01959, audio_tagging_loss=0.008507, over 15348.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.0956, pruned_loss=0.01558, audio_tagging_loss=0.009192, over 3042121.91 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:35:13,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1852633.3333333333, ans=0.125 2023-11-22 07:35:14,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277900 2023-11-22 07:35:33,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1852700.0, ans=0.2 2023-11-22 07:35:39,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1852766.6666666667, ans=0.04949747468305833 2023-11-22 07:35:41,700 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.96 vs. limit=15.0 2023-11-22 07:35:45,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1852766.6666666667, ans=0.1 2023-11-22 07:35:56,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1852833.3333333333, ans=0.0 2023-11-22 07:35:57,567 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 07:36:16,022 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1400, loss[loss=0.06933, simple_loss=0.08894, pruned_loss=0.01492, audio_tagging_loss=0.009934, over 16595.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09476, pruned_loss=0.01532, audio_tagging_loss=0.009315, over 3043646.21 frames. ], batch size: 62, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:36:19,898 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 277950 2023-11-22 07:36:23,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.15 vs. limit=15.0 2023-11-22 07:36:42,783 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.691e+01 8.253e+01 8.948e+01 9.652e+01 1.188e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-22 07:36:58,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1853166.6666666667, ans=0.125 2023-11-22 07:37:00,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1853166.6666666667, ans=0.0 2023-11-22 07:37:01,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1853166.6666666667, ans=0.1 2023-11-22 07:37:17,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1853233.3333333333, ans=0.0 2023-11-22 07:37:20,278 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1450, loss[loss=0.08306, simple_loss=0.1126, pruned_loss=0.01584, audio_tagging_loss=0.0109, over 16638.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.0952, pruned_loss=0.01533, audio_tagging_loss=0.009389, over 3044684.86 frames. ], batch size: 61, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:37:24,597 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278000 2023-11-22 07:37:37,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1853366.6666666667, ans=0.125 2023-11-22 07:37:42,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1853366.6666666667, ans=0.125 2023-11-22 07:37:49,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1853433.3333333333, ans=22.5 2023-11-22 07:37:52,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1853433.3333333333, ans=0.2 2023-11-22 07:37:59,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1853500.0, ans=0.0 2023-11-22 07:38:06,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1853500.0, ans=0.125 2023-11-22 07:38:24,852 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1500, loss[loss=0.08179, simple_loss=0.1088, pruned_loss=0.01897, audio_tagging_loss=0.008426, over 15835.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09571, pruned_loss=0.0157, audio_tagging_loss=0.009506, over 3044044.44 frames. ], batch size: 60, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:38:28,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278050 2023-11-22 07:38:31,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1853633.3333333333, ans=0.0 2023-11-22 07:38:39,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1853700.0, ans=0.125 2023-11-22 07:38:49,797 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.89 vs. limit=12.0 2023-11-22 07:38:49,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.96 vs. limit=22.5 2023-11-22 07:38:52,913 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.167e+01 8.667e+01 9.503e+01 1.230e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-22 07:38:55,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1853766.6666666667, ans=0.1 2023-11-22 07:39:01,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1853766.6666666667, ans=0.125 2023-11-22 07:39:09,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1853833.3333333333, ans=0.125 2023-11-22 07:39:29,697 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1550, loss[loss=0.06252, simple_loss=0.07677, pruned_loss=0.01323, audio_tagging_loss=0.01091, over 14770.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09589, pruned_loss=0.01583, audio_tagging_loss=0.009546, over 3041441.02 frames. ], batch size: 59, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:39:34,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278100 2023-11-22 07:39:34,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1853966.6666666667, ans=0.0 2023-11-22 07:40:20,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1854233.3333333333, ans=0.125 2023-11-22 07:40:34,407 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1600, loss[loss=0.05839, simple_loss=0.07434, pruned_loss=0.01124, audio_tagging_loss=0.009978, over 16278.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09539, pruned_loss=0.0157, audio_tagging_loss=0.009592, over 3035908.27 frames. ], batch size: 62, lr: 2.87e-03, grad_scale: 32.0 2023-11-22 07:40:37,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1854300.0, ans=0.07 2023-11-22 07:40:38,787 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278150 2023-11-22 07:40:55,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1854366.6666666667, ans=0.125 2023-11-22 07:41:04,081 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.136e+01 8.791e+01 9.603e+01 1.214e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 07:41:06,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1854433.3333333333, ans=0.125 2023-11-22 07:41:39,061 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1650, loss[loss=0.07947, simple_loss=0.1087, pruned_loss=0.01898, audio_tagging_loss=0.006115, over 15291.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09492, pruned_loss=0.01557, audio_tagging_loss=0.009605, over 3037327.91 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:41:42,744 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278200 2023-11-22 07:41:56,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1854700.0, ans=0.125 2023-11-22 07:41:59,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1854700.0, ans=0.125 2023-11-22 07:42:12,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1854766.6666666667, ans=0.0 2023-11-22 07:42:23,916 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=15.0 2023-11-22 07:42:31,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1854900.0, ans=0.125 2023-11-22 07:42:40,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1854900.0, ans=0.0 2023-11-22 07:42:43,483 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1700, loss[loss=0.04598, simple_loss=0.05503, pruned_loss=0.008133, audio_tagging_loss=0.01033, over 14540.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09483, pruned_loss=0.01555, audio_tagging_loss=0.00955, over 3044404.72 frames. ], batch size: 54, lr: 2.87e-03, grad_scale: 16.0 2023-11-22 07:42:47,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278250 2023-11-22 07:43:00,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1855033.3333333333, ans=0.125 2023-11-22 07:43:00,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1855033.3333333333, ans=0.0 2023-11-22 07:43:13,678 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.170e+01 8.792e+01 9.443e+01 1.155e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 07:43:20,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1855100.0, ans=0.1 2023-11-22 07:43:20,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1855100.0, ans=0.125 2023-11-22 07:43:41,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1855233.3333333333, ans=0.0 2023-11-22 07:43:45,059 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.14 vs. limit=10.0 2023-11-22 07:43:48,093 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1750, loss[loss=0.05887, simple_loss=0.08499, pruned_loss=0.007508, audio_tagging_loss=0.00887, over 14679.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09443, pruned_loss=0.01549, audio_tagging_loss=0.00958, over 3042868.30 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 07:43:50,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1855300.0, ans=0.1 2023-11-22 07:43:51,875 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278300 2023-11-22 07:44:06,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1855366.6666666667, ans=0.125 2023-11-22 07:44:11,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1855366.6666666667, ans=0.1 2023-11-22 07:44:44,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1855566.6666666667, ans=0.1 2023-11-22 07:44:46,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1855566.6666666667, ans=0.125 2023-11-22 07:44:50,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1855566.6666666667, ans=0.1 2023-11-22 07:44:52,594 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1800, loss[loss=0.08567, simple_loss=0.1054, pruned_loss=0.02441, audio_tagging_loss=0.008547, over 14914.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09424, pruned_loss=0.01532, audio_tagging_loss=0.009561, over 3051357.49 frames. ], batch size: 58, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 07:44:56,934 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278350 2023-11-22 07:45:22,550 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 7.991e+01 8.555e+01 9.443e+01 1.169e+02, threshold=1.711e+02, percent-clipped=0.0 2023-11-22 07:45:36,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.83 vs. limit=22.5 2023-11-22 07:45:57,030 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1850, loss[loss=0.08849, simple_loss=0.1179, pruned_loss=0.02151, audio_tagging_loss=0.008021, over 15910.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09413, pruned_loss=0.01534, audio_tagging_loss=0.009464, over 3046064.89 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 07:46:00,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1855966.6666666667, ans=0.1 2023-11-22 07:46:01,514 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278400 2023-11-22 07:46:25,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1856100.0, ans=0.2 2023-11-22 07:46:34,543 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.24 vs. limit=8.0 2023-11-22 07:46:44,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.16 vs. limit=22.5 2023-11-22 07:46:53,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1856233.3333333333, ans=0.0 2023-11-22 07:46:58,594 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.41 vs. limit=22.5 2023-11-22 07:47:02,770 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1900, loss[loss=0.05623, simple_loss=0.07137, pruned_loss=0.01206, audio_tagging_loss=0.008477, over 14845.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09325, pruned_loss=0.01517, audio_tagging_loss=0.009414, over 3049470.80 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 07:47:06,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278450 2023-11-22 07:47:17,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1856366.6666666667, ans=0.125 2023-11-22 07:47:29,954 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:47:32,029 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 7.925e+01 8.678e+01 9.471e+01 1.078e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-22 07:47:42,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1856500.0, ans=0.125 2023-11-22 07:47:42,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1856500.0, ans=0.1 2023-11-22 07:47:52,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1856500.0, ans=0.0 2023-11-22 07:48:01,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1856566.6666666667, ans=0.125 2023-11-22 07:48:07,434 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 1950, loss[loss=0.07102, simple_loss=0.08505, pruned_loss=0.01732, audio_tagging_loss=0.01118, over 15703.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09359, pruned_loss=0.01523, audio_tagging_loss=0.009364, over 3050592.60 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 07:48:10,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1856633.3333333333, ans=0.2 2023-11-22 07:48:11,217 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278500 2023-11-22 07:48:16,792 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.10 vs. limit=6.0 2023-11-22 07:48:29,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1856700.0, ans=0.125 2023-11-22 07:48:45,462 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.94 vs. limit=22.5 2023-11-22 07:49:00,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1856900.0, ans=0.125 2023-11-22 07:49:06,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1856900.0, ans=0.125 2023-11-22 07:49:10,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1856966.6666666667, ans=0.125 2023-11-22 07:49:12,018 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2000, loss[loss=0.05374, simple_loss=0.06794, pruned_loss=0.009539, audio_tagging_loss=0.01023, over 12556.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09353, pruned_loss=0.01523, audio_tagging_loss=0.009395, over 3050606.64 frames. ], batch size: 50, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:49:13,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1856966.6666666667, ans=0.125 2023-11-22 07:49:16,454 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278550 2023-11-22 07:49:24,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1857033.3333333333, ans=0.125 2023-11-22 07:49:36,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.08 vs. limit=15.0 2023-11-22 07:49:41,984 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.370e+01 9.075e+01 9.910e+01 1.919e+02, threshold=1.815e+02, percent-clipped=1.0 2023-11-22 07:49:54,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1857166.6666666667, ans=0.125 2023-11-22 07:50:01,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.80 vs. limit=22.5 2023-11-22 07:50:04,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1857233.3333333333, ans=0.09899494936611666 2023-11-22 07:50:08,794 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.81 vs. limit=10.0 2023-11-22 07:50:17,544 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2050, loss[loss=0.05596, simple_loss=0.06952, pruned_loss=0.01045, audio_tagging_loss=0.01075, over 15301.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09324, pruned_loss=0.01508, audio_tagging_loss=0.009375, over 3048173.62 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:50:21,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278600 2023-11-22 07:50:38,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.04 vs. limit=15.0 2023-11-22 07:50:39,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1857366.6666666667, ans=0.125 2023-11-22 07:50:41,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1857366.6666666667, ans=15.0 2023-11-22 07:50:58,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1857500.0, ans=0.0 2023-11-22 07:51:01,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1857500.0, ans=0.125 2023-11-22 07:51:08,335 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.13 vs. limit=22.5 2023-11-22 07:51:12,191 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.94 vs. limit=15.0 2023-11-22 07:51:22,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1857633.3333333333, ans=0.1 2023-11-22 07:51:23,217 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2100, loss[loss=0.07926, simple_loss=0.101, pruned_loss=0.02156, audio_tagging_loss=0.007211, over 15783.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09386, pruned_loss=0.0152, audio_tagging_loss=0.009289, over 3052035.63 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:51:26,205 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.88 vs. limit=15.0 2023-11-22 07:51:26,979 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278650 2023-11-22 07:51:39,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1857700.0, ans=0.125 2023-11-22 07:51:52,687 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.224e+01 8.868e+01 9.706e+01 1.360e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 07:51:53,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1857766.6666666667, ans=0.1 2023-11-22 07:52:18,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1857900.0, ans=0.2 2023-11-22 07:52:26,016 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2150, loss[loss=0.07321, simple_loss=0.09313, pruned_loss=0.01691, audio_tagging_loss=0.009733, over 15096.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09466, pruned_loss=0.01529, audio_tagging_loss=0.009276, over 3052152.05 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:52:29,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278700 2023-11-22 07:52:44,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1858033.3333333333, ans=0.1 2023-11-22 07:52:50,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1858033.3333333333, ans=0.125 2023-11-22 07:53:04,507 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:53:06,688 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 07:53:31,934 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2200, loss[loss=0.07581, simple_loss=0.1062, pruned_loss=0.01379, audio_tagging_loss=0.008937, over 15080.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09533, pruned_loss=0.01537, audio_tagging_loss=0.009243, over 3051490.40 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:53:35,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278750 2023-11-22 07:53:42,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.40 vs. limit=15.0 2023-11-22 07:53:43,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1858366.6666666667, ans=0.125 2023-11-22 07:53:57,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1858433.3333333333, ans=0.125 2023-11-22 07:54:00,458 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.169e+01 8.248e+01 8.861e+01 9.671e+01 1.420e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-22 07:54:02,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=22.5 2023-11-22 07:54:15,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1858500.0, ans=0.0 2023-11-22 07:54:15,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1858500.0, ans=0.125 2023-11-22 07:54:22,662 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=15.46 vs. limit=15.0 2023-11-22 07:54:36,134 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2250, loss[loss=0.04863, simple_loss=0.06633, pruned_loss=0.008087, audio_tagging_loss=0.00738, over 14460.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.095, pruned_loss=0.01538, audio_tagging_loss=0.009189, over 3043757.67 frames. ], batch size: 58, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:54:36,794 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.31 vs. limit=15.0 2023-11-22 07:54:39,808 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278800 2023-11-22 07:54:46,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1858633.3333333333, ans=0.09899494936611666 2023-11-22 07:55:02,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1858766.6666666667, ans=0.125 2023-11-22 07:55:05,715 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.19 vs. limit=22.5 2023-11-22 07:55:18,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1858833.3333333333, ans=0.1 2023-11-22 07:55:26,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1858900.0, ans=0.125 2023-11-22 07:55:37,903 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.59 vs. limit=15.0 2023-11-22 07:55:39,800 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2300, loss[loss=0.05496, simple_loss=0.07002, pruned_loss=0.01114, audio_tagging_loss=0.008808, over 13857.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09359, pruned_loss=0.01501, audio_tagging_loss=0.00927, over 3039923.65 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:55:43,499 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278850 2023-11-22 07:55:47,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1858966.6666666667, ans=0.125 2023-11-22 07:56:09,718 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.100e+01 8.737e+01 9.435e+01 1.101e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 07:56:12,178 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0 2023-11-22 07:56:25,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1859166.6666666667, ans=0.0 2023-11-22 07:56:37,230 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.43 vs. limit=15.0 2023-11-22 07:56:37,684 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 07:56:45,049 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2350, loss[loss=0.064, simple_loss=0.08071, pruned_loss=0.01466, audio_tagging_loss=0.008989, over 15140.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09273, pruned_loss=0.01495, audio_tagging_loss=0.009444, over 3038743.26 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 07:56:49,583 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278900 2023-11-22 07:57:06,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1859366.6666666667, ans=0.1 2023-11-22 07:57:12,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1859433.3333333333, ans=0.04949747468305833 2023-11-22 07:57:28,780 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 07:57:35,937 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.43 vs. limit=15.0 2023-11-22 07:57:49,791 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2400, loss[loss=0.07354, simple_loss=0.09759, pruned_loss=0.01469, audio_tagging_loss=0.01005, over 15937.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09383, pruned_loss=0.01518, audio_tagging_loss=0.009521, over 3031106.16 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:57:53,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 278950 2023-11-22 07:58:19,860 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.551e+01 8.282e+01 8.837e+01 9.594e+01 1.331e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 07:58:28,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1859833.3333333333, ans=0.2 2023-11-22 07:58:49,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1859900.0, ans=0.2 2023-11-22 07:58:53,796 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2450, loss[loss=0.04224, simple_loss=0.04584, pruned_loss=0.006509, audio_tagging_loss=0.01281, over 15107.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09432, pruned_loss=0.01528, audio_tagging_loss=0.009668, over 3037564.90 frames. ], batch size: 58, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 07:58:57,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279000 2023-11-22 07:59:03,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1859966.6666666667, ans=0.125 2023-11-22 07:59:11,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1860033.3333333333, ans=0.0 2023-11-22 07:59:20,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1860100.0, ans=0.04949747468305833 2023-11-22 07:59:24,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1860100.0, ans=0.1 2023-11-22 07:59:54,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1860233.3333333333, ans=0.2 2023-11-22 07:59:58,133 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2500, loss[loss=0.05809, simple_loss=0.07412, pruned_loss=0.01103, audio_tagging_loss=0.01, over 14753.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09416, pruned_loss=0.01535, audio_tagging_loss=0.00962, over 3043902.66 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:00:02,497 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279050 2023-11-22 08:00:29,442 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.264e+01 8.898e+01 9.687e+01 1.183e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-22 08:00:40,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1860500.0, ans=0.125 2023-11-22 08:00:42,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1860500.0, ans=0.05 2023-11-22 08:00:53,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1860566.6666666667, ans=0.125 2023-11-22 08:00:53,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.07 vs. limit=6.0 2023-11-22 08:01:03,385 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2550, loss[loss=0.06069, simple_loss=0.06973, pruned_loss=0.01506, audio_tagging_loss=0.01077, over 14324.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09379, pruned_loss=0.01533, audio_tagging_loss=0.009588, over 3040449.91 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:01:05,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1860633.3333333333, ans=0.125 2023-11-22 08:01:07,211 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279100 2023-11-22 08:01:13,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=1860633.3333333333, ans=0.1 2023-11-22 08:01:15,781 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=12.0 2023-11-22 08:01:18,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.31 vs. limit=22.5 2023-11-22 08:01:20,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1860700.0, ans=0.0 2023-11-22 08:02:00,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1860900.0, ans=0.125 2023-11-22 08:02:06,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1860900.0, ans=0.1 2023-11-22 08:02:06,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1860900.0, ans=0.0 2023-11-22 08:02:09,039 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2600, loss[loss=0.07305, simple_loss=0.09896, pruned_loss=0.01287, audio_tagging_loss=0.0107, over 14669.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09314, pruned_loss=0.01525, audio_tagging_loss=0.009439, over 3038103.10 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:02:12,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279150 2023-11-22 08:02:27,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1861033.3333333333, ans=0.07 2023-11-22 08:02:39,905 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.406e+01 8.247e+01 8.743e+01 9.488e+01 1.623e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 08:02:52,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1861166.6666666667, ans=0.125 2023-11-22 08:03:02,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1861233.3333333333, ans=0.0 2023-11-22 08:03:08,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1861233.3333333333, ans=0.0 2023-11-22 08:03:12,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.49 vs. limit=22.5 2023-11-22 08:03:12,768 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2650, loss[loss=0.04629, simple_loss=0.05655, pruned_loss=0.009978, audio_tagging_loss=0.008044, over 16288.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09353, pruned_loss=0.01521, audio_tagging_loss=0.009454, over 3044227.24 frames. ], batch size: 62, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:03:17,172 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279200 2023-11-22 08:03:32,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1861366.6666666667, ans=0.125 2023-11-22 08:03:39,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1861433.3333333333, ans=0.125 2023-11-22 08:03:57,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1861500.0, ans=0.1 2023-11-22 08:04:12,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1861566.6666666667, ans=0.0 2023-11-22 08:04:17,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1861633.3333333333, ans=0.125 2023-11-22 08:04:18,478 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2700, loss[loss=0.06974, simple_loss=0.09868, pruned_loss=0.01104, audio_tagging_loss=0.009364, over 13766.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09371, pruned_loss=0.0153, audio_tagging_loss=0.009311, over 3056208.50 frames. ], batch size: 52, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 08:04:20,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.57 vs. limit=15.0 2023-11-22 08:04:22,248 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279250 2023-11-22 08:04:50,492 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 7.998e+01 8.544e+01 9.286e+01 1.150e+02, threshold=1.709e+02, percent-clipped=0.0 2023-11-22 08:04:50,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1861766.6666666667, ans=0.0 2023-11-22 08:05:00,569 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:05:10,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1861900.0, ans=0.125 2023-11-22 08:05:23,839 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2750, loss[loss=0.06869, simple_loss=0.09406, pruned_loss=0.01311, audio_tagging_loss=0.008554, over 15204.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09345, pruned_loss=0.01516, audio_tagging_loss=0.009399, over 3051611.39 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0 2023-11-22 08:05:27,574 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279300 2023-11-22 08:05:47,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1862033.3333333333, ans=0.0 2023-11-22 08:06:13,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1862166.6666666667, ans=0.5 2023-11-22 08:06:20,789 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 08:06:22,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.64 vs. limit=10.0 2023-11-22 08:06:28,057 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2800, loss[loss=0.05262, simple_loss=0.06877, pruned_loss=0.009585, audio_tagging_loss=0.008652, over 14459.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09278, pruned_loss=0.01502, audio_tagging_loss=0.009456, over 3047141.23 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:06:32,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279350 2023-11-22 08:07:01,175 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 7.937e+01 8.597e+01 9.316e+01 1.176e+02, threshold=1.719e+02, percent-clipped=0.0 2023-11-22 08:07:01,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1862433.3333333333, ans=0.0 2023-11-22 08:07:03,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.92 vs. limit=15.0 2023-11-22 08:07:21,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1862566.6666666667, ans=0.0 2023-11-22 08:07:33,278 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2850, loss[loss=0.079, simple_loss=0.1106, pruned_loss=0.01415, audio_tagging_loss=0.009576, over 16311.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09364, pruned_loss=0.01501, audio_tagging_loss=0.009384, over 3050089.15 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:07:34,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1862633.3333333333, ans=0.2 2023-11-22 08:07:37,078 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279400 2023-11-22 08:07:42,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1862633.3333333333, ans=0.5 2023-11-22 08:08:02,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1862766.6666666667, ans=0.125 2023-11-22 08:08:34,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1862900.0, ans=0.1 2023-11-22 08:08:36,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1862900.0, ans=0.0 2023-11-22 08:08:37,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1862966.6666666667, ans=0.2 2023-11-22 08:08:38,453 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2900, loss[loss=0.072, simple_loss=0.09674, pruned_loss=0.01659, audio_tagging_loss=0.007037, over 16070.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09418, pruned_loss=0.01512, audio_tagging_loss=0.009374, over 3046736.15 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:08:42,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279450 2023-11-22 08:08:55,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1863033.3333333333, ans=0.125 2023-11-22 08:09:01,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1863033.3333333333, ans=0.1 2023-11-22 08:09:11,274 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.213e+01 8.733e+01 9.343e+01 1.106e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 08:09:43,399 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 2950, loss[loss=0.08286, simple_loss=0.1031, pruned_loss=0.02042, audio_tagging_loss=0.01089, over 16141.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09448, pruned_loss=0.01525, audio_tagging_loss=0.009414, over 3052893.50 frames. ], batch size: 61, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:09:47,295 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279500 2023-11-22 08:09:50,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1863300.0, ans=0.125 2023-11-22 08:09:50,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1863300.0, ans=0.0 2023-11-22 08:09:53,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.16 vs. limit=15.0 2023-11-22 08:10:03,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1863366.6666666667, ans=0.125 2023-11-22 08:10:14,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1863433.3333333333, ans=0.1 2023-11-22 08:10:31,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1863500.0, ans=0.125 2023-11-22 08:10:32,708 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.56 vs. limit=15.0 2023-11-22 08:10:43,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.76 vs. limit=6.0 2023-11-22 08:10:49,470 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3000, loss[loss=0.06366, simple_loss=0.08271, pruned_loss=0.01195, audio_tagging_loss=0.01035, over 15909.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09406, pruned_loss=0.01517, audio_tagging_loss=0.009549, over 3055085.53 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:10:49,471 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 08:11:29,444 INFO [train_asr.py:1253] (3/4) Epoch 24, validation: loss=0.0588, simple_loss=0.05168, pruned_loss=0.005124, audio_tagging_loss=0.02784, over 4681554.00 frames. 2023-11-22 08:11:29,444 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 08:11:33,212 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279550 2023-11-22 08:12:02,512 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.135e+01 8.700e+01 9.423e+01 1.185e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 08:12:34,539 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3050, loss[loss=0.08427, simple_loss=0.1135, pruned_loss=0.01968, audio_tagging_loss=0.007865, over 16211.00 frames. ], tot_loss[loss=0.07153, simple_loss=0.09388, pruned_loss=0.01505, audio_tagging_loss=0.009536, over 3048235.41 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:12:38,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279600 2023-11-22 08:13:01,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1864100.0, ans=0.125 2023-11-22 08:13:14,139 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 08:13:15,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1864166.6666666667, ans=0.0 2023-11-22 08:13:17,297 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.36 vs. limit=22.5 2023-11-22 08:13:21,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1864166.6666666667, ans=0.125 2023-11-22 08:13:27,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1864233.3333333333, ans=0.1 2023-11-22 08:13:29,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1864233.3333333333, ans=0.0 2023-11-22 08:13:30,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1864233.3333333333, ans=0.125 2023-11-22 08:13:34,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1864233.3333333333, ans=0.0 2023-11-22 08:13:41,248 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3100, loss[loss=0.07576, simple_loss=0.09937, pruned_loss=0.01646, audio_tagging_loss=0.009612, over 15230.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09379, pruned_loss=0.01488, audio_tagging_loss=0.009609, over 3044013.48 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:13:45,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279650 2023-11-22 08:13:47,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1864300.0, ans=0.125 2023-11-22 08:14:00,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1864366.6666666667, ans=0.1 2023-11-22 08:14:13,539 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.146e+01 8.664e+01 9.339e+01 1.098e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-22 08:14:25,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1864500.0, ans=0.0 2023-11-22 08:14:27,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1864500.0, ans=0.125 2023-11-22 08:14:46,716 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3150, loss[loss=0.0824, simple_loss=0.1136, pruned_loss=0.0166, audio_tagging_loss=0.00902, over 14727.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.0946, pruned_loss=0.01522, audio_tagging_loss=0.00958, over 3041376.73 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:14:50,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279700 2023-11-22 08:14:53,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1864633.3333333333, ans=0.125 2023-11-22 08:15:10,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1864700.0, ans=0.0 2023-11-22 08:15:11,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1864766.6666666667, ans=0.125 2023-11-22 08:15:20,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1864766.6666666667, ans=0.0 2023-11-22 08:15:50,947 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3200, loss[loss=0.08271, simple_loss=0.09755, pruned_loss=0.02018, audio_tagging_loss=0.01376, over 16020.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09527, pruned_loss=0.01545, audio_tagging_loss=0.00963, over 3047733.54 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:15:55,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279750 2023-11-22 08:16:14,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1865033.3333333333, ans=0.125 2023-11-22 08:16:24,179 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.350e+01 8.310e+01 8.993e+01 9.711e+01 1.257e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 08:16:31,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1865166.6666666667, ans=0.125 2023-11-22 08:16:40,751 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=21.93 vs. limit=22.5 2023-11-22 08:16:56,987 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3250, loss[loss=0.05177, simple_loss=0.05624, pruned_loss=0.009663, audio_tagging_loss=0.01398, over 15370.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09469, pruned_loss=0.01532, audio_tagging_loss=0.009676, over 3046728.58 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:17:00,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279800 2023-11-22 08:17:06,451 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:17:14,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1865366.6666666667, ans=0.0 2023-11-22 08:17:38,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.28 vs. limit=22.5 2023-11-22 08:17:44,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1865500.0, ans=0.125 2023-11-22 08:18:02,038 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3300, loss[loss=0.05417, simple_loss=0.06412, pruned_loss=0.01097, audio_tagging_loss=0.01114, over 15743.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09461, pruned_loss=0.0155, audio_tagging_loss=0.009683, over 3047446.89 frames. ], batch size: 64, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:18:06,379 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279850 2023-11-22 08:18:06,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1865633.3333333333, ans=0.1 2023-11-22 08:18:08,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1865633.3333333333, ans=0.2 2023-11-22 08:18:14,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.83 vs. limit=22.5 2023-11-22 08:18:22,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1865700.0, ans=0.07 2023-11-22 08:18:35,000 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.287e+01 8.648e+01 9.394e+01 1.159e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-22 08:18:45,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1865833.3333333333, ans=0.125 2023-11-22 08:18:45,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.whiten.whitening_limit, batch_count=1865833.3333333333, ans=12.0 2023-11-22 08:18:47,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1865833.3333333333, ans=0.125 2023-11-22 08:18:47,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1865833.3333333333, ans=0.0 2023-11-22 08:18:50,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1865833.3333333333, ans=0.125 2023-11-22 08:18:55,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1865900.0, ans=0.125 2023-11-22 08:19:06,569 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3350, loss[loss=0.06537, simple_loss=0.07379, pruned_loss=0.01545, audio_tagging_loss=0.01303, over 15329.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09448, pruned_loss=0.01545, audio_tagging_loss=0.009604, over 3043156.01 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:19:10,540 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279900 2023-11-22 08:19:17,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1865966.6666666667, ans=0.125 2023-11-22 08:19:25,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.04 vs. limit=15.0 2023-11-22 08:19:30,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0 2023-11-22 08:19:41,518 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:19:58,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.62 vs. limit=15.0 2023-11-22 08:20:11,655 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3400, loss[loss=0.08191, simple_loss=0.1199, pruned_loss=0.01738, audio_tagging_loss=0.004561, over 15082.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09534, pruned_loss=0.0156, audio_tagging_loss=0.009376, over 3046011.80 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:20:13,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1866300.0, ans=0.125 2023-11-22 08:20:15,448 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 279950 2023-11-22 08:20:18,586 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.57 vs. limit=15.0 2023-11-22 08:20:27,070 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.66 vs. limit=15.0 2023-11-22 08:20:33,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1866366.6666666667, ans=0.2 2023-11-22 08:20:42,977 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.164e+01 8.350e+01 8.855e+01 9.450e+01 1.235e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-22 08:21:07,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1866566.6666666667, ans=0.125 2023-11-22 08:21:10,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1866566.6666666667, ans=0.0 2023-11-22 08:21:15,132 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3450, loss[loss=0.0722, simple_loss=0.09273, pruned_loss=0.01527, audio_tagging_loss=0.01057, over 14561.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09619, pruned_loss=0.01571, audio_tagging_loss=0.009258, over 3048350.44 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:21:19,546 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280000 2023-11-22 08:21:38,443 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:21:44,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1866766.6666666667, ans=0.125 2023-11-22 08:22:04,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1866833.3333333333, ans=0.0 2023-11-22 08:22:05,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1866833.3333333333, ans=0.0 2023-11-22 08:22:06,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1866833.3333333333, ans=0.0 2023-11-22 08:22:09,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.45 vs. limit=15.0 2023-11-22 08:22:14,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1866900.0, ans=0.0 2023-11-22 08:22:18,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1866900.0, ans=0.1 2023-11-22 08:22:21,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1866900.0, ans=0.125 2023-11-22 08:22:23,242 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3500, loss[loss=0.06155, simple_loss=0.07552, pruned_loss=0.01299, audio_tagging_loss=0.0108, over 14701.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.0965, pruned_loss=0.0159, audio_tagging_loss=0.009176, over 3048156.12 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:22:26,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280050 2023-11-22 08:22:48,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1867100.0, ans=0.125 2023-11-22 08:22:51,601 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:22:56,272 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.244e+01 8.926e+01 9.794e+01 1.238e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-22 08:22:57,663 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 08:23:21,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1867233.3333333333, ans=0.0 2023-11-22 08:23:29,256 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3550, loss[loss=0.05611, simple_loss=0.06364, pruned_loss=0.0131, audio_tagging_loss=0.01119, over 14809.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.0959, pruned_loss=0.01579, audio_tagging_loss=0.009184, over 3049962.82 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:23:33,681 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280100 2023-11-22 08:24:00,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1867433.3333333333, ans=0.0 2023-11-22 08:24:11,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1867500.0, ans=0.05 2023-11-22 08:24:27,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1867566.6666666667, ans=10.0 2023-11-22 08:24:34,294 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3600, loss[loss=0.1032, simple_loss=0.1392, pruned_loss=0.0273, audio_tagging_loss=0.00629, over 15777.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09477, pruned_loss=0.01572, audio_tagging_loss=0.009244, over 3046523.36 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:24:38,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280150 2023-11-22 08:24:50,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1867700.0, ans=0.125 2023-11-22 08:24:51,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1867700.0, ans=0.125 2023-11-22 08:24:51,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1867700.0, ans=0.125 2023-11-22 08:25:07,384 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 7.927e+01 8.588e+01 9.278e+01 1.263e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-22 08:25:38,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.73 vs. limit=15.0 2023-11-22 08:25:39,374 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3650, loss[loss=0.06102, simple_loss=0.08198, pruned_loss=0.009789, audio_tagging_loss=0.01023, over 15714.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09476, pruned_loss=0.01557, audio_tagging_loss=0.0093, over 3050632.69 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0 2023-11-22 08:25:39,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1867966.6666666667, ans=0.0 2023-11-22 08:25:43,190 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280200 2023-11-22 08:25:55,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1868033.3333333333, ans=0.125 2023-11-22 08:25:56,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.97 vs. limit=15.0 2023-11-22 08:26:03,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1868033.3333333333, ans=0.125 2023-11-22 08:26:15,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1868100.0, ans=0.125 2023-11-22 08:26:28,644 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.77 vs. limit=15.0 2023-11-22 08:26:45,329 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3700, loss[loss=0.06663, simple_loss=0.08678, pruned_loss=0.01195, audio_tagging_loss=0.01128, over 14523.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09408, pruned_loss=0.01535, audio_tagging_loss=0.009316, over 3044650.30 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:26:49,259 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280250 2023-11-22 08:27:03,765 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=15.0 2023-11-22 08:27:18,797 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.045e+01 8.787e+01 9.464e+01 1.178e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 08:27:26,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1868500.0, ans=0.07 2023-11-22 08:27:35,990 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.68 vs. limit=15.0 2023-11-22 08:27:50,869 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3750, loss[loss=0.09395, simple_loss=0.1087, pruned_loss=0.02892, audio_tagging_loss=0.01068, over 15175.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.0939, pruned_loss=0.0154, audio_tagging_loss=0.009335, over 3046753.96 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:27:54,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280300 2023-11-22 08:28:13,180 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.10 vs. limit=15.0 2023-11-22 08:28:13,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1868700.0, ans=0.125 2023-11-22 08:28:35,746 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 08:28:44,436 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.29 vs. limit=6.0 2023-11-22 08:28:55,614 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3800, loss[loss=0.06195, simple_loss=0.08641, pruned_loss=0.01208, audio_tagging_loss=0.006668, over 14542.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09503, pruned_loss=0.01552, audio_tagging_loss=0.009365, over 3050944.15 frames. ], batch size: 53, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:28:59,315 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280350 2023-11-22 08:29:19,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1869033.3333333333, ans=0.125 2023-11-22 08:29:22,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1869100.0, ans=0.125 2023-11-22 08:29:28,707 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.943e+01 8.492e+01 9.128e+01 1.004e+02 1.242e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-22 08:29:38,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1869166.6666666667, ans=0.125 2023-11-22 08:29:42,412 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:29:43,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1869166.6666666667, ans=0.125 2023-11-22 08:29:59,889 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3850, loss[loss=0.05901, simple_loss=0.07761, pruned_loss=0.00897, audio_tagging_loss=0.01124, over 15357.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09567, pruned_loss=0.01561, audio_tagging_loss=0.009392, over 3053659.78 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:30:04,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280400 2023-11-22 08:30:20,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1869366.6666666667, ans=0.1 2023-11-22 08:31:04,435 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3900, loss[loss=0.06879, simple_loss=0.092, pruned_loss=0.014, audio_tagging_loss=0.008792, over 14584.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.09552, pruned_loss=0.01532, audio_tagging_loss=0.009425, over 3052601.69 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:31:08,391 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280450 2023-11-22 08:31:27,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1869700.0, ans=0.125 2023-11-22 08:31:27,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1869700.0, ans=0.125 2023-11-22 08:31:30,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1869766.6666666667, ans=0.07 2023-11-22 08:31:34,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1869766.6666666667, ans=0.1 2023-11-22 08:31:38,010 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.174e+01 8.873e+01 9.506e+01 1.121e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 08:31:39,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1869766.6666666667, ans=0.0 2023-11-22 08:31:49,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1869833.3333333333, ans=0.04949747468305833 2023-11-22 08:31:52,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.47 vs. limit=6.0 2023-11-22 08:31:55,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1869900.0, ans=0.125 2023-11-22 08:32:00,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1869900.0, ans=0.125 2023-11-22 08:32:10,109 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 3950, loss[loss=0.08303, simple_loss=0.1117, pruned_loss=0.01798, audio_tagging_loss=0.009171, over 16006.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09551, pruned_loss=0.01533, audio_tagging_loss=0.009476, over 3049490.50 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:32:13,971 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280500 2023-11-22 08:32:26,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1870033.3333333333, ans=0.125 2023-11-22 08:32:28,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=12.0 2023-11-22 08:32:37,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1870100.0, ans=0.07 2023-11-22 08:32:47,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1870166.6666666667, ans=0.0 2023-11-22 08:32:49,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.91 vs. limit=22.5 2023-11-22 08:33:13,412 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4000, loss[loss=0.08034, simple_loss=0.1042, pruned_loss=0.01902, audio_tagging_loss=0.009228, over 15507.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09521, pruned_loss=0.01538, audio_tagging_loss=0.009541, over 3039459.97 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:33:17,820 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280550 2023-11-22 08:33:29,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1870366.6666666667, ans=0.1 2023-11-22 08:33:44,746 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.52 vs. limit=12.0 2023-11-22 08:33:48,394 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.016e+01 8.419e+01 9.002e+01 9.753e+01 1.228e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-22 08:33:52,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1870500.0, ans=0.125 2023-11-22 08:33:55,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.26 vs. limit=12.0 2023-11-22 08:34:07,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1870566.6666666667, ans=0.125 2023-11-22 08:34:18,118 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4050, loss[loss=0.05519, simple_loss=0.06264, pruned_loss=0.01366, audio_tagging_loss=0.01021, over 14942.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09487, pruned_loss=0.01546, audio_tagging_loss=0.009534, over 3041293.19 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:34:21,961 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 08:34:21,990 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280600 2023-11-22 08:34:31,563 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.77 vs. limit=15.0 2023-11-22 08:34:37,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1870700.0, ans=0.125 2023-11-22 08:35:22,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=15.37 vs. limit=15.0 2023-11-22 08:35:23,022 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4100, loss[loss=0.08473, simple_loss=0.1076, pruned_loss=0.02304, audio_tagging_loss=0.007902, over 14034.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09429, pruned_loss=0.01516, audio_tagging_loss=0.009506, over 3046233.42 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:35:27,401 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280650 2023-11-22 08:35:36,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1871033.3333333333, ans=0.125 2023-11-22 08:35:36,986 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.13 vs. limit=15.0 2023-11-22 08:35:48,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1871100.0, ans=0.125 2023-11-22 08:35:55,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1871100.0, ans=0.125 2023-11-22 08:35:58,175 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.230e+01 8.854e+01 9.579e+01 2.804e+02, threshold=1.771e+02, percent-clipped=1.0 2023-11-22 08:36:09,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1871166.6666666667, ans=0.5 2023-11-22 08:36:27,878 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4150, loss[loss=0.06922, simple_loss=0.09062, pruned_loss=0.01493, audio_tagging_loss=0.008974, over 14993.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09506, pruned_loss=0.01532, audio_tagging_loss=0.009311, over 3038891.63 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:36:31,671 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280700 2023-11-22 08:36:42,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1871366.6666666667, ans=0.125 2023-11-22 08:37:02,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1871433.3333333333, ans=0.125 2023-11-22 08:37:06,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1871500.0, ans=0.0 2023-11-22 08:37:06,534 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:37:15,481 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 08:37:29,168 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:37:32,649 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4200, loss[loss=0.06881, simple_loss=0.09321, pruned_loss=0.01564, audio_tagging_loss=0.006567, over 15107.00 frames. ], tot_loss[loss=0.07219, simple_loss=0.09533, pruned_loss=0.01541, audio_tagging_loss=0.009115, over 3042087.37 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:37:35,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1871633.3333333333, ans=0.0 2023-11-22 08:37:36,359 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280750 2023-11-22 08:37:44,379 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.03 vs. limit=22.5 2023-11-22 08:37:50,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1871700.0, ans=0.125 2023-11-22 08:38:07,270 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.240e+01 8.912e+01 9.752e+01 1.165e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-22 08:38:17,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1871833.3333333333, ans=0.05 2023-11-22 08:38:29,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.74 vs. limit=10.0 2023-11-22 08:38:33,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.30 vs. limit=15.0 2023-11-22 08:38:34,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.61 vs. limit=10.0 2023-11-22 08:38:35,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1871900.0, ans=0.0 2023-11-22 08:38:37,256 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4250, loss[loss=0.07359, simple_loss=0.09717, pruned_loss=0.01698, audio_tagging_loss=0.008022, over 15206.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09601, pruned_loss=0.01547, audio_tagging_loss=0.009059, over 3050880.34 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:38:41,787 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280800 2023-11-22 08:38:43,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1871966.6666666667, ans=0.125 2023-11-22 08:39:00,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1872033.3333333333, ans=0.125 2023-11-22 08:39:21,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1872166.6666666667, ans=0.0 2023-11-22 08:39:38,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1872233.3333333333, ans=0.0 2023-11-22 08:39:40,253 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.30 vs. limit=15.0 2023-11-22 08:39:43,320 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4300, loss[loss=0.08843, simple_loss=0.1121, pruned_loss=0.02187, audio_tagging_loss=0.01048, over 15351.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09675, pruned_loss=0.01543, audio_tagging_loss=0.008997, over 3057298.06 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:39:47,914 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280850 2023-11-22 08:39:48,319 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.59 vs. limit=15.0 2023-11-22 08:40:04,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.24 vs. limit=15.0 2023-11-22 08:40:18,677 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.496e+01 9.037e+01 9.736e+01 1.212e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-22 08:40:25,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1872500.0, ans=0.125 2023-11-22 08:40:32,044 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=2.510e-03 2023-11-22 08:40:49,248 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4350, loss[loss=0.07921, simple_loss=0.09976, pruned_loss=0.01852, audio_tagging_loss=0.01082, over 15365.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09737, pruned_loss=0.01553, audio_tagging_loss=0.008863, over 3058453.15 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:40:53,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280900 2023-11-22 08:40:59,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1872633.3333333333, ans=0.0 2023-11-22 08:41:24,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1872766.6666666667, ans=0.5 2023-11-22 08:41:41,250 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:41:53,078 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4400, loss[loss=0.06135, simple_loss=0.08262, pruned_loss=0.0124, audio_tagging_loss=0.00764, over 15125.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09578, pruned_loss=0.01543, audio_tagging_loss=0.009039, over 3055715.20 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:41:56,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 280950 2023-11-22 08:42:00,651 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.79 vs. limit=15.0 2023-11-22 08:42:13,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1873033.3333333333, ans=0.125 2023-11-22 08:42:19,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1873100.0, ans=0.125 2023-11-22 08:42:23,208 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.40 vs. limit=12.0 2023-11-22 08:42:29,234 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.700e+01 8.342e+01 8.985e+01 9.628e+01 1.162e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 08:42:34,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1873166.6666666667, ans=0.0 2023-11-22 08:42:49,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1873233.3333333333, ans=10.0 2023-11-22 08:42:58,237 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4450, loss[loss=0.1001, simple_loss=0.1277, pruned_loss=0.02901, audio_tagging_loss=0.007268, over 15777.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09551, pruned_loss=0.0156, audio_tagging_loss=0.009099, over 3053157.07 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:43:01,956 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281000 2023-11-22 08:43:22,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1873366.6666666667, ans=0.125 2023-11-22 08:43:47,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1873500.0, ans=0.125 2023-11-22 08:44:03,750 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4500, loss[loss=0.08136, simple_loss=0.11, pruned_loss=0.01863, audio_tagging_loss=0.007738, over 16237.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09533, pruned_loss=0.01547, audio_tagging_loss=0.009165, over 3046818.26 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:44:07,683 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281050 2023-11-22 08:44:20,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1873700.0, ans=0.1 2023-11-22 08:44:36,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1873766.6666666667, ans=0.125 2023-11-22 08:44:37,532 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.057e+01 8.748e+01 9.801e+01 1.166e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 08:44:37,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1873766.6666666667, ans=0.0 2023-11-22 08:44:41,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1873833.3333333333, ans=0.09899494936611666 2023-11-22 08:44:55,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1873900.0, ans=0.5 2023-11-22 08:44:56,280 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:45:06,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1873966.6666666667, ans=0.035 2023-11-22 08:45:07,103 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4550, loss[loss=0.06748, simple_loss=0.08619, pruned_loss=0.01534, audio_tagging_loss=0.009043, over 14190.00 frames. ], tot_loss[loss=0.07262, simple_loss=0.09529, pruned_loss=0.01568, audio_tagging_loss=0.009287, over 3047731.30 frames. ], batch size: 53, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:45:10,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281100 2023-11-22 08:45:21,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1874033.3333333333, ans=0.125 2023-11-22 08:45:28,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1874033.3333333333, ans=0.125 2023-11-22 08:45:38,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1874100.0, ans=0.0 2023-11-22 08:45:38,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1874100.0, ans=0.125 2023-11-22 08:45:43,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1874100.0, ans=0.1 2023-11-22 08:45:49,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1874166.6666666667, ans=0.2 2023-11-22 08:45:54,426 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0 2023-11-22 08:45:56,306 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 08:46:11,521 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4600, loss[loss=0.076, simple_loss=0.09919, pruned_loss=0.01503, audio_tagging_loss=0.01137, over 14086.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09492, pruned_loss=0.01558, audio_tagging_loss=0.009286, over 3048311.49 frames. ], batch size: 53, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:46:12,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.69 vs. limit=15.0 2023-11-22 08:46:15,295 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281150 2023-11-22 08:46:15,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1874300.0, ans=0.2 2023-11-22 08:46:46,496 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 7.994e+01 8.789e+01 9.560e+01 1.302e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 08:46:54,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1874500.0, ans=0.125 2023-11-22 08:46:55,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1874500.0, ans=0.1 2023-11-22 08:47:16,078 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4650, loss[loss=0.07603, simple_loss=0.09941, pruned_loss=0.01548, audio_tagging_loss=0.01085, over 14907.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09479, pruned_loss=0.0156, audio_tagging_loss=0.009408, over 3048599.26 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:47:18,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.86 vs. limit=22.5 2023-11-22 08:47:19,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1874633.3333333333, ans=0.1 2023-11-22 08:47:20,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281200 2023-11-22 08:47:30,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1874700.0, ans=0.0 2023-11-22 08:47:41,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1874766.6666666667, ans=0.0 2023-11-22 08:48:13,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1874900.0, ans=0.125 2023-11-22 08:48:21,836 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4700, loss[loss=0.08154, simple_loss=0.109, pruned_loss=0.0184, audio_tagging_loss=0.008646, over 15478.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09503, pruned_loss=0.01569, audio_tagging_loss=0.009579, over 3045804.85 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:48:25,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281250 2023-11-22 08:48:40,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1875033.3333333333, ans=0.125 2023-11-22 08:48:53,997 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 08:48:54,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.78 vs. limit=15.0 2023-11-22 08:48:57,763 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 8.141e+01 8.777e+01 9.599e+01 1.409e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-22 08:49:05,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1875166.6666666667, ans=0.0 2023-11-22 08:49:18,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1875233.3333333333, ans=0.125 2023-11-22 08:49:20,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1875233.3333333333, ans=0.1 2023-11-22 08:49:25,815 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4750, loss[loss=0.07628, simple_loss=0.1004, pruned_loss=0.01654, audio_tagging_loss=0.009531, over 15082.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09442, pruned_loss=0.0154, audio_tagging_loss=0.009645, over 3042665.69 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:49:29,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281300 2023-11-22 08:49:51,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1875433.3333333333, ans=0.125 2023-11-22 08:50:04,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1875500.0, ans=0.035 2023-11-22 08:50:13,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1875500.0, ans=0.125 2023-11-22 08:50:23,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1875566.6666666667, ans=0.125 2023-11-22 08:50:29,577 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4800, loss[loss=0.06458, simple_loss=0.08659, pruned_loss=0.01277, audio_tagging_loss=0.008512, over 14554.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09508, pruned_loss=0.01545, audio_tagging_loss=0.009642, over 3051900.15 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:50:34,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281350 2023-11-22 08:50:39,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=1875633.3333333333, ans=15.0 2023-11-22 08:50:48,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1875700.0, ans=0.0 2023-11-22 08:50:57,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1875766.6666666667, ans=0.125 2023-11-22 08:51:07,400 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.460e+01 8.056e+01 8.734e+01 9.359e+01 1.284e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 08:51:32,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1875900.0, ans=0.125 2023-11-22 08:51:34,881 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4850, loss[loss=0.08399, simple_loss=0.1132, pruned_loss=0.02269, audio_tagging_loss=0.00467, over 15083.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09507, pruned_loss=0.0155, audio_tagging_loss=0.009731, over 3050467.42 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:51:38,654 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281400 2023-11-22 08:51:42,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1875966.6666666667, ans=0.2 2023-11-22 08:51:52,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1876033.3333333333, ans=0.0 2023-11-22 08:51:53,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1876033.3333333333, ans=0.1 2023-11-22 08:52:09,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1876100.0, ans=0.125 2023-11-22 08:52:12,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1876166.6666666667, ans=0.125 2023-11-22 08:52:31,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1876233.3333333333, ans=0.0 2023-11-22 08:52:36,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1876233.3333333333, ans=0.0 2023-11-22 08:52:39,572 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4900, loss[loss=0.06468, simple_loss=0.08496, pruned_loss=0.014, audio_tagging_loss=0.008202, over 15255.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09367, pruned_loss=0.01521, audio_tagging_loss=0.009699, over 3043987.72 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:52:43,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281450 2023-11-22 08:52:46,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.10 vs. limit=15.0 2023-11-22 08:53:16,823 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.163e+01 8.764e+01 9.409e+01 1.191e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-22 08:53:19,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1876500.0, ans=0.2 2023-11-22 08:53:40,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1876566.6666666667, ans=0.2 2023-11-22 08:53:43,091 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 4950, loss[loss=0.04639, simple_loss=0.0532, pruned_loss=0.009237, audio_tagging_loss=0.01056, over 13253.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09396, pruned_loss=0.01537, audio_tagging_loss=0.00945, over 3040787.29 frames. ], batch size: 53, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:53:47,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281500 2023-11-22 08:54:08,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1876766.6666666667, ans=0.125 2023-11-22 08:54:48,540 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5000, loss[loss=0.06699, simple_loss=0.08845, pruned_loss=0.01231, audio_tagging_loss=0.01046, over 14634.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09389, pruned_loss=0.01523, audio_tagging_loss=0.009372, over 3042406.94 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:54:51,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1876966.6666666667, ans=0.0 2023-11-22 08:54:52,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281550 2023-11-22 08:54:53,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1876966.6666666667, ans=0.0 2023-11-22 08:54:56,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1876966.6666666667, ans=0.125 2023-11-22 08:55:12,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1877033.3333333333, ans=0.125 2023-11-22 08:55:15,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1877100.0, ans=0.0 2023-11-22 08:55:15,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1877100.0, ans=0.04949747468305833 2023-11-22 08:55:25,179 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.261e+01 8.670e+01 9.360e+01 1.108e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-22 08:55:27,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1877166.6666666667, ans=0.125 2023-11-22 08:55:41,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1877233.3333333333, ans=10.0 2023-11-22 08:55:52,730 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5050, loss[loss=0.07937, simple_loss=0.1028, pruned_loss=0.0189, audio_tagging_loss=0.009085, over 14652.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09483, pruned_loss=0.01545, audio_tagging_loss=0.009262, over 3041195.45 frames. ], batch size: 53, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:55:57,042 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281600 2023-11-22 08:56:04,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1877366.6666666667, ans=0.125 2023-11-22 08:56:10,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.whiten.whitening_limit, batch_count=1877366.6666666667, ans=12.0 2023-11-22 08:56:19,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1877433.3333333333, ans=0.125 2023-11-22 08:56:28,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1877433.3333333333, ans=0.1 2023-11-22 08:56:29,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1877433.3333333333, ans=0.05 2023-11-22 08:56:30,952 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.25 vs. limit=15.0 2023-11-22 08:56:34,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1877500.0, ans=0.125 2023-11-22 08:56:50,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.56 vs. limit=15.0 2023-11-22 08:56:57,106 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5100, loss[loss=0.05602, simple_loss=0.07858, pruned_loss=0.00912, audio_tagging_loss=0.007605, over 15434.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09467, pruned_loss=0.01542, audio_tagging_loss=0.009282, over 3043162.57 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:56:58,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1877633.3333333333, ans=0.1 2023-11-22 08:56:59,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1877633.3333333333, ans=0.1 2023-11-22 08:57:00,794 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281650 2023-11-22 08:57:11,587 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.32 vs. limit=10.0 2023-11-22 08:57:24,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1877766.6666666667, ans=0.0 2023-11-22 08:57:34,830 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.920e+01 7.966e+01 8.782e+01 9.481e+01 2.193e+02, threshold=1.756e+02, percent-clipped=1.0 2023-11-22 08:57:36,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1877833.3333333333, ans=0.125 2023-11-22 08:57:50,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1877900.0, ans=10.0 2023-11-22 08:57:52,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.14 vs. limit=15.0 2023-11-22 08:57:52,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1877900.0, ans=0.0 2023-11-22 08:58:01,520 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5150, loss[loss=0.08191, simple_loss=0.1161, pruned_loss=0.0164, audio_tagging_loss=0.007453, over 15794.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09519, pruned_loss=0.01548, audio_tagging_loss=0.009229, over 3039016.13 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 08:58:05,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281700 2023-11-22 08:58:09,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.68 vs. limit=15.0 2023-11-22 08:58:18,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1878033.3333333333, ans=0.2 2023-11-22 08:58:19,080 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0 2023-11-22 08:58:32,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1878100.0, ans=0.125 2023-11-22 08:58:35,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.32 vs. limit=15.0 2023-11-22 08:58:37,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1878100.0, ans=0.2 2023-11-22 08:58:47,163 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.45 vs. limit=15.0 2023-11-22 08:59:04,826 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5200, loss[loss=0.06184, simple_loss=0.08637, pruned_loss=0.01126, audio_tagging_loss=0.007399, over 15655.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09571, pruned_loss=0.01544, audio_tagging_loss=0.009252, over 3041756.70 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 08:59:08,557 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281750 2023-11-22 08:59:42,534 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.104e+01 8.582e+01 9.336e+01 1.183e+02, threshold=1.716e+02, percent-clipped=0.0 2023-11-22 08:59:47,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1878500.0, ans=0.125 2023-11-22 09:00:09,535 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5250, loss[loss=0.07397, simple_loss=0.09251, pruned_loss=0.01588, audio_tagging_loss=0.01184, over 14913.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09593, pruned_loss=0.01548, audio_tagging_loss=0.009254, over 3038493.18 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 09:00:13,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281800 2023-11-22 09:00:25,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1878700.0, ans=0.5 2023-11-22 09:00:34,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1878766.6666666667, ans=0.125 2023-11-22 09:00:38,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1878766.6666666667, ans=0.1 2023-11-22 09:00:38,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1878766.6666666667, ans=0.125 2023-11-22 09:01:14,905 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5300, loss[loss=0.0853, simple_loss=0.1101, pruned_loss=0.01888, audio_tagging_loss=0.01136, over 15029.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09604, pruned_loss=0.01551, audio_tagging_loss=0.009243, over 3028873.87 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 09:01:18,629 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281850 2023-11-22 09:01:32,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1879033.3333333333, ans=0.1 2023-11-22 09:01:41,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1879100.0, ans=0.125 2023-11-22 09:01:44,351 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=12.0 2023-11-22 09:01:53,085 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.133e+01 8.772e+01 9.528e+01 1.202e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 09:02:19,282 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5350, loss[loss=0.07446, simple_loss=0.09674, pruned_loss=0.018, audio_tagging_loss=0.008088, over 15969.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09605, pruned_loss=0.01553, audio_tagging_loss=0.009144, over 3033928.29 frames. ], batch size: 63, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 09:02:23,075 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281900 2023-11-22 09:02:46,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.00 vs. limit=15.0 2023-11-22 09:02:49,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1879433.3333333333, ans=0.1 2023-11-22 09:03:16,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.20 vs. limit=15.0 2023-11-22 09:03:23,604 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5400, loss[loss=0.04751, simple_loss=0.059, pruned_loss=0.007453, audio_tagging_loss=0.01056, over 14921.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09535, pruned_loss=0.01535, audio_tagging_loss=0.00922, over 3036758.89 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 09:03:27,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 281950 2023-11-22 09:03:28,485 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.68 vs. limit=22.5 2023-11-22 09:03:30,080 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.21 vs. limit=6.0 2023-11-22 09:03:49,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1879766.6666666667, ans=0.025 2023-11-22 09:04:00,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.16 vs. limit=15.0 2023-11-22 09:04:02,265 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.289e+01 8.769e+01 9.344e+01 1.157e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 09:04:02,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1879833.3333333333, ans=0.125 2023-11-22 09:04:29,466 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5450, loss[loss=0.0709, simple_loss=0.09333, pruned_loss=0.01481, audio_tagging_loss=0.009421, over 15462.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09602, pruned_loss=0.01556, audio_tagging_loss=0.009246, over 3038368.39 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 09:04:33,198 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282000 2023-11-22 09:04:43,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1880033.3333333333, ans=0.05 2023-11-22 09:04:47,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1880033.3333333333, ans=0.0 2023-11-22 09:04:53,826 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.95 vs. limit=10.0 2023-11-22 09:04:56,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1880100.0, ans=0.125 2023-11-22 09:05:00,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1880100.0, ans=0.2 2023-11-22 09:05:03,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1880100.0, ans=0.125 2023-11-22 09:05:06,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1880166.6666666667, ans=0.125 2023-11-22 09:05:14,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1880166.6666666667, ans=0.0 2023-11-22 09:05:33,763 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5500, loss[loss=0.08959, simple_loss=0.1092, pruned_loss=0.0218, audio_tagging_loss=0.01321, over 14852.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09626, pruned_loss=0.01559, audio_tagging_loss=0.009207, over 3044564.98 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 09:05:34,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1880300.0, ans=0.1 2023-11-22 09:05:37,526 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282050 2023-11-22 09:05:40,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1880300.0, ans=0.0 2023-11-22 09:06:12,703 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.257e+01 8.828e+01 9.557e+01 1.200e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 09:06:27,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1880566.6666666667, ans=0.0 2023-11-22 09:06:34,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1880566.6666666667, ans=0.125 2023-11-22 09:06:38,157 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5550, loss[loss=0.06194, simple_loss=0.08454, pruned_loss=0.009946, audio_tagging_loss=0.009727, over 15642.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09698, pruned_loss=0.01582, audio_tagging_loss=0.009231, over 3048348.48 frames. ], batch size: 61, lr: 2.85e-03, grad_scale: 16.0 2023-11-22 09:06:41,951 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282100 2023-11-22 09:06:59,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1880700.0, ans=0.1 2023-11-22 09:07:25,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1880833.3333333333, ans=0.125 2023-11-22 09:07:28,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1880900.0, ans=0.1 2023-11-22 09:07:43,085 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5600, loss[loss=0.07803, simple_loss=0.1027, pruned_loss=0.01724, audio_tagging_loss=0.009426, over 14490.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09695, pruned_loss=0.01577, audio_tagging_loss=0.009327, over 3045164.36 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 32.0 2023-11-22 09:07:44,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1880966.6666666667, ans=0.1 2023-11-22 09:07:47,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282150 2023-11-22 09:07:50,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1880966.6666666667, ans=0.07 2023-11-22 09:08:21,231 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.179e+01 8.745e+01 9.416e+01 1.882e+02, threshold=1.749e+02, percent-clipped=1.0 2023-11-22 09:08:22,014 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=12.0 2023-11-22 09:08:30,639 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 09:08:31,142 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.59 vs. limit=15.0 2023-11-22 09:08:39,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1881233.3333333333, ans=0.125 2023-11-22 09:08:47,864 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5650, loss[loss=0.06294, simple_loss=0.07593, pruned_loss=0.01448, audio_tagging_loss=0.0105, over 14796.00 frames. ], tot_loss[loss=0.07319, simple_loss=0.09622, pruned_loss=0.01559, audio_tagging_loss=0.009489, over 3042808.83 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:08:51,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282200 2023-11-22 09:08:53,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.57 vs. limit=15.0 2023-11-22 09:08:55,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1881300.0, ans=0.1 2023-11-22 09:09:11,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1881366.6666666667, ans=0.1 2023-11-22 09:09:14,049 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.23 vs. limit=22.5 2023-11-22 09:09:20,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1881433.3333333333, ans=0.0 2023-11-22 09:09:34,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1881500.0, ans=0.0 2023-11-22 09:09:52,843 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5700, loss[loss=0.04089, simple_loss=0.05389, pruned_loss=0.004511, audio_tagging_loss=0.009441, over 14536.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09562, pruned_loss=0.01562, audio_tagging_loss=0.009429, over 3046613.12 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:09:56,666 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282250 2023-11-22 09:09:57,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1881633.3333333333, ans=0.2 2023-11-22 09:10:05,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1881700.0, ans=0.0 2023-11-22 09:10:30,655 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.915e+01 8.108e+01 8.715e+01 9.329e+01 1.149e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-22 09:10:48,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1881900.0, ans=0.2 2023-11-22 09:10:51,766 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.61 vs. limit=8.0 2023-11-22 09:10:54,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1881966.6666666667, ans=0.1 2023-11-22 09:10:55,634 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5750, loss[loss=0.0754, simple_loss=0.1063, pruned_loss=0.01495, audio_tagging_loss=0.007286, over 15457.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09484, pruned_loss=0.0154, audio_tagging_loss=0.009319, over 3044652.09 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:10:56,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1881966.6666666667, ans=0.125 2023-11-22 09:10:59,974 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282300 2023-11-22 09:11:06,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1881966.6666666667, ans=0.5 2023-11-22 09:11:59,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1882233.3333333333, ans=0.1 2023-11-22 09:12:01,180 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5800, loss[loss=0.07259, simple_loss=0.09863, pruned_loss=0.01656, audio_tagging_loss=0.006719, over 14597.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09401, pruned_loss=0.01536, audio_tagging_loss=0.009182, over 3041575.16 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:12:04,890 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282350 2023-11-22 09:12:22,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1882366.6666666667, ans=0.2 2023-11-22 09:12:36,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1882433.3333333333, ans=0.2 2023-11-22 09:12:38,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1882500.0, ans=0.04949747468305833 2023-11-22 09:12:40,750 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 8.216e+01 8.857e+01 9.622e+01 1.311e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-22 09:12:46,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1882500.0, ans=0.0 2023-11-22 09:13:05,431 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5850, loss[loss=0.06597, simple_loss=0.08506, pruned_loss=0.01386, audio_tagging_loss=0.009581, over 14783.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09344, pruned_loss=0.01543, audio_tagging_loss=0.009263, over 3043351.27 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:13:09,296 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282400 2023-11-22 09:13:09,468 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 09:13:14,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1882633.3333333333, ans=0.2 2023-11-22 09:13:35,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1882766.6666666667, ans=0.125 2023-11-22 09:13:37,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.69 vs. limit=15.0 2023-11-22 09:14:03,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1882900.0, ans=0.125 2023-11-22 09:14:07,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1882900.0, ans=0.0 2023-11-22 09:14:10,100 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5900, loss[loss=0.06385, simple_loss=0.08422, pruned_loss=0.01445, audio_tagging_loss=0.007288, over 15437.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09405, pruned_loss=0.01544, audio_tagging_loss=0.009292, over 3038611.88 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:14:14,461 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282450 2023-11-22 09:14:14,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1882966.6666666667, ans=0.1 2023-11-22 09:14:19,448 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.65 vs. limit=15.0 2023-11-22 09:14:31,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1883033.3333333333, ans=0.0 2023-11-22 09:14:31,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1883033.3333333333, ans=0.125 2023-11-22 09:14:47,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0 2023-11-22 09:14:50,064 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 8.181e+01 9.057e+01 9.624e+01 1.134e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-22 09:15:14,477 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 5950, loss[loss=0.06869, simple_loss=0.08532, pruned_loss=0.01465, audio_tagging_loss=0.01138, over 14991.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09361, pruned_loss=0.0154, audio_tagging_loss=0.009284, over 3039506.00 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:15:14,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1883300.0, ans=0.07 2023-11-22 09:15:18,795 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282500 2023-11-22 09:15:20,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1883300.0, ans=0.125 2023-11-22 09:15:27,673 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 09:15:49,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1883433.3333333333, ans=0.1 2023-11-22 09:15:59,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1883500.0, ans=0.125 2023-11-22 09:16:06,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1883566.6666666667, ans=0.0 2023-11-22 09:16:19,183 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6000, loss[loss=0.06617, simple_loss=0.08004, pruned_loss=0.01647, audio_tagging_loss=0.009676, over 15701.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09318, pruned_loss=0.01531, audio_tagging_loss=0.00927, over 3034843.40 frames. ], batch size: 61, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:16:19,183 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 09:17:01,474 INFO [train_asr.py:1253] (3/4) Epoch 24, validation: loss=0.05933, simple_loss=0.05174, pruned_loss=0.005222, audio_tagging_loss=0.02824, over 4681554.00 frames. 2023-11-22 09:17:01,475 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 09:17:04,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1883633.3333333333, ans=0.0 2023-11-22 09:17:05,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282550 2023-11-22 09:17:07,531 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.54 vs. limit=12.0 2023-11-22 09:17:19,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1883700.0, ans=0.125 2023-11-22 09:17:42,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1883833.3333333333, ans=0.95 2023-11-22 09:17:42,979 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.160e+01 8.694e+01 9.357e+01 1.133e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 09:17:49,660 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 09:17:49,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1883833.3333333333, ans=0.125 2023-11-22 09:18:01,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1883900.0, ans=0.2 2023-11-22 09:18:06,494 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6050, loss[loss=0.07085, simple_loss=0.09559, pruned_loss=0.01563, audio_tagging_loss=0.00742, over 16352.00 frames. ], tot_loss[loss=0.07153, simple_loss=0.09403, pruned_loss=0.01527, audio_tagging_loss=0.009245, over 3048583.76 frames. ], batch size: 60, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:18:10,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282600 2023-11-22 09:18:10,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1883966.6666666667, ans=0.1 2023-11-22 09:18:30,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1884033.3333333333, ans=0.125 2023-11-22 09:18:56,826 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.47 vs. limit=10.0 2023-11-22 09:19:08,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.41 vs. limit=22.5 2023-11-22 09:19:12,042 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6100, loss[loss=0.06095, simple_loss=0.0833, pruned_loss=0.009399, audio_tagging_loss=0.009897, over 15012.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09388, pruned_loss=0.01514, audio_tagging_loss=0.0092, over 3045560.93 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:19:15,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282650 2023-11-22 09:19:17,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1884300.0, ans=0.0 2023-11-22 09:19:32,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1884366.6666666667, ans=0.1 2023-11-22 09:19:53,492 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.538e+01 9.030e+01 1.014e+02 1.305e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-22 09:19:56,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1884500.0, ans=0.0 2023-11-22 09:20:04,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1884566.6666666667, ans=0.0 2023-11-22 09:20:16,943 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6150, loss[loss=0.08901, simple_loss=0.1208, pruned_loss=0.02089, audio_tagging_loss=0.007741, over 15437.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09343, pruned_loss=0.01527, audio_tagging_loss=0.009244, over 3049700.97 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:20:20,844 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282700 2023-11-22 09:20:20,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1884633.3333333333, ans=0.125 2023-11-22 09:20:22,535 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.53 vs. limit=15.0 2023-11-22 09:20:32,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1884700.0, ans=0.0 2023-11-22 09:21:21,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1884966.6666666667, ans=0.125 2023-11-22 09:21:22,137 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6200, loss[loss=0.07573, simple_loss=0.0972, pruned_loss=0.01849, audio_tagging_loss=0.008645, over 14763.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09367, pruned_loss=0.01523, audio_tagging_loss=0.009255, over 3050951.93 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:21:25,981 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282750 2023-11-22 09:21:31,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1884966.6666666667, ans=0.125 2023-11-22 09:21:55,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1885100.0, ans=0.125 2023-11-22 09:22:01,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1885166.6666666667, ans=0.2 2023-11-22 09:22:03,950 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.132e+01 8.774e+01 9.442e+01 1.403e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-22 09:22:15,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1885233.3333333333, ans=0.0 2023-11-22 09:22:18,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.25 vs. limit=15.0 2023-11-22 09:22:18,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1885233.3333333333, ans=0.0 2023-11-22 09:22:26,709 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6250, loss[loss=0.08942, simple_loss=0.1196, pruned_loss=0.0215, audio_tagging_loss=0.008128, over 14641.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09322, pruned_loss=0.01519, audio_tagging_loss=0.009393, over 3044455.61 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:22:31,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282800 2023-11-22 09:22:49,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1885366.6666666667, ans=0.2 2023-11-22 09:22:51,903 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.27 vs. limit=8.0 2023-11-22 09:22:53,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1885433.3333333333, ans=0.125 2023-11-22 09:23:13,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1885500.0, ans=0.0 2023-11-22 09:23:18,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1885566.6666666667, ans=0.2 2023-11-22 09:23:30,897 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6300, loss[loss=0.07187, simple_loss=0.09533, pruned_loss=0.01401, audio_tagging_loss=0.01019, over 15831.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09351, pruned_loss=0.01517, audio_tagging_loss=0.009467, over 3046451.30 frames. ], batch size: 61, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:23:35,191 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282850 2023-11-22 09:24:01,713 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.37 vs. limit=10.0 2023-11-22 09:24:11,932 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.966e+01 8.510e+01 9.202e+01 1.035e+02 1.385e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-22 09:24:20,033 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.42 vs. limit=15.0 2023-11-22 09:24:23,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1885900.0, ans=0.125 2023-11-22 09:24:31,952 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.83 vs. limit=12.0 2023-11-22 09:24:35,097 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6350, loss[loss=0.05244, simple_loss=0.06812, pruned_loss=0.007388, audio_tagging_loss=0.01099, over 14192.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09223, pruned_loss=0.01489, audio_tagging_loss=0.009562, over 3041990.97 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:24:38,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282900 2023-11-22 09:24:41,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0 2023-11-22 09:24:45,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1885966.6666666667, ans=0.2 2023-11-22 09:24:52,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1886033.3333333333, ans=0.0 2023-11-22 09:25:26,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.00 vs. limit=15.0 2023-11-22 09:25:27,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1886233.3333333333, ans=0.2 2023-11-22 09:25:34,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1886233.3333333333, ans=0.2 2023-11-22 09:25:38,856 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6400, loss[loss=0.0954, simple_loss=0.1172, pruned_loss=0.02744, audio_tagging_loss=0.009362, over 15736.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09311, pruned_loss=0.01503, audio_tagging_loss=0.009672, over 3043589.33 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:25:41,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1886300.0, ans=0.125 2023-11-22 09:25:43,215 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 282950 2023-11-22 09:25:50,638 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.57 vs. limit=15.0 2023-11-22 09:25:57,888 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.13 vs. limit=22.5 2023-11-22 09:26:02,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1886366.6666666667, ans=0.0 2023-11-22 09:26:02,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1886366.6666666667, ans=0.0 2023-11-22 09:26:06,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1886433.3333333333, ans=0.125 2023-11-22 09:26:20,179 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.176e+01 8.732e+01 9.554e+01 1.218e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-22 09:26:21,981 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.91 vs. limit=15.0 2023-11-22 09:26:43,529 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6450, loss[loss=0.07996, simple_loss=0.1036, pruned_loss=0.01816, audio_tagging_loss=0.01, over 15525.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09271, pruned_loss=0.01476, audio_tagging_loss=0.009706, over 3042584.83 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:26:47,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283000 2023-11-22 09:26:47,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1886633.3333333333, ans=0.1 2023-11-22 09:26:55,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1886700.0, ans=0.0 2023-11-22 09:26:57,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1886700.0, ans=0.1 2023-11-22 09:27:48,669 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6500, loss[loss=0.06919, simple_loss=0.0937, pruned_loss=0.0122, audio_tagging_loss=0.01014, over 14679.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09216, pruned_loss=0.0147, audio_tagging_loss=0.009658, over 3036432.65 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:27:52,326 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283050 2023-11-22 09:27:55,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1886966.6666666667, ans=0.1 2023-11-22 09:27:58,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1886966.6666666667, ans=0.09899494936611666 2023-11-22 09:28:07,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1887033.3333333333, ans=0.0 2023-11-22 09:28:12,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.41 vs. limit=15.0 2023-11-22 09:28:18,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1887100.0, ans=0.0 2023-11-22 09:28:30,693 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.224e+01 8.620e+01 9.457e+01 1.249e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-22 09:28:37,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1887166.6666666667, ans=0.0 2023-11-22 09:28:43,078 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.69 vs. limit=15.0 2023-11-22 09:28:47,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1887233.3333333333, ans=0.0 2023-11-22 09:28:52,083 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6550, loss[loss=0.07021, simple_loss=0.09109, pruned_loss=0.01559, audio_tagging_loss=0.009072, over 14817.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09242, pruned_loss=0.01484, audio_tagging_loss=0.00959, over 3041135.85 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:28:55,870 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283100 2023-11-22 09:29:14,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1887366.6666666667, ans=0.1 2023-11-22 09:29:39,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1887500.0, ans=0.1 2023-11-22 09:29:56,335 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6600, loss[loss=0.05618, simple_loss=0.06927, pruned_loss=0.0134, audio_tagging_loss=0.00814, over 14676.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09261, pruned_loss=0.01499, audio_tagging_loss=0.009433, over 3044363.06 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:29:59,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283150 2023-11-22 09:30:03,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.46 vs. limit=22.5 2023-11-22 09:30:38,601 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.912e+01 8.247e+01 8.842e+01 9.573e+01 1.466e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-22 09:30:45,116 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.87 vs. limit=10.0 2023-11-22 09:30:46,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=15.0 2023-11-22 09:31:00,484 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6650, loss[loss=0.089, simple_loss=0.1126, pruned_loss=0.02344, audio_tagging_loss=0.009257, over 15095.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09321, pruned_loss=0.01503, audio_tagging_loss=0.009384, over 3038253.62 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:31:04,242 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283200 2023-11-22 09:31:12,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1888033.3333333333, ans=0.125 2023-11-22 09:31:13,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1888033.3333333333, ans=0.025 2023-11-22 09:31:13,737 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.52 vs. limit=15.0 2023-11-22 09:31:13,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.74 vs. limit=10.0 2023-11-22 09:31:17,502 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.83 vs. limit=22.5 2023-11-22 09:31:22,370 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.61 vs. limit=6.0 2023-11-22 09:31:28,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1888100.0, ans=0.0 2023-11-22 09:31:37,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.33 vs. limit=15.0 2023-11-22 09:31:45,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1888166.6666666667, ans=0.125 2023-11-22 09:32:00,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1888233.3333333333, ans=0.0 2023-11-22 09:32:04,415 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6700, loss[loss=0.06845, simple_loss=0.08657, pruned_loss=0.01622, audio_tagging_loss=0.008939, over 14423.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09312, pruned_loss=0.01505, audio_tagging_loss=0.009377, over 3036865.00 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:32:08,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283250 2023-11-22 09:32:47,398 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.099e+01 8.643e+01 9.373e+01 1.139e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-22 09:32:54,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=1888500.0, ans=15.0 2023-11-22 09:33:08,701 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6750, loss[loss=0.06604, simple_loss=0.08421, pruned_loss=0.01436, audio_tagging_loss=0.009579, over 14286.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09323, pruned_loss=0.01518, audio_tagging_loss=0.009338, over 3035665.41 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:33:12,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.44 vs. limit=15.0 2023-11-22 09:33:12,468 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283300 2023-11-22 09:33:19,265 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.32 vs. limit=15.0 2023-11-22 09:33:26,254 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.67 vs. limit=22.5 2023-11-22 09:33:38,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1888766.6666666667, ans=0.125 2023-11-22 09:33:39,836 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.80 vs. limit=22.5 2023-11-22 09:33:41,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1888766.6666666667, ans=0.2 2023-11-22 09:33:53,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1888833.3333333333, ans=0.05 2023-11-22 09:33:58,745 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.71 vs. limit=6.0 2023-11-22 09:34:10,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1888900.0, ans=0.125 2023-11-22 09:34:12,850 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6800, loss[loss=0.0493, simple_loss=0.05648, pruned_loss=0.01011, audio_tagging_loss=0.01095, over 14232.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09292, pruned_loss=0.01507, audio_tagging_loss=0.009319, over 3025569.28 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:34:15,136 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.12 vs. limit=15.0 2023-11-22 09:34:17,143 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283350 2023-11-22 09:34:17,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1888966.6666666667, ans=0.1 2023-11-22 09:34:30,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1889033.3333333333, ans=0.125 2023-11-22 09:34:48,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1889100.0, ans=0.0 2023-11-22 09:34:54,668 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 7.933e+01 8.700e+01 9.598e+01 1.312e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 09:34:55,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1889166.6666666667, ans=0.2 2023-11-22 09:35:00,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1889166.6666666667, ans=0.125 2023-11-22 09:35:15,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1889233.3333333333, ans=0.125 2023-11-22 09:35:15,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.77 vs. limit=15.0 2023-11-22 09:35:17,329 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6850, loss[loss=0.07732, simple_loss=0.1048, pruned_loss=0.01403, audio_tagging_loss=0.01091, over 15840.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09388, pruned_loss=0.01522, audio_tagging_loss=0.009262, over 3030699.31 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:35:21,131 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283400 2023-11-22 09:35:54,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1889433.3333333333, ans=0.0 2023-11-22 09:36:09,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1889566.6666666667, ans=0.05 2023-11-22 09:36:21,966 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6900, loss[loss=0.06764, simple_loss=0.1007, pruned_loss=0.01048, audio_tagging_loss=0.006831, over 15536.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09323, pruned_loss=0.01512, audio_tagging_loss=0.009268, over 3034880.55 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:36:25,815 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283450 2023-11-22 09:36:37,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1889700.0, ans=0.125 2023-11-22 09:36:49,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1889766.6666666667, ans=0.125 2023-11-22 09:36:58,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1889766.6666666667, ans=0.0 2023-11-22 09:37:03,973 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.035e+01 8.885e+01 9.571e+01 1.179e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 09:37:13,106 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 09:37:19,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1889900.0, ans=0.125 2023-11-22 09:37:22,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=15.0 2023-11-22 09:37:25,251 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=15.0 2023-11-22 09:37:25,878 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 6950, loss[loss=0.06666, simple_loss=0.0845, pruned_loss=0.01325, audio_tagging_loss=0.01116, over 15177.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09392, pruned_loss=0.01516, audio_tagging_loss=0.009262, over 3039562.35 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:37:28,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1889966.6666666667, ans=0.0 2023-11-22 09:37:30,224 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283500 2023-11-22 09:37:32,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1889966.6666666667, ans=0.125 2023-11-22 09:37:43,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1890033.3333333333, ans=0.1 2023-11-22 09:37:48,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1890033.3333333333, ans=0.0 2023-11-22 09:37:58,543 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 09:38:06,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1890166.6666666667, ans=0.125 2023-11-22 09:38:10,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1890166.6666666667, ans=0.035 2023-11-22 09:38:31,035 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7000, loss[loss=0.06834, simple_loss=0.09038, pruned_loss=0.01258, audio_tagging_loss=0.01057, over 15052.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09501, pruned_loss=0.01517, audio_tagging_loss=0.009269, over 3042885.11 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:38:34,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283550 2023-11-22 09:38:58,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1890433.3333333333, ans=0.125 2023-11-22 09:39:12,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1890500.0, ans=0.1 2023-11-22 09:39:13,198 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.396e+01 8.305e+01 8.768e+01 9.472e+01 1.582e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 09:39:24,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1890566.6666666667, ans=0.0 2023-11-22 09:39:35,612 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7050, loss[loss=0.07304, simple_loss=0.09468, pruned_loss=0.01546, audio_tagging_loss=0.01024, over 15190.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09434, pruned_loss=0.01509, audio_tagging_loss=0.009285, over 3042206.18 frames. ], batch size: 62, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:39:39,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283600 2023-11-22 09:39:52,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1890700.0, ans=0.1 2023-11-22 09:40:16,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1890833.3333333333, ans=0.2 2023-11-22 09:40:21,413 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.12 vs. limit=22.5 2023-11-22 09:40:39,974 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7100, loss[loss=0.09737, simple_loss=0.1268, pruned_loss=0.02594, audio_tagging_loss=0.008046, over 15255.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09429, pruned_loss=0.0152, audio_tagging_loss=0.009392, over 3043234.66 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:40:43,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1890966.6666666667, ans=0.0 2023-11-22 09:40:44,395 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283650 2023-11-22 09:40:53,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1891033.3333333333, ans=0.125 2023-11-22 09:40:59,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.67 vs. limit=6.0 2023-11-22 09:41:00,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1891033.3333333333, ans=0.125 2023-11-22 09:41:06,062 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=15.0 2023-11-22 09:41:06,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1891100.0, ans=0.1 2023-11-22 09:41:07,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1891100.0, ans=0.125 2023-11-22 09:41:12,196 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.70 vs. limit=6.0 2023-11-22 09:41:16,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1891100.0, ans=0.0 2023-11-22 09:41:23,124 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.301e+01 8.958e+01 9.633e+01 1.183e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-22 09:41:24,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1891166.6666666667, ans=0.1 2023-11-22 09:41:44,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1891300.0, ans=0.125 2023-11-22 09:41:45,095 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7150, loss[loss=0.07657, simple_loss=0.09827, pruned_loss=0.01674, audio_tagging_loss=0.01069, over 15462.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09453, pruned_loss=0.01533, audio_tagging_loss=0.009383, over 3051100.84 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:41:48,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283700 2023-11-22 09:41:55,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.51 vs. limit=10.0 2023-11-22 09:41:57,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1891366.6666666667, ans=0.2 2023-11-22 09:41:58,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1891366.6666666667, ans=0.125 2023-11-22 09:42:29,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1891500.0, ans=0.1 2023-11-22 09:42:33,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1891500.0, ans=0.07 2023-11-22 09:42:39,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1891566.6666666667, ans=0.04949747468305833 2023-11-22 09:42:49,832 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7200, loss[loss=0.07316, simple_loss=0.1025, pruned_loss=0.01378, audio_tagging_loss=0.008151, over 16793.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09373, pruned_loss=0.01509, audio_tagging_loss=0.009472, over 3053727.28 frames. ], batch size: 62, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:42:54,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283750 2023-11-22 09:43:14,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1891766.6666666667, ans=0.95 2023-11-22 09:43:32,272 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.224e+01 8.056e+01 8.758e+01 9.674e+01 1.275e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-22 09:43:37,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1891833.3333333333, ans=0.1 2023-11-22 09:43:42,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1891900.0, ans=0.2 2023-11-22 09:43:49,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1891900.0, ans=0.0 2023-11-22 09:43:50,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1891900.0, ans=0.0 2023-11-22 09:43:50,895 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.59 vs. limit=6.0 2023-11-22 09:43:51,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0 2023-11-22 09:43:54,010 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7250, loss[loss=0.07239, simple_loss=0.1, pruned_loss=0.0131, audio_tagging_loss=0.009273, over 15290.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09344, pruned_loss=0.01509, audio_tagging_loss=0.009584, over 3047248.91 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:43:58,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283800 2023-11-22 09:43:58,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1891966.6666666667, ans=0.125 2023-11-22 09:44:14,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1892033.3333333333, ans=0.125 2023-11-22 09:44:24,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1892100.0, ans=0.09899494936611666 2023-11-22 09:44:39,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.71 vs. limit=22.5 2023-11-22 09:44:40,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1892166.6666666667, ans=0.1 2023-11-22 09:44:59,086 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7300, loss[loss=0.06991, simple_loss=0.09048, pruned_loss=0.01765, audio_tagging_loss=0.007025, over 15194.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09477, pruned_loss=0.01523, audio_tagging_loss=0.009444, over 3050145.89 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:44:59,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1892300.0, ans=0.125 2023-11-22 09:45:02,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283850 2023-11-22 09:45:30,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1892433.3333333333, ans=0.0 2023-11-22 09:45:31,468 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.21 vs. limit=15.0 2023-11-22 09:45:37,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1892500.0, ans=0.125 2023-11-22 09:45:41,608 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.095e+01 8.850e+01 9.580e+01 1.188e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-22 09:45:54,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1892566.6666666667, ans=0.0 2023-11-22 09:46:02,632 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7350, loss[loss=0.0751, simple_loss=0.1022, pruned_loss=0.01606, audio_tagging_loss=0.007926, over 14468.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09432, pruned_loss=0.0153, audio_tagging_loss=0.009292, over 3046720.42 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:46:06,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283900 2023-11-22 09:46:13,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1892633.3333333333, ans=0.1 2023-11-22 09:46:37,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1892766.6666666667, ans=0.125 2023-11-22 09:46:53,565 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0 2023-11-22 09:46:58,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1892900.0, ans=0.125 2023-11-22 09:47:06,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1892966.6666666667, ans=0.05 2023-11-22 09:47:07,025 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7400, loss[loss=0.06698, simple_loss=0.0926, pruned_loss=0.01011, audio_tagging_loss=0.01058, over 13885.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09444, pruned_loss=0.01527, audio_tagging_loss=0.00924, over 3051666.33 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:47:10,900 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 283950 2023-11-22 09:47:24,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1893033.3333333333, ans=0.125 2023-11-22 09:47:29,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1893033.3333333333, ans=0.125 2023-11-22 09:47:34,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1893100.0, ans=0.0 2023-11-22 09:47:38,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1893100.0, ans=0.125 2023-11-22 09:47:50,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1893166.6666666667, ans=0.2 2023-11-22 09:47:51,343 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.493e+01 8.994e+01 9.861e+01 1.121e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 09:48:08,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1893233.3333333333, ans=0.0 2023-11-22 09:48:12,162 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7450, loss[loss=0.07728, simple_loss=0.1085, pruned_loss=0.01574, audio_tagging_loss=0.007301, over 15135.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09516, pruned_loss=0.01542, audio_tagging_loss=0.009171, over 3048133.09 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:48:13,643 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 09:48:14,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.09 vs. limit=15.0 2023-11-22 09:48:15,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284000 2023-11-22 09:48:28,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1893366.6666666667, ans=0.125 2023-11-22 09:48:48,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1893433.3333333333, ans=0.0 2023-11-22 09:48:58,352 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.50 vs. limit=10.0 2023-11-22 09:49:10,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1893566.6666666667, ans=0.0 2023-11-22 09:49:19,187 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7500, loss[loss=0.07328, simple_loss=0.1005, pruned_loss=0.01616, audio_tagging_loss=0.006887, over 14990.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09554, pruned_loss=0.01562, audio_tagging_loss=0.009172, over 3047224.95 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:49:22,900 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284050 2023-11-22 09:49:28,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1893633.3333333333, ans=0.0 2023-11-22 09:49:55,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1893766.6666666667, ans=0.1 2023-11-22 09:50:02,452 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.170e+01 8.839e+01 9.515e+01 1.142e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-22 09:50:12,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1893900.0, ans=0.2 2023-11-22 09:50:17,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1893900.0, ans=0.1 2023-11-22 09:50:23,609 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7550, loss[loss=0.08668, simple_loss=0.1147, pruned_loss=0.01977, audio_tagging_loss=0.009575, over 14191.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09473, pruned_loss=0.01546, audio_tagging_loss=0.009222, over 3052359.64 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 16.0 2023-11-22 09:50:27,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284100 2023-11-22 09:50:33,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1893966.6666666667, ans=0.125 2023-11-22 09:50:35,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1894033.3333333333, ans=0.0 2023-11-22 09:50:41,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1894033.3333333333, ans=0.0 2023-11-22 09:50:43,715 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.70 vs. limit=15.0 2023-11-22 09:50:59,888 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.21 vs. limit=15.0 2023-11-22 09:51:02,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.51 vs. limit=22.5 2023-11-22 09:51:11,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1894166.6666666667, ans=0.125 2023-11-22 09:51:11,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1894166.6666666667, ans=0.125 2023-11-22 09:51:14,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1894233.3333333333, ans=0.125 2023-11-22 09:51:16,682 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 09:51:24,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1894233.3333333333, ans=0.0 2023-11-22 09:51:28,603 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7600, loss[loss=0.07223, simple_loss=0.1021, pruned_loss=0.01562, audio_tagging_loss=0.005577, over 15052.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09462, pruned_loss=0.01536, audio_tagging_loss=0.009291, over 3049285.39 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0 2023-11-22 09:51:32,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284150 2023-11-22 09:51:47,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1894366.6666666667, ans=0.125 2023-11-22 09:51:51,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1894366.6666666667, ans=15.0 2023-11-22 09:52:12,159 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.245e+01 8.957e+01 9.580e+01 1.201e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-22 09:52:16,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1894500.0, ans=0.125 2023-11-22 09:52:27,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1894566.6666666667, ans=0.125 2023-11-22 09:52:32,256 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7650, loss[loss=0.07561, simple_loss=0.1053, pruned_loss=0.01548, audio_tagging_loss=0.007509, over 15762.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09477, pruned_loss=0.01526, audio_tagging_loss=0.009235, over 3051271.27 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 09:52:36,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284200 2023-11-22 09:52:50,781 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.34 vs. limit=15.0 2023-11-22 09:52:58,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0 2023-11-22 09:53:07,848 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=15.0 2023-11-22 09:53:14,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1894833.3333333333, ans=0.1 2023-11-22 09:53:18,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1894833.3333333333, ans=0.1 2023-11-22 09:53:22,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1894833.3333333333, ans=0.0 2023-11-22 09:53:36,915 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7700, loss[loss=0.05834, simple_loss=0.07016, pruned_loss=0.01077, audio_tagging_loss=0.01249, over 14219.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09456, pruned_loss=0.01514, audio_tagging_loss=0.009168, over 3043541.80 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 09:53:41,328 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284250 2023-11-22 09:53:50,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1895033.3333333333, ans=0.2 2023-11-22 09:53:51,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1895033.3333333333, ans=0.1 2023-11-22 09:53:58,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.34 vs. limit=15.0 2023-11-22 09:54:00,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1895033.3333333333, ans=0.09899494936611666 2023-11-22 09:54:06,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1895100.0, ans=0.125 2023-11-22 09:54:16,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.90 vs. limit=15.0 2023-11-22 09:54:20,908 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 7.998e+01 8.702e+01 9.312e+01 1.429e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 09:54:39,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1895233.3333333333, ans=0.1 2023-11-22 09:54:41,582 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7750, loss[loss=0.05759, simple_loss=0.073, pruned_loss=0.01071, audio_tagging_loss=0.01038, over 15344.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09403, pruned_loss=0.01501, audio_tagging_loss=0.009255, over 3045785.65 frames. ], batch size: 61, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 09:54:43,606 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.62 vs. limit=22.5 2023-11-22 09:54:45,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284300 2023-11-22 09:54:50,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1895300.0, ans=0.125 2023-11-22 09:54:59,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1895366.6666666667, ans=0.125 2023-11-22 09:55:03,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1895366.6666666667, ans=0.025 2023-11-22 09:55:23,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1895500.0, ans=0.125 2023-11-22 09:55:45,648 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7800, loss[loss=0.05553, simple_loss=0.07461, pruned_loss=0.01172, audio_tagging_loss=0.006504, over 15630.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09436, pruned_loss=0.015, audio_tagging_loss=0.009239, over 3043476.97 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 09:55:49,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284350 2023-11-22 09:55:55,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1895633.3333333333, ans=0.125 2023-11-22 09:56:04,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1895700.0, ans=0.125 2023-11-22 09:56:17,204 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=12.0 2023-11-22 09:56:29,306 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.112e+01 8.791e+01 9.537e+01 1.111e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 09:56:37,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=12.0 2023-11-22 09:56:45,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1895900.0, ans=0.125 2023-11-22 09:56:49,593 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7850, loss[loss=0.07351, simple_loss=0.1016, pruned_loss=0.01406, audio_tagging_loss=0.008643, over 14829.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09486, pruned_loss=0.0152, audio_tagging_loss=0.009401, over 3045845.83 frames. ], batch size: 54, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 09:56:53,400 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284400 2023-11-22 09:57:00,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1895966.6666666667, ans=0.125 2023-11-22 09:57:01,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1896033.3333333333, ans=0.125 2023-11-22 09:57:06,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1896033.3333333333, ans=0.125 2023-11-22 09:57:20,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.46 vs. limit=22.5 2023-11-22 09:57:27,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1896166.6666666667, ans=0.05 2023-11-22 09:57:33,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1896166.6666666667, ans=0.125 2023-11-22 09:57:55,012 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7900, loss[loss=0.07352, simple_loss=0.1009, pruned_loss=0.01338, audio_tagging_loss=0.009704, over 16019.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09518, pruned_loss=0.01525, audio_tagging_loss=0.009452, over 3043760.24 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 09:57:59,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284450 2023-11-22 09:58:31,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1896500.0, ans=0.125 2023-11-22 09:58:34,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1896500.0, ans=0.125 2023-11-22 09:58:37,543 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.160e+01 8.848e+01 9.602e+01 1.825e+02, threshold=1.770e+02, percent-clipped=1.0 2023-11-22 09:58:38,211 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.01 vs. limit=15.0 2023-11-22 09:58:44,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1896566.6666666667, ans=0.0 2023-11-22 09:58:44,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1896566.6666666667, ans=0.125 2023-11-22 09:58:58,642 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 7950, loss[loss=0.06948, simple_loss=0.08822, pruned_loss=0.01359, audio_tagging_loss=0.01178, over 16285.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09389, pruned_loss=0.01527, audio_tagging_loss=0.009609, over 3038456.93 frames. ], batch size: 62, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 09:59:02,469 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284500 2023-11-22 09:59:15,232 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 09:59:22,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=1896766.6666666667, ans=0.1 2023-11-22 09:59:25,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1896766.6666666667, ans=0.125 2023-11-22 10:00:01,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1896966.6666666667, ans=0.0 2023-11-22 10:00:01,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1896966.6666666667, ans=0.0 2023-11-22 10:00:02,902 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8000, loss[loss=0.05234, simple_loss=0.07004, pruned_loss=0.008065, audio_tagging_loss=0.009254, over 15726.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.0929, pruned_loss=0.01503, audio_tagging_loss=0.009584, over 3040052.85 frames. ], batch size: 62, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:00:05,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1896966.6666666667, ans=0.125 2023-11-22 10:00:06,775 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284550 2023-11-22 10:00:47,710 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.187e+01 8.649e+01 9.156e+01 1.196e+02, threshold=1.730e+02, percent-clipped=0.0 2023-11-22 10:00:57,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1897233.3333333333, ans=0.125 2023-11-22 10:01:06,848 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8050, loss[loss=0.0788, simple_loss=0.1058, pruned_loss=0.01883, audio_tagging_loss=0.007082, over 15999.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09279, pruned_loss=0.015, audio_tagging_loss=0.009712, over 3038836.68 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:01:11,159 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284600 2023-11-22 10:01:26,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1897366.6666666667, ans=0.125 2023-11-22 10:01:32,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1897433.3333333333, ans=0.125 2023-11-22 10:01:34,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1897433.3333333333, ans=0.0 2023-11-22 10:01:47,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1897500.0, ans=0.125 2023-11-22 10:01:48,812 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.506e-03 2023-11-22 10:01:57,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1897566.6666666667, ans=0.125 2023-11-22 10:01:58,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1897566.6666666667, ans=0.1 2023-11-22 10:02:03,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1897566.6666666667, ans=0.0 2023-11-22 10:02:12,576 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8100, loss[loss=0.07644, simple_loss=0.1015, pruned_loss=0.0177, audio_tagging_loss=0.007988, over 15393.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09329, pruned_loss=0.01505, audio_tagging_loss=0.009554, over 3038147.26 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:02:14,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1897633.3333333333, ans=0.0 2023-11-22 10:02:16,380 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284650 2023-11-22 10:02:16,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1897633.3333333333, ans=0.125 2023-11-22 10:02:28,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1897700.0, ans=0.5 2023-11-22 10:02:47,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1897766.6666666667, ans=0.0 2023-11-22 10:02:55,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1897833.3333333333, ans=0.1 2023-11-22 10:02:59,268 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.417e+01 8.877e+01 9.599e+01 1.117e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 10:03:16,299 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8150, loss[loss=0.05582, simple_loss=0.07487, pruned_loss=0.01115, audio_tagging_loss=0.007239, over 15222.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09406, pruned_loss=0.01529, audio_tagging_loss=0.00941, over 3035996.38 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:03:20,672 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284700 2023-11-22 10:03:23,188 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 10:03:25,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1897966.6666666667, ans=0.1 2023-11-22 10:03:52,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1898100.0, ans=0.125 2023-11-22 10:03:55,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1898166.6666666667, ans=0.0 2023-11-22 10:03:57,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.99 vs. limit=10.0 2023-11-22 10:04:15,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1898233.3333333333, ans=0.1 2023-11-22 10:04:15,329 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.67 vs. limit=15.0 2023-11-22 10:04:17,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1898233.3333333333, ans=0.0 2023-11-22 10:04:17,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=1898233.3333333333, ans=15.0 2023-11-22 10:04:18,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1898233.3333333333, ans=0.0 2023-11-22 10:04:20,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1898300.0, ans=0.0 2023-11-22 10:04:20,966 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8200, loss[loss=0.0694, simple_loss=0.095, pruned_loss=0.01262, audio_tagging_loss=0.009275, over 15305.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09504, pruned_loss=0.01537, audio_tagging_loss=0.009272, over 3031814.07 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:04:22,286 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 10:04:25,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284750 2023-11-22 10:04:34,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1898366.6666666667, ans=0.125 2023-11-22 10:04:37,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1898366.6666666667, ans=0.125 2023-11-22 10:05:06,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1898500.0, ans=0.025 2023-11-22 10:05:06,963 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.783e+01 8.106e+01 8.680e+01 9.269e+01 1.329e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-22 10:05:15,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1898566.6666666667, ans=0.125 2023-11-22 10:05:17,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.18 vs. limit=15.0 2023-11-22 10:05:20,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1898566.6666666667, ans=0.125 2023-11-22 10:05:25,266 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8250, loss[loss=0.05938, simple_loss=0.0771, pruned_loss=0.01174, audio_tagging_loss=0.009094, over 16117.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.0958, pruned_loss=0.01556, audio_tagging_loss=0.00921, over 3036646.13 frames. ], batch size: 63, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:05:29,069 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284800 2023-11-22 10:05:37,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1898700.0, ans=0.2 2023-11-22 10:05:56,371 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.33 vs. limit=15.0 2023-11-22 10:06:01,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1898766.6666666667, ans=0.0 2023-11-22 10:06:11,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1898833.3333333333, ans=0.0 2023-11-22 10:06:26,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.63 vs. limit=15.0 2023-11-22 10:06:29,560 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8300, loss[loss=0.08276, simple_loss=0.1077, pruned_loss=0.0199, audio_tagging_loss=0.009023, over 17223.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09627, pruned_loss=0.01555, audio_tagging_loss=0.009237, over 3036410.57 frames. ], batch size: 62, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:06:30,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1898966.6666666667, ans=0.0 2023-11-22 10:06:33,255 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284850 2023-11-22 10:06:45,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1899033.3333333333, ans=0.125 2023-11-22 10:06:48,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1899033.3333333333, ans=0.125 2023-11-22 10:06:48,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1899033.3333333333, ans=0.2 2023-11-22 10:06:49,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.25 vs. limit=15.0 2023-11-22 10:06:54,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1899100.0, ans=0.1 2023-11-22 10:06:59,964 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.32 vs. limit=15.0 2023-11-22 10:07:02,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.51 vs. limit=15.0 2023-11-22 10:07:15,707 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.988e+01 8.571e+01 9.748e+01 1.296e+02, threshold=1.714e+02, percent-clipped=0.0 2023-11-22 10:07:20,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1899233.3333333333, ans=0.5 2023-11-22 10:07:33,339 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8350, loss[loss=0.05478, simple_loss=0.06267, pruned_loss=0.01144, audio_tagging_loss=0.012, over 13962.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09648, pruned_loss=0.01558, audio_tagging_loss=0.009172, over 3042417.90 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:07:37,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284900 2023-11-22 10:07:50,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1899366.6666666667, ans=0.05 2023-11-22 10:08:17,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1899500.0, ans=0.0 2023-11-22 10:08:26,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1899566.6666666667, ans=0.125 2023-11-22 10:08:33,532 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0 2023-11-22 10:08:38,227 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8400, loss[loss=0.06602, simple_loss=0.08448, pruned_loss=0.01426, audio_tagging_loss=0.009523, over 14081.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09546, pruned_loss=0.0153, audio_tagging_loss=0.009221, over 3043046.42 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:08:41,925 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 284950 2023-11-22 10:08:47,445 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.88 vs. limit=15.0 2023-11-22 10:09:09,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1899766.6666666667, ans=0.125 2023-11-22 10:09:24,814 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.199e+01 8.877e+01 9.641e+01 1.238e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 10:09:26,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1899833.3333333333, ans=0.125 2023-11-22 10:09:42,473 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8450, loss[loss=0.07373, simple_loss=0.09507, pruned_loss=0.016, audio_tagging_loss=0.01019, over 15260.00 frames. ], tot_loss[loss=0.07164, simple_loss=0.09458, pruned_loss=0.01509, audio_tagging_loss=0.009262, over 3048836.00 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:09:45,266 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 10:09:46,294 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285000 2023-11-22 10:09:55,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1900033.3333333333, ans=0.125 2023-11-22 10:09:59,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1900033.3333333333, ans=0.0 2023-11-22 10:10:05,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1900033.3333333333, ans=0.0 2023-11-22 10:10:09,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1900100.0, ans=0.125 2023-11-22 10:10:11,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1900100.0, ans=10.0 2023-11-22 10:10:25,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1900166.6666666667, ans=0.2 2023-11-22 10:10:41,166 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.66 vs. limit=15.0 2023-11-22 10:10:47,747 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8500, loss[loss=0.07912, simple_loss=0.1026, pruned_loss=0.02047, audio_tagging_loss=0.007336, over 14951.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09442, pruned_loss=0.01522, audio_tagging_loss=0.009317, over 3047661.51 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:10:48,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1900300.0, ans=0.125 2023-11-22 10:10:51,622 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285050 2023-11-22 10:10:57,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1900300.0, ans=0.125 2023-11-22 10:11:33,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.84 vs. limit=15.0 2023-11-22 10:11:33,986 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.621e+01 8.277e+01 9.051e+01 9.526e+01 1.493e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-22 10:11:38,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1900566.6666666667, ans=0.1 2023-11-22 10:11:52,684 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8550, loss[loss=0.07329, simple_loss=0.09935, pruned_loss=0.01503, audio_tagging_loss=0.008583, over 16037.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09455, pruned_loss=0.01543, audio_tagging_loss=0.00932, over 3051794.06 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:11:52,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1900633.3333333333, ans=0.2 2023-11-22 10:11:53,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.08 vs. limit=15.0 2023-11-22 10:11:56,449 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285100 2023-11-22 10:12:16,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1900766.6666666667, ans=0.1 2023-11-22 10:12:18,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1900766.6666666667, ans=0.1 2023-11-22 10:12:30,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1900833.3333333333, ans=0.0 2023-11-22 10:12:39,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1900833.3333333333, ans=0.95 2023-11-22 10:12:41,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1900833.3333333333, ans=0.125 2023-11-22 10:12:46,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1900900.0, ans=0.0 2023-11-22 10:12:55,438 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=11.11 vs. limit=12.0 2023-11-22 10:12:55,958 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8600, loss[loss=0.05559, simple_loss=0.06773, pruned_loss=0.01131, audio_tagging_loss=0.01042, over 14960.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09444, pruned_loss=0.01537, audio_tagging_loss=0.009349, over 3053000.78 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:12:57,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1900966.6666666667, ans=0.1 2023-11-22 10:12:59,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285150 2023-11-22 10:13:00,262 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.71 vs. limit=15.0 2023-11-22 10:13:37,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1901166.6666666667, ans=0.0 2023-11-22 10:13:41,806 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.761e+01 8.269e+01 8.846e+01 9.645e+01 1.157e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 10:13:44,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1901166.6666666667, ans=0.125 2023-11-22 10:13:59,992 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8650, loss[loss=0.07594, simple_loss=0.1069, pruned_loss=0.01382, audio_tagging_loss=0.008666, over 14891.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.0938, pruned_loss=0.01515, audio_tagging_loss=0.009462, over 3049918.98 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:14:02,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1901300.0, ans=0.1 2023-11-22 10:14:03,898 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285200 2023-11-22 10:14:12,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1901366.6666666667, ans=0.0 2023-11-22 10:14:19,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1901366.6666666667, ans=0.125 2023-11-22 10:14:42,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1901500.0, ans=0.125 2023-11-22 10:15:01,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1901566.6666666667, ans=0.0 2023-11-22 10:15:03,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1901566.6666666667, ans=0.125 2023-11-22 10:15:04,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1901633.3333333333, ans=0.0 2023-11-22 10:15:05,255 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8700, loss[loss=0.08154, simple_loss=0.105, pruned_loss=0.02067, audio_tagging_loss=0.008391, over 15252.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09427, pruned_loss=0.01527, audio_tagging_loss=0.009516, over 3057171.32 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:15:09,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285250 2023-11-22 10:15:19,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1901700.0, ans=0.125 2023-11-22 10:15:31,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1901766.6666666667, ans=0.125 2023-11-22 10:15:39,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1901766.6666666667, ans=0.2 2023-11-22 10:15:40,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1901766.6666666667, ans=0.125 2023-11-22 10:15:41,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1901766.6666666667, ans=0.125 2023-11-22 10:15:51,789 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.447e+01 8.934e+01 9.629e+01 1.298e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-22 10:16:07,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1901900.0, ans=10.0 2023-11-22 10:16:09,480 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8750, loss[loss=0.07251, simple_loss=0.09713, pruned_loss=0.0157, audio_tagging_loss=0.008243, over 15072.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09466, pruned_loss=0.01529, audio_tagging_loss=0.009534, over 3055474.87 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:16:13,236 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285300 2023-11-22 10:16:22,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1902033.3333333333, ans=0.1 2023-11-22 10:16:47,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1902166.6666666667, ans=0.1 2023-11-22 10:17:13,272 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8800, loss[loss=0.06629, simple_loss=0.08035, pruned_loss=0.01569, audio_tagging_loss=0.01043, over 13802.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09539, pruned_loss=0.01544, audio_tagging_loss=0.009572, over 3051640.60 frames. ], batch size: 53, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:17:16,998 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285350 2023-11-22 10:17:17,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0 2023-11-22 10:17:30,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1902366.6666666667, ans=0.2 2023-11-22 10:17:30,879 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.60 vs. limit=15.0 2023-11-22 10:17:34,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1902366.6666666667, ans=0.125 2023-11-22 10:17:44,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1902433.3333333333, ans=10.0 2023-11-22 10:17:53,672 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.48 vs. limit=22.5 2023-11-22 10:17:59,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.363e+01 9.001e+01 9.837e+01 1.230e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-22 10:18:10,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1902566.6666666667, ans=0.125 2023-11-22 10:18:16,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1902633.3333333333, ans=0.125 2023-11-22 10:18:17,783 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8850, loss[loss=0.05734, simple_loss=0.07192, pruned_loss=0.01085, audio_tagging_loss=0.01054, over 15989.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09406, pruned_loss=0.01513, audio_tagging_loss=0.009563, over 3051576.93 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:18:21,573 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285400 2023-11-22 10:18:32,188 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 10:18:32,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1902700.0, ans=0.125 2023-11-22 10:18:36,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.whiten.whitening_limit, batch_count=1902700.0, ans=12.0 2023-11-22 10:18:50,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1902766.6666666667, ans=22.5 2023-11-22 10:18:58,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1902833.3333333333, ans=0.125 2023-11-22 10:19:22,936 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8900, loss[loss=0.06384, simple_loss=0.08139, pruned_loss=0.01321, audio_tagging_loss=0.009935, over 14491.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09387, pruned_loss=0.01501, audio_tagging_loss=0.009391, over 3051751.46 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:19:26,671 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285450 2023-11-22 10:19:42,637 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.45 vs. limit=6.0 2023-11-22 10:19:57,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1903100.0, ans=0.2 2023-11-22 10:20:07,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1903166.6666666667, ans=0.125 2023-11-22 10:20:10,561 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.009e+01 8.748e+01 9.224e+01 1.095e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 10:20:12,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1903166.6666666667, ans=0.0 2023-11-22 10:20:13,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1903233.3333333333, ans=0.0 2023-11-22 10:20:26,847 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 8950, loss[loss=0.06621, simple_loss=0.09046, pruned_loss=0.01238, audio_tagging_loss=0.008593, over 14365.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09498, pruned_loss=0.01521, audio_tagging_loss=0.009162, over 3050620.77 frames. ], batch size: 53, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:20:30,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285500 2023-11-22 10:21:18,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1903566.6666666667, ans=0.125 2023-11-22 10:21:30,149 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9000, loss[loss=0.06764, simple_loss=0.08652, pruned_loss=0.0131, audio_tagging_loss=0.01128, over 14773.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09491, pruned_loss=0.01521, audio_tagging_loss=0.009106, over 3048820.23 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:21:30,149 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 10:22:12,110 INFO [train_asr.py:1253] (3/4) Epoch 24, validation: loss=0.06037, simple_loss=0.05165, pruned_loss=0.00517, audio_tagging_loss=0.02938, over 4681554.00 frames. 2023-11-22 10:22:12,111 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 10:22:15,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285550 2023-11-22 10:22:52,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1903833.3333333333, ans=0.125 2023-11-22 10:22:54,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1903833.3333333333, ans=0.1 2023-11-22 10:22:59,082 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.344e+01 8.279e+01 8.807e+01 9.755e+01 1.198e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-22 10:23:12,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1903900.0, ans=0.125 2023-11-22 10:23:15,771 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9050, loss[loss=0.06506, simple_loss=0.08303, pruned_loss=0.01473, audio_tagging_loss=0.008811, over 14802.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09429, pruned_loss=0.01507, audio_tagging_loss=0.009105, over 3053261.30 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:23:19,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285600 2023-11-22 10:23:29,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1904033.3333333333, ans=0.125 2023-11-22 10:23:41,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1904100.0, ans=0.0 2023-11-22 10:23:50,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1904100.0, ans=0.0 2023-11-22 10:23:50,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1904100.0, ans=0.0 2023-11-22 10:24:18,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1904300.0, ans=0.0 2023-11-22 10:24:20,411 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9100, loss[loss=0.07461, simple_loss=0.09292, pruned_loss=0.0157, audio_tagging_loss=0.01245, over 14231.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.0937, pruned_loss=0.01493, audio_tagging_loss=0.00918, over 3052963.62 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:24:24,064 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285650 2023-11-22 10:24:55,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1904433.3333333333, ans=0.0 2023-11-22 10:24:57,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1904500.0, ans=0.125 2023-11-22 10:24:57,696 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.96 vs. limit=15.0 2023-11-22 10:25:07,533 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.735e+01 8.029e+01 8.849e+01 9.580e+01 1.187e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-22 10:25:24,632 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9150, loss[loss=0.07426, simple_loss=0.09967, pruned_loss=0.01644, audio_tagging_loss=0.007985, over 15567.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09462, pruned_loss=0.01504, audio_tagging_loss=0.00914, over 3058947.51 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 16.0 2023-11-22 10:25:28,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285700 2023-11-22 10:25:47,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1904700.0, ans=0.125 2023-11-22 10:25:54,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1904766.6666666667, ans=0.125 2023-11-22 10:25:55,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1904766.6666666667, ans=0.125 2023-11-22 10:26:15,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1904900.0, ans=0.05 2023-11-22 10:26:28,569 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9200, loss[loss=0.07817, simple_loss=0.1042, pruned_loss=0.01491, audio_tagging_loss=0.01117, over 16150.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.0939, pruned_loss=0.01495, audio_tagging_loss=0.009149, over 3060978.91 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:26:32,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285750 2023-11-22 10:26:33,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1904966.6666666667, ans=0.0 2023-11-22 10:27:15,868 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.192e+01 8.757e+01 9.519e+01 1.252e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-22 10:27:27,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1905233.3333333333, ans=0.0 2023-11-22 10:27:27,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=12.0 2023-11-22 10:27:32,529 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9250, loss[loss=0.07218, simple_loss=0.09402, pruned_loss=0.01656, audio_tagging_loss=0.008612, over 15500.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09413, pruned_loss=0.01515, audio_tagging_loss=0.009181, over 3055930.56 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:27:36,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285800 2023-11-22 10:27:37,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1905300.0, ans=0.0 2023-11-22 10:27:43,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1905300.0, ans=0.125 2023-11-22 10:27:50,106 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.35 vs. limit=15.0 2023-11-22 10:27:52,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.08 vs. limit=6.0 2023-11-22 10:28:05,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1905433.3333333333, ans=0.5 2023-11-22 10:28:10,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1905500.0, ans=0.05 2023-11-22 10:28:33,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1905566.6666666667, ans=0.0 2023-11-22 10:28:37,340 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9300, loss[loss=0.08881, simple_loss=0.1033, pruned_loss=0.02665, audio_tagging_loss=0.01051, over 15186.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09416, pruned_loss=0.01516, audio_tagging_loss=0.009209, over 3050961.86 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:28:41,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285850 2023-11-22 10:28:41,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1905633.3333333333, ans=0.125 2023-11-22 10:28:55,937 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.25 vs. limit=15.0 2023-11-22 10:29:00,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1905700.0, ans=0.0 2023-11-22 10:29:01,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1905766.6666666667, ans=0.125 2023-11-22 10:29:20,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1905833.3333333333, ans=0.95 2023-11-22 10:29:25,412 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 8.386e+01 8.892e+01 9.597e+01 1.135e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 10:29:25,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1905833.3333333333, ans=0.025 2023-11-22 10:29:27,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1905833.3333333333, ans=0.0 2023-11-22 10:29:33,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1905900.0, ans=0.125 2023-11-22 10:29:34,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1905900.0, ans=10.0 2023-11-22 10:29:41,935 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9350, loss[loss=0.08285, simple_loss=0.09723, pruned_loss=0.02449, audio_tagging_loss=0.009749, over 15389.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09426, pruned_loss=0.01533, audio_tagging_loss=0.009283, over 3053249.58 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:29:45,667 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285900 2023-11-22 10:29:57,349 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0 2023-11-22 10:30:45,403 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9400, loss[loss=0.08791, simple_loss=0.1171, pruned_loss=0.0218, audio_tagging_loss=0.00758, over 15099.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09446, pruned_loss=0.01552, audio_tagging_loss=0.009379, over 3053271.71 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:30:45,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1906300.0, ans=0.0 2023-11-22 10:30:49,784 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 285950 2023-11-22 10:31:32,536 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.313e+01 8.349e+01 8.920e+01 9.542e+01 1.196e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 10:31:36,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1906566.6666666667, ans=0.0 2023-11-22 10:31:48,481 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 10:31:49,693 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9450, loss[loss=0.08197, simple_loss=0.1004, pruned_loss=0.01747, audio_tagging_loss=0.01432, over 16345.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09473, pruned_loss=0.0155, audio_tagging_loss=0.009433, over 3059753.60 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:31:54,022 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286000 2023-11-22 10:31:55,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1906633.3333333333, ans=0.1 2023-11-22 10:31:58,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1906633.3333333333, ans=0.07 2023-11-22 10:32:04,836 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.81 vs. limit=15.0 2023-11-22 10:32:31,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1906833.3333333333, ans=0.125 2023-11-22 10:32:39,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1906833.3333333333, ans=0.125 2023-11-22 10:32:49,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1906900.0, ans=0.0 2023-11-22 10:32:54,481 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9500, loss[loss=0.07756, simple_loss=0.09846, pruned_loss=0.01992, audio_tagging_loss=0.008405, over 14890.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09476, pruned_loss=0.01547, audio_tagging_loss=0.009616, over 3059625.25 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:32:58,863 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286050 2023-11-22 10:33:02,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1906966.6666666667, ans=0.125 2023-11-22 10:33:14,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1907033.3333333333, ans=0.125 2023-11-22 10:33:33,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1907166.6666666667, ans=0.125 2023-11-22 10:33:40,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1907166.6666666667, ans=0.0 2023-11-22 10:33:42,356 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.223e+01 8.226e+01 8.759e+01 9.603e+01 1.524e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-22 10:33:46,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1907233.3333333333, ans=0.125 2023-11-22 10:33:48,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1907233.3333333333, ans=0.0 2023-11-22 10:33:58,990 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9550, loss[loss=0.08185, simple_loss=0.1072, pruned_loss=0.01663, audio_tagging_loss=0.01162, over 15730.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09453, pruned_loss=0.01526, audio_tagging_loss=0.009684, over 3060850.13 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:34:02,727 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286100 2023-11-22 10:34:09,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1907300.0, ans=0.0 2023-11-22 10:34:15,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1907366.6666666667, ans=0.125 2023-11-22 10:34:19,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1907366.6666666667, ans=0.125 2023-11-22 10:35:04,211 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9600, loss[loss=0.06401, simple_loss=0.08036, pruned_loss=0.01442, audio_tagging_loss=0.009413, over 15375.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09514, pruned_loss=0.01531, audio_tagging_loss=0.009621, over 3057431.58 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:35:07,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286150 2023-11-22 10:35:13,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.06 vs. limit=22.5 2023-11-22 10:35:34,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1907766.6666666667, ans=0.125 2023-11-22 10:35:40,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1907833.3333333333, ans=0.125 2023-11-22 10:35:40,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1907833.3333333333, ans=0.0 2023-11-22 10:35:46,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1907833.3333333333, ans=0.0 2023-11-22 10:35:51,452 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.074e+01 8.745e+01 9.476e+01 1.191e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 10:36:07,695 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9650, loss[loss=0.05722, simple_loss=0.07586, pruned_loss=0.009866, audio_tagging_loss=0.009419, over 15216.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.0937, pruned_loss=0.01498, audio_tagging_loss=0.00966, over 3053848.07 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0 2023-11-22 10:36:12,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286200 2023-11-22 10:36:14,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1907966.6666666667, ans=0.125 2023-11-22 10:36:15,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1907966.6666666667, ans=0.0 2023-11-22 10:36:28,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1908033.3333333333, ans=0.0 2023-11-22 10:36:58,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1908233.3333333333, ans=0.2 2023-11-22 10:37:04,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.63 vs. limit=22.5 2023-11-22 10:37:05,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1908233.3333333333, ans=0.125 2023-11-22 10:37:09,380 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.86 vs. limit=12.0 2023-11-22 10:37:12,271 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9700, loss[loss=0.0874, simple_loss=0.1154, pruned_loss=0.02123, audio_tagging_loss=0.008481, over 15327.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09488, pruned_loss=0.01522, audio_tagging_loss=0.009397, over 3052810.85 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:37:16,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286250 2023-11-22 10:37:17,945 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.72 vs. limit=15.0 2023-11-22 10:37:35,788 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 10:37:36,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=1908433.3333333333, ans=0.02 2023-11-22 10:37:39,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1908433.3333333333, ans=0.0 2023-11-22 10:37:59,364 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.165e+01 7.968e+01 8.645e+01 9.511e+01 1.252e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-22 10:38:10,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1908566.6666666667, ans=0.125 2023-11-22 10:38:16,143 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9750, loss[loss=0.07023, simple_loss=0.09111, pruned_loss=0.01381, audio_tagging_loss=0.01087, over 14257.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09468, pruned_loss=0.01525, audio_tagging_loss=0.009302, over 3048884.12 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:38:20,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286300 2023-11-22 10:38:46,584 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.79 vs. limit=22.5 2023-11-22 10:38:55,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1908833.3333333333, ans=0.1 2023-11-22 10:39:04,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1908833.3333333333, ans=0.125 2023-11-22 10:39:20,637 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9800, loss[loss=0.06506, simple_loss=0.0755, pruned_loss=0.01594, audio_tagging_loss=0.01137, over 15753.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.095, pruned_loss=0.01535, audio_tagging_loss=0.009173, over 3051949.42 frames. ], batch size: 61, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:39:24,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286350 2023-11-22 10:39:26,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1908966.6666666667, ans=0.125 2023-11-22 10:40:08,401 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.345e+01 8.979e+01 9.748e+01 1.178e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-22 10:40:18,749 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 10:40:20,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1909233.3333333333, ans=0.04949747468305833 2023-11-22 10:40:25,480 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9850, loss[loss=0.07519, simple_loss=0.09196, pruned_loss=0.02033, audio_tagging_loss=0.008876, over 15185.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09552, pruned_loss=0.01556, audio_tagging_loss=0.009102, over 3048268.50 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:40:29,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286400 2023-11-22 10:40:42,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1909366.6666666667, ans=0.125 2023-11-22 10:40:53,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1909433.3333333333, ans=0.2 2023-11-22 10:41:00,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1909433.3333333333, ans=0.125 2023-11-22 10:41:21,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1909566.6666666667, ans=0.0 2023-11-22 10:41:28,606 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.49 vs. limit=5.0 2023-11-22 10:41:30,065 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9900, loss[loss=0.09014, simple_loss=0.1278, pruned_loss=0.01956, audio_tagging_loss=0.006659, over 15288.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09524, pruned_loss=0.01553, audio_tagging_loss=0.009126, over 3051231.36 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:41:34,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286450 2023-11-22 10:41:46,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1909700.0, ans=0.0 2023-11-22 10:41:58,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1909766.6666666667, ans=0.0 2023-11-22 10:41:59,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1909766.6666666667, ans=0.2 2023-11-22 10:42:05,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1909766.6666666667, ans=0.2 2023-11-22 10:42:05,997 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.83 vs. limit=10.0 2023-11-22 10:42:11,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1909833.3333333333, ans=0.2 2023-11-22 10:42:16,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1909833.3333333333, ans=0.125 2023-11-22 10:42:19,147 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.212e+01 8.837e+01 9.450e+01 1.215e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 10:42:34,518 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 9950, loss[loss=0.05829, simple_loss=0.07629, pruned_loss=0.01169, audio_tagging_loss=0.008453, over 15080.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09445, pruned_loss=0.01545, audio_tagging_loss=0.009147, over 3053084.91 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 10:42:35,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.97 vs. limit=22.5 2023-11-22 10:42:38,350 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286500 2023-11-22 10:42:48,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1910033.3333333333, ans=0.1 2023-11-22 10:42:57,189 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=12.0 2023-11-22 10:42:59,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1910100.0, ans=0.125 2023-11-22 10:42:59,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1910100.0, ans=0.2 2023-11-22 10:43:10,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0 2023-11-22 10:43:39,014 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10000, loss[loss=0.07411, simple_loss=0.09718, pruned_loss=0.01758, audio_tagging_loss=0.007941, over 15925.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09342, pruned_loss=0.01514, audio_tagging_loss=0.009186, over 3052927.90 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:43:42,861 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286550 2023-11-22 10:43:56,660 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=16.54 vs. limit=15.0 2023-11-22 10:44:11,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1910433.3333333333, ans=0.125 2023-11-22 10:44:23,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1910500.0, ans=0.0 2023-11-22 10:44:27,758 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.515e+01 8.076e+01 8.734e+01 9.444e+01 1.191e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 10:44:29,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1910566.6666666667, ans=0.2 2023-11-22 10:44:43,784 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10050, loss[loss=0.05104, simple_loss=0.06109, pruned_loss=0.008986, audio_tagging_loss=0.01151, over 14914.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09234, pruned_loss=0.01476, audio_tagging_loss=0.00931, over 3050220.89 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:44:43,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1910633.3333333333, ans=0.5 2023-11-22 10:44:47,536 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286600 2023-11-22 10:44:50,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1910633.3333333333, ans=0.125 2023-11-22 10:45:20,907 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.601e-03 2023-11-22 10:45:48,202 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10100, loss[loss=0.06142, simple_loss=0.07738, pruned_loss=0.01188, audio_tagging_loss=0.01085, over 15081.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09228, pruned_loss=0.01472, audio_tagging_loss=0.00937, over 3055731.20 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 10:45:52,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286650 2023-11-22 10:45:57,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.75 vs. limit=12.0 2023-11-22 10:46:09,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1911033.3333333333, ans=0.0 2023-11-22 10:46:14,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1911100.0, ans=0.125 2023-11-22 10:46:38,286 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.221e+01 8.968e+01 9.727e+01 1.162e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-22 10:46:40,874 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 10:46:43,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1911233.3333333333, ans=0.0 2023-11-22 10:46:52,490 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10150, loss[loss=0.0608, simple_loss=0.06747, pruned_loss=0.01337, audio_tagging_loss=0.0137, over 15728.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09274, pruned_loss=0.01484, audio_tagging_loss=0.009434, over 3055877.65 frames. ], batch size: 61, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 10:46:56,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286700 2023-11-22 10:46:59,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.01 vs. limit=10.0 2023-11-22 10:47:23,855 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 10:47:56,800 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10200, loss[loss=0.07621, simple_loss=0.1074, pruned_loss=0.01576, audio_tagging_loss=0.006728, over 16308.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09349, pruned_loss=0.0149, audio_tagging_loss=0.009485, over 3061303.24 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 10:48:00,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286750 2023-11-22 10:48:13,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1911700.0, ans=0.2 2023-11-22 10:48:16,746 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.17 vs. limit=15.0 2023-11-22 10:48:20,845 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 10:48:25,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1911766.6666666667, ans=0.0 2023-11-22 10:48:38,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1911833.3333333333, ans=0.0 2023-11-22 10:48:46,799 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.248e+01 8.751e+01 9.441e+01 1.217e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 10:49:01,130 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10250, loss[loss=0.08046, simple_loss=0.09936, pruned_loss=0.01736, audio_tagging_loss=0.01342, over 14387.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09406, pruned_loss=0.01504, audio_tagging_loss=0.009536, over 3052347.55 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 10:49:04,941 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286800 2023-11-22 10:49:19,805 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.71 vs. limit=15.0 2023-11-22 10:49:23,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.53 vs. limit=15.0 2023-11-22 10:49:28,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1912100.0, ans=0.125 2023-11-22 10:49:29,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1912100.0, ans=0.125 2023-11-22 10:49:41,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1912166.6666666667, ans=0.125 2023-11-22 10:49:47,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1912166.6666666667, ans=0.125 2023-11-22 10:49:59,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1912233.3333333333, ans=0.0 2023-11-22 10:50:05,639 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10300, loss[loss=0.07175, simple_loss=0.09498, pruned_loss=0.01497, audio_tagging_loss=0.00929, over 14990.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09348, pruned_loss=0.01497, audio_tagging_loss=0.009594, over 3048094.13 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 10:50:05,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1912300.0, ans=0.1 2023-11-22 10:50:09,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286850 2023-11-22 10:50:16,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1912300.0, ans=0.1 2023-11-22 10:50:17,824 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.35 vs. limit=22.5 2023-11-22 10:50:56,529 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.119e+01 8.625e+01 9.295e+01 1.228e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-22 10:50:56,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1912566.6666666667, ans=0.125 2023-11-22 10:51:02,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=15.0 2023-11-22 10:51:09,896 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10350, loss[loss=0.08076, simple_loss=0.1002, pruned_loss=0.02164, audio_tagging_loss=0.009037, over 15000.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09276, pruned_loss=0.01488, audio_tagging_loss=0.009684, over 3046296.21 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 10:51:14,292 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286900 2023-11-22 10:51:33,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1912700.0, ans=0.0 2023-11-22 10:51:33,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.77 vs. limit=15.0 2023-11-22 10:51:36,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1912766.6666666667, ans=0.125 2023-11-22 10:51:48,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1912833.3333333333, ans=0.2 2023-11-22 10:51:55,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1912833.3333333333, ans=0.125 2023-11-22 10:52:15,706 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10400, loss[loss=0.09072, simple_loss=0.1243, pruned_loss=0.02046, audio_tagging_loss=0.008115, over 15443.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.0932, pruned_loss=0.01494, audio_tagging_loss=0.009739, over 3048531.27 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:52:18,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1912966.6666666667, ans=0.0 2023-11-22 10:52:19,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 286950 2023-11-22 10:53:04,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1913166.6666666667, ans=0.0 2023-11-22 10:53:05,473 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.131e+01 8.865e+01 9.715e+01 1.299e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-22 10:53:07,506 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.25 vs. limit=15.0 2023-11-22 10:53:12,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1913233.3333333333, ans=0.07 2023-11-22 10:53:15,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1913233.3333333333, ans=0.125 2023-11-22 10:53:18,880 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10450, loss[loss=0.08032, simple_loss=0.1067, pruned_loss=0.01644, audio_tagging_loss=0.01055, over 16565.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09324, pruned_loss=0.01506, audio_tagging_loss=0.00972, over 3048310.07 frames. ], batch size: 61, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:53:23,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287000 2023-11-22 10:53:26,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.28 vs. limit=8.0 2023-11-22 10:53:49,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1913433.3333333333, ans=0.0 2023-11-22 10:53:52,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1913433.3333333333, ans=0.2 2023-11-22 10:54:22,306 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10500, loss[loss=0.06789, simple_loss=0.08422, pruned_loss=0.01759, audio_tagging_loss=0.008194, over 15227.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09321, pruned_loss=0.015, audio_tagging_loss=0.009517, over 3048171.88 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:54:26,105 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287050 2023-11-22 10:54:30,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff2.min_abs, batch_count=1913633.3333333333, ans=0.1 2023-11-22 10:54:56,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1913766.6666666667, ans=0.2 2023-11-22 10:55:03,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1913833.3333333333, ans=0.125 2023-11-22 10:55:10,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.65 vs. limit=6.0 2023-11-22 10:55:13,012 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.123e+01 8.688e+01 9.437e+01 1.274e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-22 10:55:23,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1913900.0, ans=15.0 2023-11-22 10:55:28,216 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10550, loss[loss=0.06595, simple_loss=0.09065, pruned_loss=0.01122, audio_tagging_loss=0.009402, over 14589.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09404, pruned_loss=0.01504, audio_tagging_loss=0.009283, over 3044690.22 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:55:32,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287100 2023-11-22 10:56:26,821 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.24 vs. limit=10.0 2023-11-22 10:56:27,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1914233.3333333333, ans=0.2 2023-11-22 10:56:33,340 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10600, loss[loss=0.06372, simple_loss=0.08777, pruned_loss=0.01246, audio_tagging_loss=0.007376, over 14885.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09452, pruned_loss=0.01518, audio_tagging_loss=0.009302, over 3045005.23 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:56:37,245 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287150 2023-11-22 10:56:43,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1914300.0, ans=0.125 2023-11-22 10:57:18,153 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=22.5 2023-11-22 10:57:23,282 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 8.242e+01 9.024e+01 9.598e+01 1.340e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-22 10:57:30,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1914566.6666666667, ans=0.0 2023-11-22 10:57:36,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1914633.3333333333, ans=0.125 2023-11-22 10:57:37,622 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10650, loss[loss=0.06918, simple_loss=0.08711, pruned_loss=0.01742, audio_tagging_loss=0.008207, over 15160.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09437, pruned_loss=0.01526, audio_tagging_loss=0.009231, over 3045166.87 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:57:37,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1914633.3333333333, ans=0.125 2023-11-22 10:57:41,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287200 2023-11-22 10:57:42,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1914633.3333333333, ans=0.2 2023-11-22 10:57:47,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1914633.3333333333, ans=0.0 2023-11-22 10:57:54,100 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.82 vs. limit=12.0 2023-11-22 10:58:01,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1914700.0, ans=0.1 2023-11-22 10:58:08,142 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.68 vs. limit=22.5 2023-11-22 10:58:12,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1914766.6666666667, ans=0.125 2023-11-22 10:58:24,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1914833.3333333333, ans=0.125 2023-11-22 10:58:24,493 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.73 vs. limit=22.5 2023-11-22 10:58:29,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1914900.0, ans=0.125 2023-11-22 10:58:32,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1914900.0, ans=0.0 2023-11-22 10:58:33,059 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.13 vs. limit=15.0 2023-11-22 10:58:36,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1914900.0, ans=0.125 2023-11-22 10:58:39,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1914900.0, ans=0.125 2023-11-22 10:58:42,781 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10700, loss[loss=0.0807, simple_loss=0.108, pruned_loss=0.01881, audio_tagging_loss=0.007894, over 15404.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09539, pruned_loss=0.01541, audio_tagging_loss=0.009155, over 3047203.15 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:58:47,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287250 2023-11-22 10:58:53,692 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.57 vs. limit=6.0 2023-11-22 10:59:00,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1915033.3333333333, ans=0.2 2023-11-22 10:59:08,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1915100.0, ans=0.1 2023-11-22 10:59:12,178 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.75 vs. limit=6.0 2023-11-22 10:59:20,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1915166.6666666667, ans=0.125 2023-11-22 10:59:22,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1915166.6666666667, ans=0.0 2023-11-22 10:59:31,222 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0 2023-11-22 10:59:33,546 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.175e+01 8.915e+01 9.385e+01 1.445e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-22 10:59:35,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1915233.3333333333, ans=0.0 2023-11-22 10:59:47,527 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10750, loss[loss=0.09792, simple_loss=0.1409, pruned_loss=0.02125, audio_tagging_loss=0.006226, over 15719.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09573, pruned_loss=0.01559, audio_tagging_loss=0.009138, over 3055377.12 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 10:59:47,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1915300.0, ans=0.125 2023-11-22 10:59:50,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1915300.0, ans=0.125 2023-11-22 10:59:51,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287300 2023-11-22 11:00:03,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1915366.6666666667, ans=0.125 2023-11-22 11:00:17,737 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.96 vs. limit=12.0 2023-11-22 11:00:26,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1915500.0, ans=0.125 2023-11-22 11:00:51,830 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10800, loss[loss=0.05473, simple_loss=0.07538, pruned_loss=0.006946, audio_tagging_loss=0.01009, over 13916.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09556, pruned_loss=0.01554, audio_tagging_loss=0.00905, over 3055555.58 frames. ], batch size: 53, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 11:00:55,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287350 2023-11-22 11:01:07,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1915700.0, ans=0.0 2023-11-22 11:01:08,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1915700.0, ans=0.2 2023-11-22 11:01:32,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1915833.3333333333, ans=0.1 2023-11-22 11:01:40,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1915833.3333333333, ans=0.1 2023-11-22 11:01:42,403 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.103e+01 8.562e+01 9.437e+01 1.159e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-22 11:01:47,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1915900.0, ans=0.125 2023-11-22 11:01:56,473 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10850, loss[loss=0.05401, simple_loss=0.07098, pruned_loss=0.01076, audio_tagging_loss=0.00776, over 15492.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09535, pruned_loss=0.01541, audio_tagging_loss=0.009118, over 3053548.15 frames. ], batch size: 61, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:02:00,941 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287400 2023-11-22 11:02:37,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1916166.6666666667, ans=0.125 2023-11-22 11:02:57,557 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 11:03:01,204 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10900, loss[loss=0.06359, simple_loss=0.08637, pruned_loss=0.01197, audio_tagging_loss=0.008434, over 15263.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09535, pruned_loss=0.01528, audio_tagging_loss=0.009192, over 3053369.78 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:03:04,956 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287450 2023-11-22 11:03:10,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1916300.0, ans=0.125 2023-11-22 11:03:13,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1916366.6666666667, ans=0.0 2023-11-22 11:03:16,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1916366.6666666667, ans=0.125 2023-11-22 11:03:44,939 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.37 vs. limit=22.5 2023-11-22 11:03:52,812 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.229e+01 8.892e+01 9.728e+01 1.443e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 11:03:55,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1916566.6666666667, ans=0.125 2023-11-22 11:04:06,226 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 10950, loss[loss=0.058, simple_loss=0.07995, pruned_loss=0.007253, audio_tagging_loss=0.01077, over 14666.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09487, pruned_loss=0.01519, audio_tagging_loss=0.00928, over 3048341.57 frames. ], batch size: 54, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:04:10,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287500 2023-11-22 11:04:35,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.78 vs. limit=6.0 2023-11-22 11:04:41,388 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.20 vs. limit=15.0 2023-11-22 11:04:41,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.91 vs. limit=15.0 2023-11-22 11:04:43,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1916833.3333333333, ans=0.0 2023-11-22 11:04:50,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1916833.3333333333, ans=0.2 2023-11-22 11:05:10,039 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11000, loss[loss=0.04657, simple_loss=0.05847, pruned_loss=0.006866, audio_tagging_loss=0.01047, over 14889.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09448, pruned_loss=0.01507, audio_tagging_loss=0.009299, over 3038605.68 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:05:13,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287550 2023-11-22 11:05:18,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1916966.6666666667, ans=0.0 2023-11-22 11:05:21,822 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 11:05:23,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1917033.3333333333, ans=0.125 2023-11-22 11:05:23,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1917033.3333333333, ans=0.1 2023-11-22 11:05:40,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1917100.0, ans=0.1 2023-11-22 11:06:01,247 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.398e+01 8.039e+01 8.665e+01 9.378e+01 1.568e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-22 11:06:09,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1917233.3333333333, ans=0.125 2023-11-22 11:06:09,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.79 vs. limit=15.0 2023-11-22 11:06:13,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1917300.0, ans=0.0 2023-11-22 11:06:14,094 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11050, loss[loss=0.0837, simple_loss=0.1036, pruned_loss=0.01921, audio_tagging_loss=0.0127, over 16234.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.0947, pruned_loss=0.01506, audio_tagging_loss=0.009392, over 3041609.30 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:06:17,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287600 2023-11-22 11:06:18,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1917300.0, ans=0.125 2023-11-22 11:06:25,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1917366.6666666667, ans=0.0 2023-11-22 11:06:55,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1917500.0, ans=0.125 2023-11-22 11:07:17,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1917633.3333333333, ans=0.1 2023-11-22 11:07:18,035 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11100, loss[loss=0.0724, simple_loss=0.1008, pruned_loss=0.01329, audio_tagging_loss=0.008705, over 15723.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09446, pruned_loss=0.01522, audio_tagging_loss=0.009531, over 3050576.61 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:07:22,463 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287650 2023-11-22 11:07:26,663 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.03 vs. limit=15.0 2023-11-22 11:07:35,710 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.01 vs. limit=12.0 2023-11-22 11:07:52,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1917766.6666666667, ans=0.125 2023-11-22 11:07:54,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1917766.6666666667, ans=0.125 2023-11-22 11:07:55,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.31 vs. limit=15.0 2023-11-22 11:08:09,638 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.222e+01 9.043e+01 9.663e+01 1.186e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-22 11:08:12,654 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.76 vs. limit=10.0 2023-11-22 11:08:22,598 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11150, loss[loss=0.07448, simple_loss=0.09461, pruned_loss=0.0149, audio_tagging_loss=0.01228, over 15195.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09428, pruned_loss=0.01531, audio_tagging_loss=0.009616, over 3048451.81 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:08:26,281 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287700 2023-11-22 11:08:44,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1918033.3333333333, ans=0.125 2023-11-22 11:09:05,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1918166.6666666667, ans=0.1 2023-11-22 11:09:06,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.45 vs. limit=22.5 2023-11-22 11:09:10,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1918166.6666666667, ans=0.07 2023-11-22 11:09:26,704 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11200, loss[loss=0.07134, simple_loss=0.09173, pruned_loss=0.01643, audio_tagging_loss=0.009045, over 15616.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09425, pruned_loss=0.01514, audio_tagging_loss=0.009681, over 3041072.74 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 11:09:30,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287750 2023-11-22 11:09:34,370 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:09:51,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1918433.3333333333, ans=0.125 2023-11-22 11:10:01,050 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.91 vs. limit=15.0 2023-11-22 11:10:09,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1918500.0, ans=0.125 2023-11-22 11:10:17,765 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.119e+01 8.627e+01 9.513e+01 1.631e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-22 11:10:30,733 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11250, loss[loss=0.07885, simple_loss=0.1039, pruned_loss=0.01955, audio_tagging_loss=0.007348, over 14854.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09362, pruned_loss=0.01497, audio_tagging_loss=0.009622, over 3040832.43 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 11:10:34,645 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287800 2023-11-22 11:11:04,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=15.0 2023-11-22 11:11:06,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1918766.6666666667, ans=0.2 2023-11-22 11:11:35,612 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11300, loss[loss=0.07439, simple_loss=0.09472, pruned_loss=0.01705, audio_tagging_loss=0.009973, over 14755.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09255, pruned_loss=0.01496, audio_tagging_loss=0.009563, over 3032385.40 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 11:11:39,990 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287850 2023-11-22 11:12:20,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1919166.6666666667, ans=0.2 2023-11-22 11:12:28,499 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.163e+01 8.307e+01 9.060e+01 9.651e+01 1.397e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-22 11:12:38,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.42 vs. limit=15.0 2023-11-22 11:12:39,791 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.90 vs. limit=15.0 2023-11-22 11:12:40,207 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11350, loss[loss=0.07081, simple_loss=0.09555, pruned_loss=0.01413, audio_tagging_loss=0.008907, over 14750.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09358, pruned_loss=0.01523, audio_tagging_loss=0.009373, over 3033609.17 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:12:41,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1919300.0, ans=0.1 2023-11-22 11:12:44,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287900 2023-11-22 11:12:44,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1919300.0, ans=0.2 2023-11-22 11:13:06,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1919433.3333333333, ans=0.125 2023-11-22 11:13:20,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1919500.0, ans=0.0 2023-11-22 11:13:25,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1919500.0, ans=0.125 2023-11-22 11:13:31,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=1919566.6666666667, ans=0.2 2023-11-22 11:13:41,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1919566.6666666667, ans=0.0 2023-11-22 11:13:43,682 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11400, loss[loss=0.04908, simple_loss=0.05213, pruned_loss=0.009152, audio_tagging_loss=0.01386, over 15328.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09333, pruned_loss=0.01526, audio_tagging_loss=0.009324, over 3035586.27 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:13:47,965 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 287950 2023-11-22 11:13:49,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.whiten.whitening_limit, batch_count=1919633.3333333333, ans=12.0 2023-11-22 11:14:14,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1919766.6666666667, ans=0.125 2023-11-22 11:14:17,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.22 vs. limit=15.0 2023-11-22 11:14:20,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1919766.6666666667, ans=0.1 2023-11-22 11:14:26,568 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.48 vs. limit=15.0 2023-11-22 11:14:36,225 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.311e+01 8.086e+01 8.647e+01 9.538e+01 1.200e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-22 11:14:47,053 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11450, loss[loss=0.08533, simple_loss=0.1074, pruned_loss=0.02121, audio_tagging_loss=0.01043, over 14616.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09351, pruned_loss=0.01536, audio_tagging_loss=0.009299, over 3037368.40 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:14:50,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288000 2023-11-22 11:15:28,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1920166.6666666667, ans=0.1 2023-11-22 11:15:35,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1920166.6666666667, ans=0.0 2023-11-22 11:15:36,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1920166.6666666667, ans=0.2 2023-11-22 11:15:46,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1920233.3333333333, ans=0.125 2023-11-22 11:15:53,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1920233.3333333333, ans=0.04949747468305833 2023-11-22 11:15:55,792 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11500, loss[loss=0.08063, simple_loss=0.111, pruned_loss=0.01635, audio_tagging_loss=0.008799, over 15948.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09325, pruned_loss=0.01525, audio_tagging_loss=0.009294, over 3048175.91 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:15:56,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.93 vs. limit=15.0 2023-11-22 11:15:59,519 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288050 2023-11-22 11:15:59,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1920300.0, ans=0.125 2023-11-22 11:16:03,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1920300.0, ans=0.125 2023-11-22 11:16:08,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.46 vs. limit=15.0 2023-11-22 11:16:09,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1920366.6666666667, ans=0.125 2023-11-22 11:16:11,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1920366.6666666667, ans=0.125 2023-11-22 11:16:12,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1920366.6666666667, ans=0.04949747468305833 2023-11-22 11:16:27,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1920433.3333333333, ans=0.0 2023-11-22 11:16:35,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1920500.0, ans=0.125 2023-11-22 11:16:47,947 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.254e+01 8.955e+01 9.456e+01 2.559e+02, threshold=1.791e+02, percent-clipped=1.0 2023-11-22 11:16:49,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1920566.6666666667, ans=0.125 2023-11-22 11:16:50,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1920566.6666666667, ans=0.125 2023-11-22 11:16:59,016 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11550, loss[loss=0.07871, simple_loss=0.1157, pruned_loss=0.01286, audio_tagging_loss=0.007988, over 15435.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09398, pruned_loss=0.01538, audio_tagging_loss=0.009325, over 3046865.78 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:17:02,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288100 2023-11-22 11:17:22,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1920700.0, ans=0.125 2023-11-22 11:17:24,691 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:17:37,689 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 11:17:41,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1920833.3333333333, ans=0.1 2023-11-22 11:18:01,814 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.19 vs. limit=22.5 2023-11-22 11:18:02,514 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11600, loss[loss=0.07532, simple_loss=0.1072, pruned_loss=0.01181, audio_tagging_loss=0.009904, over 16385.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09419, pruned_loss=0.01512, audio_tagging_loss=0.009355, over 3050379.89 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 32.0 2023-11-22 11:18:06,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288150 2023-11-22 11:18:09,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1920966.6666666667, ans=0.1 2023-11-22 11:18:09,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1920966.6666666667, ans=0.1 2023-11-22 11:18:22,209 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.79 vs. limit=15.0 2023-11-22 11:18:24,459 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.91 vs. limit=15.0 2023-11-22 11:18:29,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.52 vs. limit=10.0 2023-11-22 11:18:32,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1921100.0, ans=0.0 2023-11-22 11:18:39,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1921166.6666666667, ans=0.0 2023-11-22 11:18:49,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1921166.6666666667, ans=0.125 2023-11-22 11:18:56,243 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.284e+01 8.414e+01 8.937e+01 9.626e+01 1.842e+02, threshold=1.787e+02, percent-clipped=1.0 2023-11-22 11:19:07,319 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11650, loss[loss=0.06886, simple_loss=0.09372, pruned_loss=0.01327, audio_tagging_loss=0.008727, over 14757.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09461, pruned_loss=0.01519, audio_tagging_loss=0.009264, over 3045782.53 frames. ], batch size: 53, lr: 2.82e-03, grad_scale: 16.0 2023-11-22 11:19:11,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288200 2023-11-22 11:19:14,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1921300.0, ans=0.0 2023-11-22 11:19:30,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.whiten.whitening_limit, batch_count=1921366.6666666667, ans=12.0 2023-11-22 11:19:34,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1921433.3333333333, ans=0.125 2023-11-22 11:19:43,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1921433.3333333333, ans=0.125 2023-11-22 11:19:59,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1921566.6666666667, ans=0.0 2023-11-22 11:20:11,338 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11700, loss[loss=0.0656, simple_loss=0.09031, pruned_loss=0.01003, audio_tagging_loss=0.01041, over 16537.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09362, pruned_loss=0.015, audio_tagging_loss=0.009345, over 3041999.18 frames. ], batch size: 59, lr: 2.81e-03, grad_scale: 16.0 2023-11-22 11:20:15,016 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288250 2023-11-22 11:20:15,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1921633.3333333333, ans=0.125 2023-11-22 11:20:29,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1921700.0, ans=0.0 2023-11-22 11:20:30,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.35 vs. limit=15.0 2023-11-22 11:20:56,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1921833.3333333333, ans=0.125 2023-11-22 11:20:59,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1921833.3333333333, ans=0.125 2023-11-22 11:21:04,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1921900.0, ans=0.125 2023-11-22 11:21:05,065 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.380e+01 8.961e+01 9.495e+01 1.206e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-22 11:21:08,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1921900.0, ans=0.0 2023-11-22 11:21:09,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1921900.0, ans=0.1 2023-11-22 11:21:09,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1921900.0, ans=0.2 2023-11-22 11:21:15,682 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11750, loss[loss=0.08418, simple_loss=0.1113, pruned_loss=0.01906, audio_tagging_loss=0.009467, over 15889.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09415, pruned_loss=0.0151, audio_tagging_loss=0.009236, over 3047451.12 frames. ], batch size: 58, lr: 2.81e-03, grad_scale: 16.0 2023-11-22 11:21:19,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288300 2023-11-22 11:21:20,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1921966.6666666667, ans=0.0 2023-11-22 11:21:25,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1921966.6666666667, ans=0.0 2023-11-22 11:21:41,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1922100.0, ans=0.125 2023-11-22 11:21:51,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1922100.0, ans=0.2 2023-11-22 11:21:54,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1922166.6666666667, ans=0.05 2023-11-22 11:22:20,069 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11800, loss[loss=0.08018, simple_loss=0.1022, pruned_loss=0.02005, audio_tagging_loss=0.009024, over 15251.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09435, pruned_loss=0.01516, audio_tagging_loss=0.009232, over 3042470.32 frames. ], batch size: 56, lr: 2.81e-03, grad_scale: 16.0 2023-11-22 11:22:20,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1922300.0, ans=0.125 2023-11-22 11:22:24,507 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288350 2023-11-22 11:22:24,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1922300.0, ans=0.2 2023-11-22 11:22:27,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1922300.0, ans=0.125 2023-11-22 11:22:36,644 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.31 vs. limit=15.0 2023-11-22 11:22:42,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.69 vs. limit=15.0 2023-11-22 11:22:49,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1922433.3333333333, ans=0.05 2023-11-22 11:22:56,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1922433.3333333333, ans=0.0 2023-11-22 11:22:56,582 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:23:14,677 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.163e+01 8.779e+01 9.476e+01 1.167e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 11:23:25,260 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11850, loss[loss=0.06223, simple_loss=0.08281, pruned_loss=0.006773, audio_tagging_loss=0.01406, over 14841.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09486, pruned_loss=0.01514, audio_tagging_loss=0.009357, over 3047616.07 frames. ], batch size: 55, lr: 2.81e-03, grad_scale: 16.0 2023-11-22 11:23:29,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288400 2023-11-22 11:23:53,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1922766.6666666667, ans=0.125 2023-11-22 11:23:57,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1922766.6666666667, ans=0.1 2023-11-22 11:24:17,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1922900.0, ans=0.125 2023-11-22 11:24:20,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1922900.0, ans=0.025 2023-11-22 11:24:29,689 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11900, loss[loss=0.07143, simple_loss=0.09944, pruned_loss=0.01412, audio_tagging_loss=0.007597, over 15225.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09483, pruned_loss=0.0151, audio_tagging_loss=0.009465, over 3051359.30 frames. ], batch size: 57, lr: 2.81e-03, grad_scale: 16.0 2023-11-22 11:24:33,484 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288450 2023-11-22 11:24:38,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1922966.6666666667, ans=0.0 2023-11-22 11:25:23,691 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.115e+01 8.834e+01 9.649e+01 1.264e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 11:25:34,086 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 11950, loss[loss=0.08757, simple_loss=0.116, pruned_loss=0.02018, audio_tagging_loss=0.009398, over 15607.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09432, pruned_loss=0.01521, audio_tagging_loss=0.009603, over 3043563.18 frames. ], batch size: 57, lr: 2.81e-03, grad_scale: 16.0 2023-11-22 11:25:37,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288500 2023-11-22 11:25:44,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1923300.0, ans=0.125 2023-11-22 11:25:44,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1923300.0, ans=0.125 2023-11-22 11:25:45,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1923366.6666666667, ans=0.1 2023-11-22 11:26:01,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1923433.3333333333, ans=0.0 2023-11-22 11:26:03,329 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.23 vs. limit=15.0 2023-11-22 11:26:06,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1923433.3333333333, ans=0.0 2023-11-22 11:26:33,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1923566.6666666667, ans=0.0 2023-11-22 11:26:35,555 INFO [train_asr.py:1221] (3/4) Epoch 24, batch 12000, loss[loss=0.09079, simple_loss=0.1224, pruned_loss=0.02205, audio_tagging_loss=0.007559, over 16553.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09494, pruned_loss=0.01536, audio_tagging_loss=0.009608, over 3047261.10 frames. ], batch size: 59, lr: 2.81e-03, grad_scale: 32.0 2023-11-22 11:26:35,556 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 11:27:12,215 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4104, 3.6836, 2.7076, 3.6856], device='cuda:3') 2023-11-22 11:27:12,446 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.0333, 5.9156, 5.7210, 5.6590], device='cuda:3') 2023-11-22 11:27:17,208 INFO [train_asr.py:1253] (3/4) Epoch 24, validation: loss=0.05896, simple_loss=0.05166, pruned_loss=0.00516, audio_tagging_loss=0.02797, over 4681554.00 frames. 2023-11-22 11:27:17,209 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 11:27:20,734 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288550 2023-11-22 11:27:27,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1923700.0, ans=0.1 2023-11-22 11:27:43,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=1923766.6666666667, ans=0.02 2023-11-22 11:28:19,667 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 0, loss[loss=0.08067, simple_loss=0.08069, pruned_loss=0.01454, audio_tagging_loss=0.02579, over 15704.00 frames. ], tot_loss[loss=0.08067, simple_loss=0.08069, pruned_loss=0.01454, audio_tagging_loss=0.02579, over 15704.00 frames. ], batch size: 60, lr: 2.76e-03, grad_scale: 32.0 2023-11-22 11:28:19,668 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 11:28:55,801 INFO [train_asr.py:1253] (3/4) Epoch 25, validation: loss=0.05903, simple_loss=0.05164, pruned_loss=0.005146, audio_tagging_loss=0.02807, over 4681554.00 frames. 2023-11-22 11:28:55,802 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 11:29:08,310 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.39 vs. limit=15.0 2023-11-22 11:29:18,972 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.611e+01 9.501e+01 1.042e+02 1.380e+02, threshold=1.900e+02, percent-clipped=0.0 2023-11-22 11:29:19,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1923860.0, ans=0.0 2023-11-22 11:29:23,203 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=15.0 2023-11-22 11:29:29,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1923926.6666666667, ans=0.0 2023-11-22 11:29:34,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288600 2023-11-22 11:29:43,379 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.47 vs. limit=15.0 2023-11-22 11:30:00,632 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 50, loss[loss=0.07828, simple_loss=0.09188, pruned_loss=0.01378, audio_tagging_loss=0.01856, over 15164.00 frames. ], tot_loss[loss=0.08167, simple_loss=0.09503, pruned_loss=0.01589, audio_tagging_loss=0.01826, over 687896.88 frames. ], batch size: 57, lr: 2.76e-03, grad_scale: 32.0 2023-11-22 11:30:04,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.44 vs. limit=6.0 2023-11-22 11:30:07,434 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.86 vs. limit=15.0 2023-11-22 11:30:31,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1924260.0, ans=0.125 2023-11-22 11:30:38,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288650 2023-11-22 11:30:38,502 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:30:54,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1924393.3333333333, ans=0.0 2023-11-22 11:31:04,554 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 100, loss[loss=0.09308, simple_loss=0.1122, pruned_loss=0.02338, audio_tagging_loss=0.0136, over 15602.00 frames. ], tot_loss[loss=0.08015, simple_loss=0.09388, pruned_loss=0.01575, audio_tagging_loss=0.01746, over 1210480.28 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:31:23,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1924526.6666666667, ans=0.07 2023-11-22 11:31:28,373 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.104e+01 8.752e+01 9.299e+01 1.007e+02 1.330e+02, threshold=1.860e+02, percent-clipped=0.0 2023-11-22 11:31:42,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288700 2023-11-22 11:31:46,734 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.90 vs. limit=12.0 2023-11-22 11:32:10,067 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 150, loss[loss=0.1008, simple_loss=0.1385, pruned_loss=0.01949, audio_tagging_loss=0.01206, over 15071.00 frames. ], tot_loss[loss=0.07881, simple_loss=0.0956, pruned_loss=0.01549, audio_tagging_loss=0.01552, over 1619615.84 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:32:15,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1924793.3333333333, ans=0.95 2023-11-22 11:32:47,844 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288750 2023-11-22 11:33:14,440 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 200, loss[loss=0.06974, simple_loss=0.09353, pruned_loss=0.01616, audio_tagging_loss=0.006814, over 14679.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09673, pruned_loss=0.01559, audio_tagging_loss=0.01343, over 1939636.25 frames. ], batch size: 55, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:33:25,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1925126.6666666667, ans=0.025 2023-11-22 11:33:37,541 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.935e+01 8.185e+01 8.728e+01 9.456e+01 1.257e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-22 11:33:37,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1925193.3333333333, ans=0.0 2023-11-22 11:33:39,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1925260.0, ans=0.2 2023-11-22 11:33:40,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1925260.0, ans=0.09899494936611666 2023-11-22 11:33:41,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1925260.0, ans=0.125 2023-11-22 11:33:50,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1925260.0, ans=0.125 2023-11-22 11:33:51,607 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288800 2023-11-22 11:33:56,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1925326.6666666667, ans=0.125 2023-11-22 11:34:05,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.15 vs. limit=10.0 2023-11-22 11:34:13,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1925393.3333333333, ans=0.0 2023-11-22 11:34:13,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1925393.3333333333, ans=0.0 2023-11-22 11:34:18,345 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 250, loss[loss=0.07689, simple_loss=0.1047, pruned_loss=0.01661, audio_tagging_loss=0.007915, over 15336.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.0971, pruned_loss=0.01555, audio_tagging_loss=0.01216, over 2188065.72 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:34:18,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1925460.0, ans=0.0 2023-11-22 11:34:44,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1925593.3333333333, ans=0.0 2023-11-22 11:34:54,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1925593.3333333333, ans=0.125 2023-11-22 11:34:55,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288850 2023-11-22 11:35:02,764 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.32 vs. limit=15.0 2023-11-22 11:35:22,893 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 300, loss[loss=0.06661, simple_loss=0.07997, pruned_loss=0.01626, audio_tagging_loss=0.01037, over 14398.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09729, pruned_loss=0.01568, audio_tagging_loss=0.01126, over 2381070.44 frames. ], batch size: 53, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:35:28,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1925793.3333333333, ans=0.0 2023-11-22 11:35:36,066 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:35:44,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1925860.0, ans=0.0 2023-11-22 11:35:45,376 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.207e+01 8.755e+01 9.551e+01 1.256e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-22 11:35:51,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1925926.6666666667, ans=0.1 2023-11-22 11:35:59,590 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288900 2023-11-22 11:36:06,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1925993.3333333333, ans=0.1 2023-11-22 11:36:14,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1926060.0, ans=10.0 2023-11-22 11:36:15,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1926060.0, ans=0.125 2023-11-22 11:36:27,374 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 350, loss[loss=0.05918, simple_loss=0.07996, pruned_loss=0.007944, audio_tagging_loss=0.01126, over 15158.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09708, pruned_loss=0.01553, audio_tagging_loss=0.01069, over 2529807.49 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:36:31,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1926126.6666666667, ans=0.1 2023-11-22 11:36:45,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1926193.3333333333, ans=0.04949747468305833 2023-11-22 11:36:50,561 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.26 vs. limit=22.5 2023-11-22 11:36:58,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1926260.0, ans=0.1 2023-11-22 11:37:05,082 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 288950 2023-11-22 11:37:08,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1926326.6666666667, ans=0.07 2023-11-22 11:37:20,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1926393.3333333333, ans=0.125 2023-11-22 11:37:32,235 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 400, loss[loss=0.07353, simple_loss=0.09711, pruned_loss=0.01468, audio_tagging_loss=0.0103, over 16727.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09619, pruned_loss=0.01537, audio_tagging_loss=0.01037, over 2646212.86 frames. ], batch size: 62, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:37:48,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.whiten.whitening_limit, batch_count=1926526.6666666667, ans=12.0 2023-11-22 11:37:49,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1926526.6666666667, ans=0.2 2023-11-22 11:37:55,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.456e+01 8.000e+01 8.654e+01 9.629e+01 1.287e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-22 11:37:59,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1926593.3333333333, ans=0.0 2023-11-22 11:38:02,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1926593.3333333333, ans=0.1 2023-11-22 11:38:03,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1926593.3333333333, ans=0.125 2023-11-22 11:38:03,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1926593.3333333333, ans=0.0 2023-11-22 11:38:09,846 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289000 2023-11-22 11:38:10,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.22 vs. limit=22.5 2023-11-22 11:38:37,046 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 450, loss[loss=0.07822, simple_loss=0.1024, pruned_loss=0.01643, audio_tagging_loss=0.01058, over 14507.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09555, pruned_loss=0.01534, audio_tagging_loss=0.01018, over 2732299.67 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:38:41,961 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.07 vs. limit=15.0 2023-11-22 11:39:03,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1926926.6666666667, ans=0.015 2023-11-22 11:39:08,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1926926.6666666667, ans=0.125 2023-11-22 11:39:14,787 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289050 2023-11-22 11:39:24,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1926993.3333333333, ans=0.1 2023-11-22 11:39:42,508 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 500, loss[loss=0.08429, simple_loss=0.1091, pruned_loss=0.02376, audio_tagging_loss=0.005956, over 15165.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09456, pruned_loss=0.01542, audio_tagging_loss=0.009998, over 2797256.78 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:39:56,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1927193.3333333333, ans=0.2 2023-11-22 11:40:02,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1927193.3333333333, ans=0.125 2023-11-22 11:40:05,712 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.152e+01 8.293e+01 8.940e+01 9.677e+01 1.305e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-22 11:40:13,371 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.61 vs. limit=15.0 2023-11-22 11:40:15,608 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=12.0 2023-11-22 11:40:19,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289100 2023-11-22 11:40:33,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.78 vs. limit=15.0 2023-11-22 11:40:42,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.36 vs. limit=15.0 2023-11-22 11:40:44,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1927393.3333333333, ans=0.125 2023-11-22 11:40:47,558 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 550, loss[loss=0.07841, simple_loss=0.1038, pruned_loss=0.0175, audio_tagging_loss=0.009016, over 15279.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09434, pruned_loss=0.01536, audio_tagging_loss=0.009844, over 2851936.20 frames. ], batch size: 55, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:41:24,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289150 2023-11-22 11:41:24,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1927660.0, ans=0.125 2023-11-22 11:41:51,756 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 600, loss[loss=0.07721, simple_loss=0.1058, pruned_loss=0.01698, audio_tagging_loss=0.007312, over 16177.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09401, pruned_loss=0.01513, audio_tagging_loss=0.009776, over 2900627.86 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:42:01,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1927793.3333333333, ans=0.0 2023-11-22 11:42:05,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1927860.0, ans=0.125 2023-11-22 11:42:16,625 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.617e+01 7.918e+01 8.734e+01 9.306e+01 1.583e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 11:42:26,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1927926.6666666667, ans=0.025 2023-11-22 11:42:29,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289200 2023-11-22 11:42:56,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1928126.6666666667, ans=0.125 2023-11-22 11:42:57,766 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 650, loss[loss=0.07568, simple_loss=0.09847, pruned_loss=0.0174, audio_tagging_loss=0.009035, over 16006.00 frames. ], tot_loss[loss=0.07167, simple_loss=0.09379, pruned_loss=0.01506, audio_tagging_loss=0.009715, over 2928183.09 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:43:11,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1928193.3333333333, ans=0.0 2023-11-22 11:43:19,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1928193.3333333333, ans=0.125 2023-11-22 11:43:34,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289250 2023-11-22 11:43:43,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1928326.6666666667, ans=0.125 2023-11-22 11:43:44,075 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.40 vs. limit=15.0 2023-11-22 11:43:50,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1928393.3333333333, ans=0.125 2023-11-22 11:44:01,492 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 700, loss[loss=0.09508, simple_loss=0.1192, pruned_loss=0.02554, audio_tagging_loss=0.009937, over 14859.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09393, pruned_loss=0.01503, audio_tagging_loss=0.009748, over 2961923.49 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:44:26,163 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.213e+01 8.759e+01 9.279e+01 1.127e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-22 11:44:39,757 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289300 2023-11-22 11:44:42,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1928660.0, ans=0.125 2023-11-22 11:44:54,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1928726.6666666667, ans=0.0 2023-11-22 11:45:05,545 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 750, loss[loss=0.07268, simple_loss=0.09404, pruned_loss=0.01806, audio_tagging_loss=0.007605, over 15210.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09407, pruned_loss=0.01505, audio_tagging_loss=0.009706, over 2985425.85 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:45:05,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1928793.3333333333, ans=0.125 2023-11-22 11:45:07,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1928793.3333333333, ans=0.0 2023-11-22 11:45:11,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1928793.3333333333, ans=0.0 2023-11-22 11:45:25,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1928860.0, ans=0.1 2023-11-22 11:45:43,744 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289350 2023-11-22 11:45:43,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1928993.3333333333, ans=0.0 2023-11-22 11:45:48,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1928993.3333333333, ans=0.125 2023-11-22 11:46:10,234 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 800, loss[loss=0.04701, simple_loss=0.05877, pruned_loss=0.008352, audio_tagging_loss=0.009278, over 15119.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09348, pruned_loss=0.01511, audio_tagging_loss=0.009745, over 2991984.97 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:46:20,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1929126.6666666667, ans=0.125 2023-11-22 11:46:20,914 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.68 vs. limit=15.0 2023-11-22 11:46:34,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1929193.3333333333, ans=0.1 2023-11-22 11:46:36,818 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.118e+01 8.155e+01 8.807e+01 9.499e+01 1.198e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-22 11:46:39,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1929260.0, ans=0.125 2023-11-22 11:46:48,647 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289400 2023-11-22 11:46:55,692 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.04 vs. limit=15.0 2023-11-22 11:46:59,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1929326.6666666667, ans=0.125 2023-11-22 11:46:59,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1929326.6666666667, ans=0.125 2023-11-22 11:47:16,288 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 850, loss[loss=0.07037, simple_loss=0.09069, pruned_loss=0.01494, audio_tagging_loss=0.01009, over 16257.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09364, pruned_loss=0.01518, audio_tagging_loss=0.009721, over 3000364.97 frames. ], batch size: 60, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:47:30,778 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:47:32,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1929526.6666666667, ans=0.1 2023-11-22 11:47:34,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1929526.6666666667, ans=0.125 2023-11-22 11:47:41,831 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.15 vs. limit=12.0 2023-11-22 11:47:45,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1929593.3333333333, ans=0.0 2023-11-22 11:47:54,050 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289450 2023-11-22 11:47:56,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.29 vs. limit=15.0 2023-11-22 11:48:01,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1929660.0, ans=0.0 2023-11-22 11:48:06,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1929660.0, ans=0.0 2023-11-22 11:48:15,272 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:48:21,304 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 900, loss[loss=0.05495, simple_loss=0.07321, pruned_loss=0.00882, audio_tagging_loss=0.009529, over 15433.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09397, pruned_loss=0.01518, audio_tagging_loss=0.009782, over 3018606.58 frames. ], batch size: 60, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:48:34,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1929860.0, ans=0.125 2023-11-22 11:48:39,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1929860.0, ans=0.2 2023-11-22 11:48:41,770 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.03 vs. limit=15.0 2023-11-22 11:48:46,853 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.150e+01 8.654e+01 9.309e+01 1.377e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-22 11:48:51,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1929926.6666666667, ans=0.125 2023-11-22 11:48:58,865 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289500 2023-11-22 11:49:07,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1929993.3333333333, ans=0.125 2023-11-22 11:49:26,014 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 950, loss[loss=0.07194, simple_loss=0.1008, pruned_loss=0.01379, audio_tagging_loss=0.007772, over 15557.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.0943, pruned_loss=0.01531, audio_tagging_loss=0.009655, over 3028585.76 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:49:31,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1930126.6666666667, ans=0.1 2023-11-22 11:49:43,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1930193.3333333333, ans=0.125 2023-11-22 11:49:51,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1930260.0, ans=0.125 2023-11-22 11:50:03,012 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.86 vs. limit=15.0 2023-11-22 11:50:04,413 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289550 2023-11-22 11:50:05,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1930326.6666666667, ans=0.0 2023-11-22 11:50:07,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1930326.6666666667, ans=0.2 2023-11-22 11:50:11,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1930326.6666666667, ans=0.0 2023-11-22 11:50:21,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1930393.3333333333, ans=0.0 2023-11-22 11:50:26,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1930393.3333333333, ans=0.2 2023-11-22 11:50:30,823 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1000, loss[loss=0.06409, simple_loss=0.0767, pruned_loss=0.01493, audio_tagging_loss=0.01082, over 16712.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.09379, pruned_loss=0.0151, audio_tagging_loss=0.009486, over 3028423.36 frames. ], batch size: 63, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:50:38,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.86 vs. limit=12.0 2023-11-22 11:50:39,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1930460.0, ans=0.125 2023-11-22 11:50:43,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1930526.6666666667, ans=0.1 2023-11-22 11:50:57,562 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.182e+01 8.192e+01 8.953e+01 9.947e+01 1.294e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-22 11:50:58,856 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 11:51:02,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1930593.3333333333, ans=0.125 2023-11-22 11:51:07,552 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:51:08,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289600 2023-11-22 11:51:14,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1930660.0, ans=0.2 2023-11-22 11:51:16,421 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.04 vs. limit=22.5 2023-11-22 11:51:20,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1930660.0, ans=0.125 2023-11-22 11:51:36,491 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1050, loss[loss=0.07659, simple_loss=0.09523, pruned_loss=0.01986, audio_tagging_loss=0.009122, over 15079.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09381, pruned_loss=0.01514, audio_tagging_loss=0.009288, over 3035606.91 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:51:48,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1930860.0, ans=0.09899494936611666 2023-11-22 11:52:00,215 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0 2023-11-22 11:52:13,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289650 2023-11-22 11:52:21,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1930993.3333333333, ans=0.2 2023-11-22 11:52:40,619 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1100, loss[loss=0.04931, simple_loss=0.0506, pruned_loss=0.01184, audio_tagging_loss=0.01217, over 14884.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09326, pruned_loss=0.01489, audio_tagging_loss=0.009223, over 3041958.16 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:52:41,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.73 vs. limit=15.0 2023-11-22 11:52:44,283 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 11:53:05,836 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=15.0 2023-11-22 11:53:06,984 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.014e+01 8.517e+01 9.464e+01 1.257e+02, threshold=1.703e+02, percent-clipped=0.0 2023-11-22 11:53:14,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1931260.0, ans=0.0 2023-11-22 11:53:17,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1931260.0, ans=0.1 2023-11-22 11:53:18,202 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289700 2023-11-22 11:53:44,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.59 vs. limit=22.5 2023-11-22 11:53:45,307 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1150, loss[loss=0.0487, simple_loss=0.05834, pruned_loss=0.007408, audio_tagging_loss=0.01213, over 14774.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09295, pruned_loss=0.01479, audio_tagging_loss=0.009171, over 3038223.87 frames. ], batch size: 60, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:53:58,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1931526.6666666667, ans=0.0 2023-11-22 11:54:04,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1931526.6666666667, ans=0.125 2023-11-22 11:54:08,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1931526.6666666667, ans=0.1 2023-11-22 11:54:19,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1931593.3333333333, ans=0.0 2023-11-22 11:54:22,866 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289750 2023-11-22 11:54:27,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1931660.0, ans=0.0 2023-11-22 11:54:30,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1931660.0, ans=0.125 2023-11-22 11:54:45,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1931726.6666666667, ans=0.0 2023-11-22 11:54:47,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1931726.6666666667, ans=0.125 2023-11-22 11:54:50,872 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1200, loss[loss=0.06982, simple_loss=0.09327, pruned_loss=0.01376, audio_tagging_loss=0.009428, over 16382.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09364, pruned_loss=0.01495, audio_tagging_loss=0.009126, over 3037667.29 frames. ], batch size: 61, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:54:51,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1931793.3333333333, ans=0.0 2023-11-22 11:55:05,797 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.98 vs. limit=15.0 2023-11-22 11:55:14,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1931860.0, ans=0.2 2023-11-22 11:55:16,756 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.526e+01 8.025e+01 8.580e+01 9.215e+01 1.708e+02, threshold=1.716e+02, percent-clipped=1.0 2023-11-22 11:55:17,095 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:55:19,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1931926.6666666667, ans=0.125 2023-11-22 11:55:23,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1931926.6666666667, ans=0.04949747468305833 2023-11-22 11:55:28,149 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289800 2023-11-22 11:55:31,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1931993.3333333333, ans=0.2 2023-11-22 11:55:38,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1931993.3333333333, ans=0.0 2023-11-22 11:55:38,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1931993.3333333333, ans=0.04949747468305833 2023-11-22 11:55:46,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1932060.0, ans=0.2 2023-11-22 11:55:56,224 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1250, loss[loss=0.05002, simple_loss=0.06883, pruned_loss=0.008755, audio_tagging_loss=0.006853, over 15477.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09448, pruned_loss=0.01498, audio_tagging_loss=0.009125, over 3043956.72 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:55:56,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1932126.6666666667, ans=0.125 2023-11-22 11:56:01,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1932126.6666666667, ans=0.125 2023-11-22 11:56:13,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.47 vs. limit=15.0 2023-11-22 11:56:33,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289850 2023-11-22 11:56:36,959 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.38 vs. limit=15.0 2023-11-22 11:56:49,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1932393.3333333333, ans=0.125 2023-11-22 11:56:53,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1932393.3333333333, ans=0.0 2023-11-22 11:57:00,754 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1300, loss[loss=0.07807, simple_loss=0.1052, pruned_loss=0.01738, audio_tagging_loss=0.008066, over 15887.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.0938, pruned_loss=0.01489, audio_tagging_loss=0.009101, over 3038259.40 frames. ], batch size: 60, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:57:08,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1932460.0, ans=0.125 2023-11-22 11:57:16,981 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.41 vs. limit=22.5 2023-11-22 11:57:26,642 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.940e+01 8.056e+01 8.885e+01 9.522e+01 1.296e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 11:57:27,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.87 vs. limit=12.0 2023-11-22 11:57:38,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289900 2023-11-22 11:58:01,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1932726.6666666667, ans=0.125 2023-11-22 11:58:05,424 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1350, loss[loss=0.06747, simple_loss=0.09478, pruned_loss=0.01226, audio_tagging_loss=0.007816, over 15815.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09373, pruned_loss=0.01495, audio_tagging_loss=0.009174, over 3036777.82 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 11:58:06,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1932793.3333333333, ans=0.0 2023-11-22 11:58:30,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1932926.6666666667, ans=0.125 2023-11-22 11:58:41,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1932926.6666666667, ans=0.09899494936611666 2023-11-22 11:58:42,940 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 289950 2023-11-22 11:58:49,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1932993.3333333333, ans=0.0 2023-11-22 11:58:52,193 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 11:59:08,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1933060.0, ans=0.0 2023-11-22 11:59:09,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1933126.6666666667, ans=0.125 2023-11-22 11:59:10,476 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1400, loss[loss=0.04534, simple_loss=0.05011, pruned_loss=0.009567, audio_tagging_loss=0.01072, over 15889.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09336, pruned_loss=0.01493, audio_tagging_loss=0.009216, over 3042124.91 frames. ], batch size: 61, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 11:59:20,796 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.33 vs. limit=10.0 2023-11-22 11:59:33,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1933193.3333333333, ans=0.125 2023-11-22 11:59:36,916 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.595e+01 8.260e+01 8.672e+01 9.990e+01 1.427e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-22 11:59:37,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.68 vs. limit=10.0 2023-11-22 11:59:46,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1933260.0, ans=0.2 2023-11-22 11:59:47,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290000 2023-11-22 11:59:47,834 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 11:59:48,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.60 vs. limit=15.0 2023-11-22 12:00:15,348 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1450, loss[loss=0.05822, simple_loss=0.06975, pruned_loss=0.0109, audio_tagging_loss=0.01244, over 16382.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09305, pruned_loss=0.01485, audio_tagging_loss=0.009339, over 3047856.15 frames. ], batch size: 65, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:00:32,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1933526.6666666667, ans=0.1 2023-11-22 12:00:36,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1933526.6666666667, ans=0.125 2023-11-22 12:00:53,414 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290050 2023-11-22 12:01:03,488 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.09 vs. limit=15.0 2023-11-22 12:01:20,170 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1500, loss[loss=0.07153, simple_loss=0.09082, pruned_loss=0.01411, audio_tagging_loss=0.01201, over 14777.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09439, pruned_loss=0.01522, audio_tagging_loss=0.00935, over 3045400.14 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:01:37,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1933860.0, ans=0.2 2023-11-22 12:01:38,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1933860.0, ans=0.1 2023-11-22 12:01:46,716 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.126e+01 8.732e+01 9.513e+01 1.269e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-22 12:01:48,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1933926.6666666667, ans=0.1 2023-11-22 12:01:51,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1933926.6666666667, ans=0.125 2023-11-22 12:01:57,404 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290100 2023-11-22 12:01:57,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1933993.3333333333, ans=0.125 2023-11-22 12:01:59,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0 2023-11-22 12:02:01,777 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.52 vs. limit=22.5 2023-11-22 12:02:24,723 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1550, loss[loss=0.07123, simple_loss=0.08686, pruned_loss=0.01801, audio_tagging_loss=0.009792, over 14893.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09485, pruned_loss=0.01547, audio_tagging_loss=0.009402, over 3047022.04 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:02:27,224 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.81 vs. limit=22.5 2023-11-22 12:02:35,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1934126.6666666667, ans=0.1 2023-11-22 12:02:43,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1934193.3333333333, ans=0.125 2023-11-22 12:02:52,963 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.14 vs. limit=22.5 2023-11-22 12:02:53,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1934260.0, ans=0.0 2023-11-22 12:03:02,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290150 2023-11-22 12:03:19,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1934393.3333333333, ans=0.125 2023-11-22 12:03:20,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1934393.3333333333, ans=0.125 2023-11-22 12:03:30,913 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1600, loss[loss=0.07094, simple_loss=0.09476, pruned_loss=0.01448, audio_tagging_loss=0.009076, over 14911.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09433, pruned_loss=0.01543, audio_tagging_loss=0.009417, over 3045170.38 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 32.0 2023-11-22 12:03:38,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1934460.0, ans=0.07 2023-11-22 12:03:49,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1934526.6666666667, ans=0.2 2023-11-22 12:03:58,924 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.200e+01 8.897e+01 9.742e+01 1.180e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-22 12:04:01,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1934593.3333333333, ans=0.2 2023-11-22 12:04:08,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290200 2023-11-22 12:04:34,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1934793.3333333333, ans=0.1 2023-11-22 12:04:35,562 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1650, loss[loss=0.06343, simple_loss=0.06738, pruned_loss=0.01666, audio_tagging_loss=0.01308, over 14288.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09389, pruned_loss=0.01518, audio_tagging_loss=0.009559, over 3042313.40 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:04:35,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=1934793.3333333333, ans=0.025 2023-11-22 12:04:40,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1934793.3333333333, ans=0.1 2023-11-22 12:04:51,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1934860.0, ans=0.0 2023-11-22 12:04:52,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1934860.0, ans=0.125 2023-11-22 12:05:13,978 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290250 2023-11-22 12:05:14,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1934993.3333333333, ans=0.125 2023-11-22 12:05:21,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1934993.3333333333, ans=0.0 2023-11-22 12:05:21,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1934993.3333333333, ans=0.125 2023-11-22 12:05:24,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1934993.3333333333, ans=0.125 2023-11-22 12:05:28,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1935060.0, ans=0.125 2023-11-22 12:05:40,422 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1700, loss[loss=0.06465, simple_loss=0.08412, pruned_loss=0.01178, audio_tagging_loss=0.01082, over 13671.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09341, pruned_loss=0.01489, audio_tagging_loss=0.009605, over 3045510.53 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 8.0 2023-11-22 12:05:45,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1935126.6666666667, ans=0.125 2023-11-22 12:05:55,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1935193.3333333333, ans=0.025 2023-11-22 12:06:04,752 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 12:06:06,421 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.58 vs. limit=15.0 2023-11-22 12:06:09,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1935260.0, ans=0.2 2023-11-22 12:06:10,637 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.333e+01 9.072e+01 9.807e+01 1.367e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-22 12:06:18,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290300 2023-11-22 12:06:18,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1935326.6666666667, ans=0.0 2023-11-22 12:06:25,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1935326.6666666667, ans=0.2 2023-11-22 12:06:42,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1935393.3333333333, ans=0.1 2023-11-22 12:06:45,797 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1750, loss[loss=0.07665, simple_loss=0.1025, pruned_loss=0.0172, audio_tagging_loss=0.008213, over 14927.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09323, pruned_loss=0.01485, audio_tagging_loss=0.009501, over 3045590.84 frames. ], batch size: 55, lr: 2.75e-03, grad_scale: 8.0 2023-11-22 12:06:54,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1935460.0, ans=0.125 2023-11-22 12:07:09,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1935526.6666666667, ans=0.125 2023-11-22 12:07:14,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1935593.3333333333, ans=0.0 2023-11-22 12:07:23,620 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290350 2023-11-22 12:07:31,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1935660.0, ans=0.1 2023-11-22 12:07:43,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.78 vs. limit=15.0 2023-11-22 12:07:45,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0 2023-11-22 12:07:50,640 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1800, loss[loss=0.08612, simple_loss=0.1027, pruned_loss=0.0231, audio_tagging_loss=0.01169, over 14229.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09353, pruned_loss=0.01478, audio_tagging_loss=0.009326, over 3050216.24 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 8.0 2023-11-22 12:08:12,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1935860.0, ans=0.025 2023-11-22 12:08:19,855 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.690e+01 8.156e+01 8.799e+01 9.735e+01 1.149e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 12:08:21,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5 2023-11-22 12:08:28,524 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290400 2023-11-22 12:08:55,126 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1850, loss[loss=0.09365, simple_loss=0.1274, pruned_loss=0.02345, audio_tagging_loss=0.006529, over 14901.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09346, pruned_loss=0.01474, audio_tagging_loss=0.009291, over 3042745.99 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 8.0 2023-11-22 12:09:16,071 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.41 vs. limit=15.0 2023-11-22 12:09:33,468 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290450 2023-11-22 12:09:34,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1936326.6666666667, ans=0.125 2023-11-22 12:09:42,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1936326.6666666667, ans=0.125 2023-11-22 12:09:59,883 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1900, loss[loss=0.07702, simple_loss=0.102, pruned_loss=0.01744, audio_tagging_loss=0.008598, over 16004.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09333, pruned_loss=0.01462, audio_tagging_loss=0.009273, over 3035041.11 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 8.0 2023-11-22 12:10:08,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1936460.0, ans=0.125 2023-11-22 12:10:16,123 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.78 vs. limit=15.0 2023-11-22 12:10:23,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1936526.6666666667, ans=0.0 2023-11-22 12:10:26,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1936593.3333333333, ans=0.125 2023-11-22 12:10:29,143 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.157e+01 8.748e+01 9.500e+01 1.354e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 12:10:37,254 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290500 2023-11-22 12:10:38,753 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 12:10:45,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.86 vs. limit=6.0 2023-11-22 12:11:04,309 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 1950, loss[loss=0.06896, simple_loss=0.0919, pruned_loss=0.0157, audio_tagging_loss=0.007316, over 16616.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09264, pruned_loss=0.01452, audio_tagging_loss=0.009259, over 3035558.48 frames. ], batch size: 64, lr: 2.75e-03, grad_scale: 8.0 2023-11-22 12:11:23,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1936860.0, ans=0.2 2023-11-22 12:11:31,411 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=15.0 2023-11-22 12:11:41,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290550 2023-11-22 12:11:42,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1936993.3333333333, ans=0.125 2023-11-22 12:12:08,545 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2000, loss[loss=0.08783, simple_loss=0.1146, pruned_loss=0.0229, audio_tagging_loss=0.007645, over 16258.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09246, pruned_loss=0.01476, audio_tagging_loss=0.009381, over 3033604.44 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:12:12,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.29 vs. limit=15.0 2023-11-22 12:12:15,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1937126.6666666667, ans=0.5 2023-11-22 12:12:35,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1937260.0, ans=0.2 2023-11-22 12:12:38,528 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.037e+01 8.579e+01 9.384e+01 1.253e+02, threshold=1.716e+02, percent-clipped=0.0 2023-11-22 12:12:45,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290600 2023-11-22 12:12:47,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1937326.6666666667, ans=0.1 2023-11-22 12:12:53,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1937326.6666666667, ans=0.125 2023-11-22 12:12:58,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1937326.6666666667, ans=0.1 2023-11-22 12:13:04,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1937393.3333333333, ans=0.125 2023-11-22 12:13:13,350 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2050, loss[loss=0.06717, simple_loss=0.07997, pruned_loss=0.01652, audio_tagging_loss=0.01066, over 14363.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09271, pruned_loss=0.01483, audio_tagging_loss=0.009401, over 3038274.59 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:13:31,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1937526.6666666667, ans=0.0 2023-11-22 12:13:48,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1937593.3333333333, ans=0.1 2023-11-22 12:13:50,724 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290650 2023-11-22 12:13:52,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1937660.0, ans=0.125 2023-11-22 12:14:18,248 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2100, loss[loss=0.06747, simple_loss=0.09736, pruned_loss=0.01069, audio_tagging_loss=0.0081, over 15319.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09265, pruned_loss=0.01474, audio_tagging_loss=0.009265, over 3038312.02 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:14:41,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1937860.0, ans=0.125 2023-11-22 12:14:46,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1937926.6666666667, ans=0.125 2023-11-22 12:14:47,390 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.119e+01 8.164e+01 8.808e+01 9.351e+01 1.143e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 12:14:51,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1937926.6666666667, ans=0.1 2023-11-22 12:14:54,721 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290700 2023-11-22 12:15:02,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1937993.3333333333, ans=0.1 2023-11-22 12:15:16,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1938060.0, ans=0.125 2023-11-22 12:15:22,346 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2150, loss[loss=0.07868, simple_loss=0.1084, pruned_loss=0.01827, audio_tagging_loss=0.006204, over 15089.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.0936, pruned_loss=0.0147, audio_tagging_loss=0.009219, over 3039065.70 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0 2023-11-22 12:15:23,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1938126.6666666667, ans=0.1 2023-11-22 12:15:25,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1938126.6666666667, ans=0.125 2023-11-22 12:15:47,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1938260.0, ans=0.0 2023-11-22 12:15:59,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290750 2023-11-22 12:16:01,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.55 vs. limit=22.5 2023-11-22 12:16:01,975 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 12:16:05,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1938326.6666666667, ans=0.125 2023-11-22 12:16:15,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1938393.3333333333, ans=0.125 2023-11-22 12:16:15,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1938393.3333333333, ans=0.0 2023-11-22 12:16:26,553 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2200, loss[loss=0.08971, simple_loss=0.1207, pruned_loss=0.01896, audio_tagging_loss=0.01041, over 16391.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09364, pruned_loss=0.01487, audio_tagging_loss=0.009256, over 3044115.56 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:16:26,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1938460.0, ans=0.125 2023-11-22 12:16:30,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1938460.0, ans=0.0 2023-11-22 12:16:38,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1938526.6666666667, ans=0.09899494936611666 2023-11-22 12:16:42,663 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0 2023-11-22 12:16:56,427 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.385e+01 8.783e+01 9.663e+01 1.140e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 12:17:03,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290800 2023-11-22 12:17:26,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.07 vs. limit=10.0 2023-11-22 12:17:30,928 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2250, loss[loss=0.08476, simple_loss=0.1126, pruned_loss=0.02092, audio_tagging_loss=0.007549, over 14673.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09291, pruned_loss=0.0147, audio_tagging_loss=0.009265, over 3039113.94 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:18:08,459 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290850 2023-11-22 12:18:25,609 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.78 vs. limit=12.0 2023-11-22 12:18:32,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1939060.0, ans=0.0 2023-11-22 12:18:32,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1939060.0, ans=0.07 2023-11-22 12:18:36,068 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2300, loss[loss=0.08196, simple_loss=0.1073, pruned_loss=0.02058, audio_tagging_loss=0.007723, over 14443.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09223, pruned_loss=0.01465, audio_tagging_loss=0.00937, over 3036621.47 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:18:41,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1939126.6666666667, ans=0.1 2023-11-22 12:18:48,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1939193.3333333333, ans=0.09899494936611666 2023-11-22 12:19:03,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1939260.0, ans=0.125 2023-11-22 12:19:06,744 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.734e+01 8.285e+01 8.843e+01 9.474e+01 1.631e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 12:19:06,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1939260.0, ans=0.0 2023-11-22 12:19:08,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=1939260.0, ans=0.1 2023-11-22 12:19:11,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1939260.0, ans=0.0 2023-11-22 12:19:12,975 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290900 2023-11-22 12:19:14,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1939326.6666666667, ans=0.125 2023-11-22 12:19:16,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1939326.6666666667, ans=0.0 2023-11-22 12:19:31,992 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 12:19:37,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1939393.3333333333, ans=0.125 2023-11-22 12:19:39,818 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2350, loss[loss=0.07442, simple_loss=0.09497, pruned_loss=0.01627, audio_tagging_loss=0.01066, over 15371.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09351, pruned_loss=0.01483, audio_tagging_loss=0.009345, over 3038007.82 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:20:06,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1939593.3333333333, ans=0.125 2023-11-22 12:20:16,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1939593.3333333333, ans=0.07 2023-11-22 12:20:17,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 290950 2023-11-22 12:20:27,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1939660.0, ans=0.125 2023-11-22 12:20:32,509 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.63 vs. limit=10.0 2023-11-22 12:20:38,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1939726.6666666667, ans=0.0 2023-11-22 12:20:44,659 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2400, loss[loss=0.0442, simple_loss=0.05195, pruned_loss=0.007289, audio_tagging_loss=0.01093, over 15978.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.0943, pruned_loss=0.01491, audio_tagging_loss=0.009338, over 3042923.71 frames. ], batch size: 62, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:20:46,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1939793.3333333333, ans=0.125 2023-11-22 12:21:06,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1939860.0, ans=0.125 2023-11-22 12:21:15,530 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.176e+01 8.394e+01 8.930e+01 9.736e+01 4.109e+02, threshold=1.786e+02, percent-clipped=1.0 2023-11-22 12:21:21,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291000 2023-11-22 12:21:33,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1939993.3333333333, ans=0.09899494936611666 2023-11-22 12:21:46,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1940060.0, ans=0.0 2023-11-22 12:21:50,309 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2450, loss[loss=0.09147, simple_loss=0.1228, pruned_loss=0.02077, audio_tagging_loss=0.009315, over 15371.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09374, pruned_loss=0.01489, audio_tagging_loss=0.009439, over 3030325.75 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:21:56,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1940126.6666666667, ans=0.2 2023-11-22 12:22:06,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1940193.3333333333, ans=0.125 2023-11-22 12:22:12,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1940193.3333333333, ans=0.125 2023-11-22 12:22:15,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1940260.0, ans=0.125 2023-11-22 12:22:28,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291050 2023-11-22 12:22:56,584 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2500, loss[loss=0.0684, simple_loss=0.09629, pruned_loss=0.01285, audio_tagging_loss=0.007405, over 14893.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09336, pruned_loss=0.0148, audio_tagging_loss=0.009438, over 3033222.76 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:22:57,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.92 vs. limit=15.0 2023-11-22 12:23:26,947 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.685e+01 8.038e+01 8.739e+01 9.351e+01 1.232e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 12:23:27,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1940593.3333333333, ans=0.0 2023-11-22 12:23:31,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1940593.3333333333, ans=0.125 2023-11-22 12:23:33,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291100 2023-11-22 12:23:42,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1940660.0, ans=0.125 2023-11-22 12:23:55,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1940726.6666666667, ans=0.0 2023-11-22 12:24:01,990 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2550, loss[loss=0.07244, simple_loss=0.1009, pruned_loss=0.01331, audio_tagging_loss=0.00868, over 14386.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.0932, pruned_loss=0.01483, audio_tagging_loss=0.009405, over 3026169.71 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:24:10,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1940793.3333333333, ans=0.0 2023-11-22 12:24:25,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1940860.0, ans=0.0 2023-11-22 12:24:37,117 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.90 vs. limit=12.0 2023-11-22 12:24:40,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291150 2023-11-22 12:25:01,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.06 vs. limit=15.0 2023-11-22 12:25:05,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1941126.6666666667, ans=0.125 2023-11-22 12:25:06,889 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2600, loss[loss=0.04315, simple_loss=0.05005, pruned_loss=0.006048, audio_tagging_loss=0.01208, over 14773.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09269, pruned_loss=0.01485, audio_tagging_loss=0.009386, over 3028196.20 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:25:20,678 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.48 vs. limit=12.0 2023-11-22 12:25:31,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.44 vs. limit=12.0 2023-11-22 12:25:33,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1941260.0, ans=0.2 2023-11-22 12:25:39,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.159e+01 9.048e+01 9.727e+01 1.323e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-22 12:25:40,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1941260.0, ans=0.125 2023-11-22 12:25:46,468 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291200 2023-11-22 12:25:54,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1941326.6666666667, ans=0.125 2023-11-22 12:26:14,632 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2650, loss[loss=0.07525, simple_loss=0.1034, pruned_loss=0.0121, audio_tagging_loss=0.01144, over 13883.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09293, pruned_loss=0.01479, audio_tagging_loss=0.009402, over 3028734.17 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:26:29,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1941526.6666666667, ans=0.0 2023-11-22 12:26:40,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1941593.3333333333, ans=0.125 2023-11-22 12:26:42,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1941593.3333333333, ans=0.0 2023-11-22 12:26:48,078 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.98 vs. limit=6.0 2023-11-22 12:26:53,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291250 2023-11-22 12:26:57,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1941660.0, ans=0.125 2023-11-22 12:27:00,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1941660.0, ans=0.1 2023-11-22 12:27:15,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1941726.6666666667, ans=0.1 2023-11-22 12:27:20,077 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2700, loss[loss=0.05685, simple_loss=0.07843, pruned_loss=0.011, audio_tagging_loss=0.006634, over 16683.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09326, pruned_loss=0.01465, audio_tagging_loss=0.009296, over 3035833.58 frames. ], batch size: 61, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:27:21,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.44 vs. limit=10.0 2023-11-22 12:27:36,160 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 12:27:50,981 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.955e+01 8.601e+01 9.459e+01 1.243e+02, threshold=1.720e+02, percent-clipped=0.0 2023-11-22 12:27:58,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291300 2023-11-22 12:28:01,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0 2023-11-22 12:28:11,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.61 vs. limit=15.0 2023-11-22 12:28:20,722 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.14 vs. limit=15.0 2023-11-22 12:28:25,087 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2750, loss[loss=0.04329, simple_loss=0.05465, pruned_loss=0.006049, audio_tagging_loss=0.00992, over 14983.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.0921, pruned_loss=0.01446, audio_tagging_loss=0.009357, over 3030352.85 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:28:25,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1942126.6666666667, ans=0.125 2023-11-22 12:28:34,099 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.51 vs. limit=15.0 2023-11-22 12:28:54,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1942260.0, ans=0.125 2023-11-22 12:29:03,592 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291350 2023-11-22 12:29:09,112 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.99 vs. limit=10.0 2023-11-22 12:29:21,457 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 12:29:26,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1942393.3333333333, ans=0.125 2023-11-22 12:29:30,167 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2800, loss[loss=0.06715, simple_loss=0.08762, pruned_loss=0.01354, audio_tagging_loss=0.009798, over 14619.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.0916, pruned_loss=0.01462, audio_tagging_loss=0.009371, over 3040019.03 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:29:32,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1942460.0, ans=0.125 2023-11-22 12:29:47,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1942526.6666666667, ans=0.2 2023-11-22 12:29:54,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1942526.6666666667, ans=0.125 2023-11-22 12:29:57,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1942593.3333333333, ans=0.125 2023-11-22 12:30:01,634 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.537e+01 8.249e+01 8.938e+01 9.573e+01 1.382e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-22 12:30:07,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291400 2023-11-22 12:30:32,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1942726.6666666667, ans=0.0 2023-11-22 12:30:36,166 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2850, loss[loss=0.07539, simple_loss=0.09676, pruned_loss=0.01581, audio_tagging_loss=0.01121, over 15382.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09255, pruned_loss=0.01478, audio_tagging_loss=0.009314, over 3042033.73 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:31:08,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1942926.6666666667, ans=0.125 2023-11-22 12:31:08,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.06 vs. limit=10.0 2023-11-22 12:31:13,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291450 2023-11-22 12:31:17,858 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.79 vs. limit=15.0 2023-11-22 12:31:22,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1942993.3333333333, ans=0.0 2023-11-22 12:31:36,025 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 12:31:40,799 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2900, loss[loss=0.07338, simple_loss=0.09343, pruned_loss=0.01723, audio_tagging_loss=0.009437, over 15231.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09311, pruned_loss=0.01511, audio_tagging_loss=0.009251, over 3040063.26 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:31:42,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1943126.6666666667, ans=0.125 2023-11-22 12:32:01,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1943193.3333333333, ans=0.0 2023-11-22 12:32:01,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1943193.3333333333, ans=10.0 2023-11-22 12:32:11,960 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.290e+01 8.892e+01 9.564e+01 1.220e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 12:32:13,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1943260.0, ans=0.125 2023-11-22 12:32:15,203 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.39 vs. limit=6.0 2023-11-22 12:32:19,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291500 2023-11-22 12:32:35,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1943393.3333333333, ans=0.0 2023-11-22 12:32:45,625 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 2950, loss[loss=0.05838, simple_loss=0.07586, pruned_loss=0.01031, audio_tagging_loss=0.01014, over 15449.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09274, pruned_loss=0.01495, audio_tagging_loss=0.009331, over 3046618.97 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:33:04,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1943526.6666666667, ans=0.1 2023-11-22 12:33:07,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1943526.6666666667, ans=0.07 2023-11-22 12:33:11,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1943593.3333333333, ans=0.125 2023-11-22 12:33:17,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1943593.3333333333, ans=0.125 2023-11-22 12:33:19,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1943593.3333333333, ans=0.125 2023-11-22 12:33:23,477 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291550 2023-11-22 12:33:26,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1943660.0, ans=0.125 2023-11-22 12:33:27,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1943660.0, ans=0.0 2023-11-22 12:33:47,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1943726.6666666667, ans=0.2 2023-11-22 12:33:50,807 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3000, loss[loss=0.07783, simple_loss=0.1012, pruned_loss=0.02004, audio_tagging_loss=0.007219, over 15518.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09454, pruned_loss=0.01511, audio_tagging_loss=0.009197, over 3051263.14 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:33:50,808 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 12:34:15,510 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9703, 3.1771, 2.9708, 3.1276, 3.4661, 2.8760, 3.4339, 2.5445], device='cuda:3') 2023-11-22 12:34:30,473 INFO [train_asr.py:1253] (3/4) Epoch 25, validation: loss=0.05876, simple_loss=0.05157, pruned_loss=0.005103, audio_tagging_loss=0.02788, over 4681554.00 frames. 2023-11-22 12:34:30,474 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 12:34:32,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1943793.3333333333, ans=0.125 2023-11-22 12:34:34,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1943793.3333333333, ans=0.125 2023-11-22 12:34:41,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1943793.3333333333, ans=0.125 2023-11-22 12:34:57,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1943926.6666666667, ans=0.95 2023-11-22 12:35:01,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.344e+01 8.945e+01 9.581e+01 1.218e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-22 12:35:08,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291600 2023-11-22 12:35:15,836 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.35 vs. limit=15.0 2023-11-22 12:35:28,737 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.71 vs. limit=6.0 2023-11-22 12:35:30,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1944060.0, ans=0.125 2023-11-22 12:35:35,435 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3050, loss[loss=0.09119, simple_loss=0.1199, pruned_loss=0.0205, audio_tagging_loss=0.01071, over 15081.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09483, pruned_loss=0.01548, audio_tagging_loss=0.00929, over 3046381.45 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:35:49,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1944193.3333333333, ans=0.125 2023-11-22 12:35:52,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1944193.3333333333, ans=0.125 2023-11-22 12:36:12,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1944260.0, ans=0.125 2023-11-22 12:36:13,318 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 12:36:13,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291650 2023-11-22 12:36:13,963 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.84 vs. limit=15.0 2023-11-22 12:36:29,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1944393.3333333333, ans=0.2 2023-11-22 12:36:30,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1944393.3333333333, ans=0.0 2023-11-22 12:36:39,988 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3100, loss[loss=0.07571, simple_loss=0.09707, pruned_loss=0.01703, audio_tagging_loss=0.01015, over 15457.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09566, pruned_loss=0.01561, audio_tagging_loss=0.009328, over 3045863.89 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:36:48,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1944460.0, ans=0.09899494936611666 2023-11-22 12:36:51,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1944460.0, ans=0.125 2023-11-22 12:36:54,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1944526.6666666667, ans=0.2 2023-11-22 12:37:00,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1944526.6666666667, ans=0.5 2023-11-22 12:37:10,654 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.355e+01 8.957e+01 9.791e+01 1.144e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-22 12:37:17,524 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291700 2023-11-22 12:37:29,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.85 vs. limit=12.0 2023-11-22 12:37:44,673 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3150, loss[loss=0.05312, simple_loss=0.05779, pruned_loss=0.00817, audio_tagging_loss=0.01605, over 16623.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09493, pruned_loss=0.0153, audio_tagging_loss=0.009455, over 3049625.26 frames. ], batch size: 65, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:37:46,583 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.20 vs. limit=15.0 2023-11-22 12:38:21,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291750 2023-11-22 12:38:24,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1944993.3333333333, ans=0.125 2023-11-22 12:38:25,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.37 vs. limit=22.5 2023-11-22 12:38:48,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1945126.6666666667, ans=0.125 2023-11-22 12:38:49,295 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3200, loss[loss=0.07238, simple_loss=0.1024, pruned_loss=0.01383, audio_tagging_loss=0.007378, over 15610.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09479, pruned_loss=0.01534, audio_tagging_loss=0.00948, over 3048763.86 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:38:54,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1945126.6666666667, ans=0.125 2023-11-22 12:39:15,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1945260.0, ans=0.1 2023-11-22 12:39:20,219 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.187e+01 8.718e+01 9.345e+01 1.176e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-22 12:39:27,159 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291800 2023-11-22 12:39:34,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1945326.6666666667, ans=0.0 2023-11-22 12:39:46,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=1945393.3333333333, ans=0.05 2023-11-22 12:39:53,932 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3250, loss[loss=0.05573, simple_loss=0.06707, pruned_loss=0.006963, audio_tagging_loss=0.01523, over 13870.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09485, pruned_loss=0.01534, audio_tagging_loss=0.009621, over 3054475.33 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:39:58,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1945460.0, ans=0.125 2023-11-22 12:40:23,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.64 vs. limit=10.0 2023-11-22 12:40:23,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1945593.3333333333, ans=0.0 2023-11-22 12:40:24,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.82 vs. limit=15.0 2023-11-22 12:40:30,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1945593.3333333333, ans=0.125 2023-11-22 12:40:31,219 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291850 2023-11-22 12:40:36,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1945660.0, ans=0.0 2023-11-22 12:40:44,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.58 vs. limit=15.0 2023-11-22 12:40:56,135 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 12:40:58,266 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3300, loss[loss=0.0613, simple_loss=0.07346, pruned_loss=0.01294, audio_tagging_loss=0.01163, over 15295.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.0947, pruned_loss=0.01519, audio_tagging_loss=0.009721, over 3053891.64 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:41:29,369 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.525e+01 8.169e+01 8.738e+01 9.487e+01 1.174e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 12:41:35,675 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291900 2023-11-22 12:41:39,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1945993.3333333333, ans=0.125 2023-11-22 12:41:48,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1946060.0, ans=0.125 2023-11-22 12:42:03,233 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3350, loss[loss=0.06888, simple_loss=0.09396, pruned_loss=0.01146, audio_tagging_loss=0.01044, over 15328.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09434, pruned_loss=0.0151, audio_tagging_loss=0.009666, over 3052733.80 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:42:18,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.21 vs. limit=15.0 2023-11-22 12:42:40,631 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 291950 2023-11-22 12:42:44,679 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 12:42:48,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1946326.6666666667, ans=0.125 2023-11-22 12:42:49,852 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.20 vs. limit=15.0 2023-11-22 12:42:55,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1946393.3333333333, ans=0.125 2023-11-22 12:43:03,693 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0 2023-11-22 12:43:08,073 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3400, loss[loss=0.06984, simple_loss=0.09883, pruned_loss=0.01223, audio_tagging_loss=0.008189, over 15581.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09387, pruned_loss=0.015, audio_tagging_loss=0.009496, over 3050704.54 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 32.0 2023-11-22 12:43:13,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1946460.0, ans=0.0 2023-11-22 12:43:23,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1946526.6666666667, ans=0.125 2023-11-22 12:43:35,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.05 vs. limit=15.0 2023-11-22 12:43:38,751 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.227e+01 8.817e+01 9.429e+01 1.318e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-22 12:43:45,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292000 2023-11-22 12:44:11,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1946726.6666666667, ans=0.1 2023-11-22 12:44:15,283 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3450, loss[loss=0.07543, simple_loss=0.09784, pruned_loss=0.01711, audio_tagging_loss=0.009409, over 15930.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09382, pruned_loss=0.01507, audio_tagging_loss=0.00941, over 3048410.12 frames. ], batch size: 61, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:44:30,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1946860.0, ans=0.2 2023-11-22 12:44:32,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1946860.0, ans=0.125 2023-11-22 12:44:51,947 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292050 2023-11-22 12:45:01,367 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 12:45:02,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1946993.3333333333, ans=0.125 2023-11-22 12:45:16,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.08 vs. limit=22.5 2023-11-22 12:45:19,189 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3500, loss[loss=0.08225, simple_loss=0.1083, pruned_loss=0.02013, audio_tagging_loss=0.007986, over 14667.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.0947, pruned_loss=0.01517, audio_tagging_loss=0.009261, over 3047514.87 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:45:24,688 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.72 vs. limit=15.0 2023-11-22 12:45:27,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1947126.6666666667, ans=0.1 2023-11-22 12:45:32,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1947193.3333333333, ans=0.0 2023-11-22 12:45:39,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=12.0 2023-11-22 12:45:51,047 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 12:45:52,127 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 7.990e+01 8.840e+01 9.495e+01 1.155e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-22 12:45:55,977 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292100 2023-11-22 12:46:05,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1947326.6666666667, ans=0.125 2023-11-22 12:46:23,363 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3550, loss[loss=0.05411, simple_loss=0.06935, pruned_loss=0.01062, audio_tagging_loss=0.008816, over 15145.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09348, pruned_loss=0.01506, audio_tagging_loss=0.009204, over 3049791.24 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:46:24,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1947460.0, ans=0.2 2023-11-22 12:46:27,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1947460.0, ans=0.0 2023-11-22 12:46:50,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1947593.3333333333, ans=0.125 2023-11-22 12:46:52,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1947593.3333333333, ans=0.0 2023-11-22 12:47:00,196 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292150 2023-11-22 12:47:04,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1947660.0, ans=0.0 2023-11-22 12:47:18,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1947726.6666666667, ans=0.0 2023-11-22 12:47:26,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1947793.3333333333, ans=0.1 2023-11-22 12:47:27,290 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3600, loss[loss=0.07188, simple_loss=0.1003, pruned_loss=0.01384, audio_tagging_loss=0.007909, over 15005.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09349, pruned_loss=0.01495, audio_tagging_loss=0.009205, over 3056225.06 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:47:29,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.11 vs. limit=15.0 2023-11-22 12:47:46,622 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=15.0 2023-11-22 12:47:54,645 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0 2023-11-22 12:48:00,886 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 8.103e+01 8.938e+01 1.008e+02 1.423e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-22 12:48:04,701 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292200 2023-11-22 12:48:14,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1947993.3333333333, ans=0.125 2023-11-22 12:48:20,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.44 vs. limit=15.0 2023-11-22 12:48:31,933 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3650, loss[loss=0.06724, simple_loss=0.08428, pruned_loss=0.01396, audio_tagging_loss=0.01114, over 16449.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.0939, pruned_loss=0.01501, audio_tagging_loss=0.009173, over 3056875.48 frames. ], batch size: 61, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:48:47,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1948193.3333333333, ans=0.125 2023-11-22 12:49:03,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1948260.0, ans=0.0 2023-11-22 12:49:09,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292250 2023-11-22 12:49:15,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1948326.6666666667, ans=0.125 2023-11-22 12:49:30,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1948393.3333333333, ans=0.2 2023-11-22 12:49:37,043 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3700, loss[loss=0.07195, simple_loss=0.0948, pruned_loss=0.01564, audio_tagging_loss=0.008913, over 15303.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09434, pruned_loss=0.01515, audio_tagging_loss=0.009194, over 3061244.12 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:49:38,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1948460.0, ans=0.125 2023-11-22 12:50:10,188 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.131e+01 8.890e+01 9.560e+01 1.123e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 12:50:13,990 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292300 2023-11-22 12:50:16,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1948660.0, ans=0.125 2023-11-22 12:50:23,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1948660.0, ans=0.0 2023-11-22 12:50:41,530 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3750, loss[loss=0.07757, simple_loss=0.1157, pruned_loss=0.01297, audio_tagging_loss=0.006727, over 16524.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09469, pruned_loss=0.01517, audio_tagging_loss=0.009103, over 3059167.04 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:51:03,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1948860.0, ans=0.0 2023-11-22 12:51:09,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1948926.6666666667, ans=0.125 2023-11-22 12:51:19,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292350 2023-11-22 12:51:25,172 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 12:51:29,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1948993.3333333333, ans=0.125 2023-11-22 12:51:37,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1949060.0, ans=0.1 2023-11-22 12:51:39,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1949060.0, ans=0.125 2023-11-22 12:51:45,488 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3800, loss[loss=0.05739, simple_loss=0.06332, pruned_loss=0.01317, audio_tagging_loss=0.01256, over 14814.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09305, pruned_loss=0.01483, audio_tagging_loss=0.00924, over 3050079.38 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:52:01,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1949193.3333333333, ans=0.125 2023-11-22 12:52:16,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1949260.0, ans=0.1 2023-11-22 12:52:21,359 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.073e+01 8.669e+01 9.455e+01 1.608e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-22 12:52:23,918 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292400 2023-11-22 12:52:50,537 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3850, loss[loss=0.06992, simple_loss=0.08698, pruned_loss=0.01631, audio_tagging_loss=0.01011, over 16657.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09379, pruned_loss=0.01495, audio_tagging_loss=0.009303, over 3053003.69 frames. ], batch size: 64, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:53:23,236 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=22.5 2023-11-22 12:53:28,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292450 2023-11-22 12:53:55,291 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3900, loss[loss=0.07369, simple_loss=0.09044, pruned_loss=0.01814, audio_tagging_loss=0.01033, over 14623.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09336, pruned_loss=0.01483, audio_tagging_loss=0.009345, over 3040497.89 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:54:30,348 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.059e+01 8.183e+01 8.833e+01 9.463e+01 1.290e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 12:54:33,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292500 2023-11-22 12:54:46,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1950060.0, ans=10.0 2023-11-22 12:54:49,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1950060.0, ans=0.2 2023-11-22 12:55:00,197 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 3950, loss[loss=0.0818, simple_loss=0.1117, pruned_loss=0.01778, audio_tagging_loss=0.008146, over 15739.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09303, pruned_loss=0.01481, audio_tagging_loss=0.009505, over 3042448.29 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 8.0 2023-11-22 12:55:14,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1950193.3333333333, ans=0.125 2023-11-22 12:55:19,041 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0 2023-11-22 12:55:28,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1950260.0, ans=0.5 2023-11-22 12:55:37,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1950260.0, ans=0.09899494936611666 2023-11-22 12:55:38,532 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292550 2023-11-22 12:55:43,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1950326.6666666667, ans=0.2 2023-11-22 12:56:04,980 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4000, loss[loss=0.06337, simple_loss=0.08786, pruned_loss=0.01278, audio_tagging_loss=0.006664, over 14574.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09401, pruned_loss=0.01519, audio_tagging_loss=0.009514, over 3039668.17 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:56:05,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.37 vs. limit=6.0 2023-11-22 12:56:28,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1950526.6666666667, ans=0.0 2023-11-22 12:56:40,626 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.359e+01 8.523e+01 9.261e+01 9.950e+01 1.651e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-22 12:56:43,817 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292600 2023-11-22 12:56:46,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1950660.0, ans=0.0 2023-11-22 12:56:53,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1950660.0, ans=0.0 2023-11-22 12:57:11,686 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4050, loss[loss=0.07706, simple_loss=0.09236, pruned_loss=0.02066, audio_tagging_loss=0.01022, over 14508.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09427, pruned_loss=0.01514, audio_tagging_loss=0.009508, over 3044785.35 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:57:14,242 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 12:57:26,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1950860.0, ans=0.125 2023-11-22 12:57:32,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1950860.0, ans=0.2 2023-11-22 12:57:49,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292650 2023-11-22 12:57:52,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.55 vs. limit=15.0 2023-11-22 12:57:53,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1950993.3333333333, ans=0.0 2023-11-22 12:58:05,015 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.60 vs. limit=15.0 2023-11-22 12:58:08,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1951060.0, ans=0.0 2023-11-22 12:58:08,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1951060.0, ans=0.0 2023-11-22 12:58:17,016 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4100, loss[loss=0.06774, simple_loss=0.08479, pruned_loss=0.01575, audio_tagging_loss=0.00959, over 12912.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09407, pruned_loss=0.01514, audio_tagging_loss=0.009574, over 3033531.27 frames. ], batch size: 50, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:58:18,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1951126.6666666667, ans=0.2 2023-11-22 12:58:31,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1951193.3333333333, ans=0.2 2023-11-22 12:58:40,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1951193.3333333333, ans=0.0 2023-11-22 12:58:48,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1951260.0, ans=0.125 2023-11-22 12:58:52,138 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.488e+01 8.924e+01 9.565e+01 1.327e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-22 12:58:54,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292700 2023-11-22 12:59:04,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=15.0 2023-11-22 12:59:22,193 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4150, loss[loss=0.07271, simple_loss=0.08959, pruned_loss=0.01573, audio_tagging_loss=0.01218, over 15058.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09479, pruned_loss=0.01527, audio_tagging_loss=0.009372, over 3030482.46 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 12:59:35,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1951526.6666666667, ans=0.125 2023-11-22 12:59:35,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1951526.6666666667, ans=0.0 2023-11-22 12:59:50,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.79 vs. limit=15.0 2023-11-22 13:00:00,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292750 2023-11-22 13:00:04,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1951660.0, ans=10.0 2023-11-22 13:00:09,154 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 13:00:14,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1951726.6666666667, ans=0.125 2023-11-22 13:00:26,787 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4200, loss[loss=0.06345, simple_loss=0.09159, pruned_loss=0.009315, audio_tagging_loss=0.008336, over 14528.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09512, pruned_loss=0.01538, audio_tagging_loss=0.009324, over 3032302.18 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 13:00:31,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1951793.3333333333, ans=0.125 2023-11-22 13:00:54,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1951926.6666666667, ans=0.05 2023-11-22 13:01:00,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1951926.6666666667, ans=10.0 2023-11-22 13:01:02,470 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.208e+01 8.744e+01 9.952e+01 1.323e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 13:01:05,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292800 2023-11-22 13:01:33,223 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4250, loss[loss=0.06512, simple_loss=0.08589, pruned_loss=0.01389, audio_tagging_loss=0.008285, over 15333.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09491, pruned_loss=0.01536, audio_tagging_loss=0.009292, over 3041510.03 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 13:01:57,322 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 13:02:06,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1952260.0, ans=0.125 2023-11-22 13:02:09,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292850 2023-11-22 13:02:23,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1952393.3333333333, ans=0.125 2023-11-22 13:02:26,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1952393.3333333333, ans=0.125 2023-11-22 13:02:34,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.21 vs. limit=22.5 2023-11-22 13:02:37,485 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4300, loss[loss=0.06409, simple_loss=0.08204, pruned_loss=0.01004, audio_tagging_loss=0.01303, over 15699.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09605, pruned_loss=0.01569, audio_tagging_loss=0.009174, over 3042196.06 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 16.0 2023-11-22 13:03:12,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.605e+01 8.363e+01 9.136e+01 9.974e+01 1.277e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-22 13:03:14,726 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292900 2023-11-22 13:03:26,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1952660.0, ans=0.2 2023-11-22 13:03:37,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1952726.6666666667, ans=0.125 2023-11-22 13:03:41,701 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4350, loss[loss=0.06756, simple_loss=0.09484, pruned_loss=0.01176, audio_tagging_loss=0.008383, over 14844.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09626, pruned_loss=0.01552, audio_tagging_loss=0.009114, over 3041479.54 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:03:43,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.49 vs. limit=15.0 2023-11-22 13:03:56,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1952860.0, ans=0.1 2023-11-22 13:03:59,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1952860.0, ans=0.125 2023-11-22 13:04:19,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 292950 2023-11-22 13:04:27,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1952993.3333333333, ans=0.0 2023-11-22 13:04:41,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1953060.0, ans=0.09899494936611666 2023-11-22 13:04:46,743 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4400, loss[loss=0.08485, simple_loss=0.1188, pruned_loss=0.01683, audio_tagging_loss=0.008628, over 15041.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.0965, pruned_loss=0.01552, audio_tagging_loss=0.009116, over 3046142.35 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:05:03,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1953193.3333333333, ans=0.125 2023-11-22 13:05:07,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1953193.3333333333, ans=0.2 2023-11-22 13:05:21,495 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.093e+01 8.881e+01 9.679e+01 1.352e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-22 13:05:24,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293000 2023-11-22 13:05:24,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1953326.6666666667, ans=0.125 2023-11-22 13:05:25,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1953326.6666666667, ans=0.0 2023-11-22 13:05:40,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1953393.3333333333, ans=0.125 2023-11-22 13:05:40,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.76 vs. limit=10.0 2023-11-22 13:05:50,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1953460.0, ans=0.07 2023-11-22 13:05:51,671 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4450, loss[loss=0.05767, simple_loss=0.076, pruned_loss=0.008293, audio_tagging_loss=0.01138, over 15811.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.0959, pruned_loss=0.01535, audio_tagging_loss=0.00911, over 3048859.88 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:05:53,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1953460.0, ans=0.125 2023-11-22 13:06:10,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1953526.6666666667, ans=0.125 2023-11-22 13:06:14,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1953526.6666666667, ans=0.2 2023-11-22 13:06:22,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.12 vs. limit=12.0 2023-11-22 13:06:24,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1953593.3333333333, ans=0.2 2023-11-22 13:06:28,185 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293050 2023-11-22 13:06:29,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1953660.0, ans=0.2 2023-11-22 13:06:34,106 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.88 vs. limit=6.0 2023-11-22 13:06:45,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1953726.6666666667, ans=0.1 2023-11-22 13:06:54,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1953793.3333333333, ans=0.0 2023-11-22 13:06:54,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.22 vs. limit=15.0 2023-11-22 13:06:55,427 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4500, loss[loss=0.07099, simple_loss=0.1121, pruned_loss=0.009891, audio_tagging_loss=0.005029, over 14903.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09629, pruned_loss=0.01529, audio_tagging_loss=0.009026, over 3040871.20 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:07:04,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1953793.3333333333, ans=0.1 2023-11-22 13:07:09,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1953860.0, ans=0.09899494936611666 2023-11-22 13:07:12,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1953860.0, ans=0.0 2023-11-22 13:07:31,539 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.144e+01 8.843e+01 9.621e+01 1.771e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 13:07:32,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293100 2023-11-22 13:07:35,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1953993.3333333333, ans=10.0 2023-11-22 13:07:58,964 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4550, loss[loss=0.06224, simple_loss=0.08354, pruned_loss=0.01139, audio_tagging_loss=0.009076, over 13755.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.0953, pruned_loss=0.0152, audio_tagging_loss=0.009033, over 3038562.36 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:08:04,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1954126.6666666667, ans=0.0 2023-11-22 13:08:11,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1954193.3333333333, ans=0.2 2023-11-22 13:08:24,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1954260.0, ans=0.125 2023-11-22 13:08:32,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1954260.0, ans=0.0 2023-11-22 13:08:37,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293150 2023-11-22 13:08:38,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1954326.6666666667, ans=0.1 2023-11-22 13:08:48,014 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 13:09:03,946 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4600, loss[loss=0.09013, simple_loss=0.1319, pruned_loss=0.01953, audio_tagging_loss=0.004637, over 16006.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09357, pruned_loss=0.01495, audio_tagging_loss=0.009223, over 3039551.85 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:09:06,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1954460.0, ans=0.1 2023-11-22 13:09:21,414 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 13:09:40,088 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 8.168e+01 8.767e+01 9.419e+01 1.196e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-22 13:09:41,520 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293200 2023-11-22 13:10:07,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.45 vs. limit=15.0 2023-11-22 13:10:09,634 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4650, loss[loss=0.0565, simple_loss=0.07537, pruned_loss=0.00906, audio_tagging_loss=0.009759, over 14946.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09331, pruned_loss=0.01487, audio_tagging_loss=0.009431, over 3043182.09 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:10:23,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1954860.0, ans=0.04949747468305833 2023-11-22 13:10:33,111 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.33 vs. limit=12.0 2023-11-22 13:10:39,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1954926.6666666667, ans=0.0 2023-11-22 13:10:42,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1954926.6666666667, ans=0.2 2023-11-22 13:10:46,878 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293250 2023-11-22 13:11:13,271 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4700, loss[loss=0.04158, simple_loss=0.04247, pruned_loss=0.009475, audio_tagging_loss=0.01087, over 14345.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09313, pruned_loss=0.01489, audio_tagging_loss=0.009456, over 3042634.09 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:11:40,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1955260.0, ans=0.125 2023-11-22 13:11:49,575 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.061e+01 8.708e+01 9.431e+01 1.267e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-22 13:11:50,922 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293300 2023-11-22 13:11:51,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1955326.6666666667, ans=0.125 2023-11-22 13:12:04,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1955393.3333333333, ans=0.1 2023-11-22 13:12:10,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1955393.3333333333, ans=0.0 2023-11-22 13:12:15,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1955393.3333333333, ans=0.1 2023-11-22 13:12:17,201 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4750, loss[loss=0.0698, simple_loss=0.08688, pruned_loss=0.01378, audio_tagging_loss=0.01258, over 15517.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09295, pruned_loss=0.01493, audio_tagging_loss=0.009587, over 3038035.72 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:12:23,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1955460.0, ans=0.0 2023-11-22 13:12:24,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1955460.0, ans=0.0 2023-11-22 13:12:55,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293350 2023-11-22 13:12:58,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1955660.0, ans=0.2 2023-11-22 13:13:07,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1955660.0, ans=0.125 2023-11-22 13:13:08,745 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.62 vs. limit=22.5 2023-11-22 13:13:23,048 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4800, loss[loss=0.06608, simple_loss=0.07773, pruned_loss=0.01394, audio_tagging_loss=0.01328, over 14383.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09332, pruned_loss=0.01493, audio_tagging_loss=0.009611, over 3042136.89 frames. ], batch size: 53, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:13:43,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1955860.0, ans=0.0 2023-11-22 13:13:43,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1955860.0, ans=0.125 2023-11-22 13:13:58,940 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 8.018e+01 8.726e+01 9.477e+01 1.215e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-22 13:14:00,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293400 2023-11-22 13:14:04,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1955993.3333333333, ans=0.2 2023-11-22 13:14:08,652 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.61 vs. limit=15.0 2023-11-22 13:14:14,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1956060.0, ans=0.0 2023-11-22 13:14:28,262 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4850, loss[loss=0.06811, simple_loss=0.09467, pruned_loss=0.01249, audio_tagging_loss=0.008279, over 15357.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09477, pruned_loss=0.01521, audio_tagging_loss=0.009629, over 3046296.73 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:14:30,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1956126.6666666667, ans=0.0 2023-11-22 13:14:38,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.07 vs. limit=15.0 2023-11-22 13:14:51,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1956193.3333333333, ans=0.025 2023-11-22 13:14:55,764 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.11 vs. limit=15.0 2023-11-22 13:15:06,486 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293450 2023-11-22 13:15:06,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1956326.6666666667, ans=0.125 2023-11-22 13:15:26,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.25 vs. limit=15.0 2023-11-22 13:15:27,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.89 vs. limit=15.0 2023-11-22 13:15:33,150 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4900, loss[loss=0.06603, simple_loss=0.08711, pruned_loss=0.01233, audio_tagging_loss=0.01015, over 15035.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09491, pruned_loss=0.01508, audio_tagging_loss=0.009498, over 3039062.82 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:15:44,158 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0 2023-11-22 13:15:49,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1956526.6666666667, ans=0.125 2023-11-22 13:16:00,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1956593.3333333333, ans=0.125 2023-11-22 13:16:06,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1956593.3333333333, ans=0.0 2023-11-22 13:16:10,099 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.038e+01 8.107e+01 8.809e+01 9.600e+01 1.365e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 13:16:11,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293500 2023-11-22 13:16:35,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1956726.6666666667, ans=0.0 2023-11-22 13:16:38,754 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 4950, loss[loss=0.07624, simple_loss=0.1017, pruned_loss=0.01903, audio_tagging_loss=0.006344, over 15227.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09403, pruned_loss=0.01512, audio_tagging_loss=0.009389, over 3036376.00 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:16:38,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1956793.3333333333, ans=0.125 2023-11-22 13:16:51,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1956860.0, ans=0.125 2023-11-22 13:16:52,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1956860.0, ans=0.0 2023-11-22 13:16:55,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1956860.0, ans=0.0 2023-11-22 13:17:00,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1956860.0, ans=0.125 2023-11-22 13:17:05,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1956926.6666666667, ans=0.125 2023-11-22 13:17:11,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1956926.6666666667, ans=0.125 2023-11-22 13:17:11,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1956926.6666666667, ans=0.1 2023-11-22 13:17:16,595 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293550 2023-11-22 13:17:23,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.09 vs. limit=15.0 2023-11-22 13:17:44,370 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5000, loss[loss=0.07313, simple_loss=0.1023, pruned_loss=0.01422, audio_tagging_loss=0.007745, over 16156.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09413, pruned_loss=0.01513, audio_tagging_loss=0.009292, over 3037863.81 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:17:48,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1957126.6666666667, ans=0.035 2023-11-22 13:17:49,131 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.45 vs. limit=15.0 2023-11-22 13:18:15,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1957260.0, ans=0.1 2023-11-22 13:18:20,731 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.208e+01 8.762e+01 9.402e+01 1.160e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-22 13:18:22,146 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293600 2023-11-22 13:18:49,655 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5050, loss[loss=0.0715, simple_loss=0.1002, pruned_loss=0.01308, audio_tagging_loss=0.008297, over 16556.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09339, pruned_loss=0.01502, audio_tagging_loss=0.009266, over 3037779.16 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:18:49,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1957460.0, ans=0.125 2023-11-22 13:18:57,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1957460.0, ans=0.125 2023-11-22 13:19:09,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.31 vs. limit=12.0 2023-11-22 13:19:17,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1957593.3333333333, ans=0.015 2023-11-22 13:19:27,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293650 2023-11-22 13:19:37,964 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.05 vs. limit=6.0 2023-11-22 13:19:50,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1957726.6666666667, ans=0.125 2023-11-22 13:19:54,029 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5100, loss[loss=0.05725, simple_loss=0.08027, pruned_loss=0.007063, audio_tagging_loss=0.01005, over 14275.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09341, pruned_loss=0.01496, audio_tagging_loss=0.009252, over 3033052.36 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:20:00,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1957793.3333333333, ans=0.2 2023-11-22 13:20:00,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.15 vs. limit=22.5 2023-11-22 13:20:05,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1957793.3333333333, ans=0.0 2023-11-22 13:20:07,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1957860.0, ans=0.5 2023-11-22 13:20:10,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1957860.0, ans=0.2 2023-11-22 13:20:30,030 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.490e+01 8.194e+01 8.730e+01 9.544e+01 1.434e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-22 13:20:31,370 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293700 2023-11-22 13:20:49,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1958060.0, ans=0.125 2023-11-22 13:20:52,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1958060.0, ans=0.125 2023-11-22 13:20:58,908 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5150, loss[loss=0.07967, simple_loss=0.1078, pruned_loss=0.01554, audio_tagging_loss=0.01022, over 14794.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09339, pruned_loss=0.01487, audio_tagging_loss=0.009135, over 3039787.85 frames. ], batch size: 53, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:21:11,486 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.39 vs. limit=5.0 2023-11-22 13:21:14,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.70 vs. limit=15.0 2023-11-22 13:21:26,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1958260.0, ans=0.0 2023-11-22 13:21:35,844 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293750 2023-11-22 13:21:50,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1958393.3333333333, ans=0.0 2023-11-22 13:22:03,634 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5200, loss[loss=0.08039, simple_loss=0.09245, pruned_loss=0.02182, audio_tagging_loss=0.01234, over 14876.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09319, pruned_loss=0.01469, audio_tagging_loss=0.009197, over 3038722.68 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:22:06,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1958460.0, ans=0.05 2023-11-22 13:22:11,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1958460.0, ans=0.2 2023-11-22 13:22:18,403 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.97 vs. limit=15.0 2023-11-22 13:22:26,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1958526.6666666667, ans=0.125 2023-11-22 13:22:39,710 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.666e+01 8.256e+01 9.056e+01 9.741e+01 1.453e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-22 13:22:41,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293800 2023-11-22 13:22:42,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.44 vs. limit=10.0 2023-11-22 13:23:08,261 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5250, loss[loss=0.07106, simple_loss=0.09956, pruned_loss=0.01346, audio_tagging_loss=0.007823, over 15069.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09346, pruned_loss=0.01478, audio_tagging_loss=0.009225, over 3036353.72 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:23:20,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1958860.0, ans=0.125 2023-11-22 13:23:34,981 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.95 vs. limit=15.0 2023-11-22 13:23:46,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293850 2023-11-22 13:24:03,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1959060.0, ans=0.125 2023-11-22 13:24:13,217 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5300, loss[loss=0.07837, simple_loss=0.1039, pruned_loss=0.01982, audio_tagging_loss=0.006595, over 14306.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09364, pruned_loss=0.01489, audio_tagging_loss=0.009176, over 3026747.34 frames. ], batch size: 53, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:24:45,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1959260.0, ans=0.125 2023-11-22 13:24:51,123 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.505e+01 9.104e+01 9.788e+01 1.254e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-22 13:24:51,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293900 2023-11-22 13:24:53,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1959326.6666666667, ans=0.0 2023-11-22 13:24:55,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1959326.6666666667, ans=0.125 2023-11-22 13:25:04,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1959393.3333333333, ans=0.0 2023-11-22 13:25:12,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1959393.3333333333, ans=0.1 2023-11-22 13:25:18,118 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5350, loss[loss=0.0756, simple_loss=0.1022, pruned_loss=0.01493, audio_tagging_loss=0.009589, over 16571.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09303, pruned_loss=0.01482, audio_tagging_loss=0.009273, over 3025814.33 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:25:26,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1959460.0, ans=0.0 2023-11-22 13:25:36,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1959526.6666666667, ans=0.2 2023-11-22 13:25:42,178 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.81 vs. limit=15.0 2023-11-22 13:25:50,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1959593.3333333333, ans=0.2 2023-11-22 13:25:55,599 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 293950 2023-11-22 13:26:20,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1959726.6666666667, ans=0.0 2023-11-22 13:26:23,245 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5400, loss[loss=0.05253, simple_loss=0.05999, pruned_loss=0.01209, audio_tagging_loss=0.01045, over 14967.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09326, pruned_loss=0.01498, audio_tagging_loss=0.009262, over 3026607.29 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:26:23,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1959793.3333333333, ans=0.1 2023-11-22 13:26:24,800 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 13:26:30,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1959793.3333333333, ans=0.0 2023-11-22 13:26:32,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1959793.3333333333, ans=0.07 2023-11-22 13:26:46,356 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.28 vs. limit=15.0 2023-11-22 13:27:00,599 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.190e+01 8.796e+01 9.372e+01 1.218e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 13:27:00,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294000 2023-11-22 13:27:07,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1959993.3333333333, ans=0.5 2023-11-22 13:27:15,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1960060.0, ans=0.125 2023-11-22 13:27:27,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1960126.6666666667, ans=0.0 2023-11-22 13:27:28,013 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5450, loss[loss=0.07608, simple_loss=0.09809, pruned_loss=0.01404, audio_tagging_loss=0.013, over 14746.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.0941, pruned_loss=0.01512, audio_tagging_loss=0.009294, over 3032814.33 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:27:31,232 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.20 vs. limit=15.0 2023-11-22 13:27:57,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1960260.0, ans=0.0 2023-11-22 13:27:58,385 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 13:28:06,115 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294050 2023-11-22 13:28:12,848 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.46 vs. limit=12.0 2023-11-22 13:28:22,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1960393.3333333333, ans=0.1 2023-11-22 13:28:27,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1960393.3333333333, ans=0.0 2023-11-22 13:28:32,363 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5500, loss[loss=0.05508, simple_loss=0.07193, pruned_loss=0.009482, audio_tagging_loss=0.009634, over 15940.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09435, pruned_loss=0.01509, audio_tagging_loss=0.009111, over 3036782.50 frames. ], batch size: 61, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:28:41,086 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 13:28:53,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1960526.6666666667, ans=0.125 2023-11-22 13:29:03,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1960593.3333333333, ans=0.0 2023-11-22 13:29:05,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1960593.3333333333, ans=0.2 2023-11-22 13:29:10,071 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.609e+01 8.387e+01 8.932e+01 9.464e+01 1.203e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 13:29:10,230 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294100 2023-11-22 13:29:31,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1960726.6666666667, ans=0.125 2023-11-22 13:29:33,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1960726.6666666667, ans=0.125 2023-11-22 13:29:34,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1960726.6666666667, ans=0.04949747468305833 2023-11-22 13:29:36,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1960793.3333333333, ans=0.0 2023-11-22 13:29:37,780 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5550, loss[loss=0.07137, simple_loss=0.1038, pruned_loss=0.01436, audio_tagging_loss=0.005086, over 14262.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09453, pruned_loss=0.01492, audio_tagging_loss=0.009188, over 3039743.44 frames. ], batch size: 53, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:29:47,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.06 vs. limit=22.5 2023-11-22 13:30:14,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294150 2023-11-22 13:30:19,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1960993.3333333333, ans=0.07 2023-11-22 13:30:37,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=15.17 vs. limit=15.0 2023-11-22 13:30:41,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1961126.6666666667, ans=0.125 2023-11-22 13:30:42,358 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5600, loss[loss=0.09581, simple_loss=0.1189, pruned_loss=0.02409, audio_tagging_loss=0.01227, over 15770.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09473, pruned_loss=0.01505, audio_tagging_loss=0.009306, over 3046221.70 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:30:45,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1961126.6666666667, ans=0.125 2023-11-22 13:30:50,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1961126.6666666667, ans=0.0 2023-11-22 13:31:00,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1961193.3333333333, ans=0.0 2023-11-22 13:31:00,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1961193.3333333333, ans=0.125 2023-11-22 13:31:05,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1961193.3333333333, ans=0.0 2023-11-22 13:31:19,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294200 2023-11-22 13:31:21,012 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.047e+01 8.683e+01 9.396e+01 1.171e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-22 13:31:28,720 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 13:31:33,497 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.04 vs. limit=15.0 2023-11-22 13:31:34,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.21 vs. limit=15.0 2023-11-22 13:31:38,662 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.83 vs. limit=15.0 2023-11-22 13:31:46,604 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5650, loss[loss=0.0763, simple_loss=0.1019, pruned_loss=0.01295, audio_tagging_loss=0.01237, over 15582.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09535, pruned_loss=0.01532, audio_tagging_loss=0.009345, over 3057826.28 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:32:10,202 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.99 vs. limit=6.0 2023-11-22 13:32:18,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1961593.3333333333, ans=0.0 2023-11-22 13:32:21,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.83 vs. limit=15.0 2023-11-22 13:32:24,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294250 2023-11-22 13:32:25,158 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.21 vs. limit=15.0 2023-11-22 13:32:51,462 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5700, loss[loss=0.07869, simple_loss=0.09789, pruned_loss=0.02123, audio_tagging_loss=0.008513, over 14419.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09451, pruned_loss=0.01525, audio_tagging_loss=0.009413, over 3051633.33 frames. ], batch size: 52, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:32:51,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1961793.3333333333, ans=0.0 2023-11-22 13:32:59,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1961793.3333333333, ans=0.125 2023-11-22 13:33:15,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1961860.0, ans=0.125 2023-11-22 13:33:28,462 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294300 2023-11-22 13:33:28,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1961993.3333333333, ans=0.0 2023-11-22 13:33:29,497 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.416e+01 9.012e+01 9.956e+01 1.838e+02, threshold=1.802e+02, percent-clipped=1.0 2023-11-22 13:33:42,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.77 vs. limit=15.0 2023-11-22 13:33:54,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.45 vs. limit=15.0 2023-11-22 13:33:55,988 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5750, loss[loss=0.05287, simple_loss=0.06667, pruned_loss=0.009092, audio_tagging_loss=0.01045, over 15225.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09413, pruned_loss=0.01513, audio_tagging_loss=0.009403, over 3052812.20 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:34:05,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1962126.6666666667, ans=0.125 2023-11-22 13:34:15,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1962193.3333333333, ans=0.1 2023-11-22 13:34:27,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1962260.0, ans=0.0 2023-11-22 13:34:33,968 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294350 2023-11-22 13:34:35,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1962326.6666666667, ans=0.0 2023-11-22 13:34:36,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1962326.6666666667, ans=0.0 2023-11-22 13:34:44,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.94 vs. limit=22.5 2023-11-22 13:35:00,433 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5800, loss[loss=0.07364, simple_loss=0.08139, pruned_loss=0.0226, audio_tagging_loss=0.01035, over 14248.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09479, pruned_loss=0.01526, audio_tagging_loss=0.009256, over 3057896.33 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:35:16,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1962526.6666666667, ans=0.1 2023-11-22 13:35:33,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.14 vs. limit=6.0 2023-11-22 13:35:34,879 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.79 vs. limit=15.0 2023-11-22 13:35:38,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294400 2023-11-22 13:35:39,731 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.175e+01 8.400e+01 8.933e+01 9.518e+01 1.972e+02, threshold=1.787e+02, percent-clipped=1.0 2023-11-22 13:36:05,425 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5850, loss[loss=0.06262, simple_loss=0.08096, pruned_loss=0.01336, audio_tagging_loss=0.008782, over 15370.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09445, pruned_loss=0.01507, audio_tagging_loss=0.009263, over 3057746.71 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:36:14,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1962793.3333333333, ans=0.125 2023-11-22 13:36:35,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1962926.6666666667, ans=0.0 2023-11-22 13:36:42,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294450 2023-11-22 13:36:43,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1962993.3333333333, ans=0.125 2023-11-22 13:37:00,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1963060.0, ans=0.125 2023-11-22 13:37:05,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1963060.0, ans=0.0 2023-11-22 13:37:07,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.57 vs. limit=10.0 2023-11-22 13:37:10,379 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5900, loss[loss=0.1043, simple_loss=0.1422, pruned_loss=0.02762, audio_tagging_loss=0.005555, over 14732.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09528, pruned_loss=0.0153, audio_tagging_loss=0.009254, over 3056950.65 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:37:39,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1963260.0, ans=0.1 2023-11-22 13:37:47,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294500 2023-11-22 13:37:49,036 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.759e+01 8.196e+01 8.876e+01 9.676e+01 1.604e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 13:38:08,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0 2023-11-22 13:38:11,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1963393.3333333333, ans=0.1 2023-11-22 13:38:14,677 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 5950, loss[loss=0.0668, simple_loss=0.08472, pruned_loss=0.0143, audio_tagging_loss=0.01014, over 14350.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.09425, pruned_loss=0.01506, audio_tagging_loss=0.009303, over 3051296.98 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:38:33,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1963526.6666666667, ans=0.1 2023-11-22 13:38:36,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1963526.6666666667, ans=0.125 2023-11-22 13:38:46,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1963593.3333333333, ans=0.05 2023-11-22 13:38:52,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294550 2023-11-22 13:39:05,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1963726.6666666667, ans=0.2 2023-11-22 13:39:16,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1963726.6666666667, ans=0.0 2023-11-22 13:39:19,046 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6000, loss[loss=0.08029, simple_loss=0.1053, pruned_loss=0.01984, audio_tagging_loss=0.007783, over 15502.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09378, pruned_loss=0.01496, audio_tagging_loss=0.009302, over 3048826.18 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:39:19,047 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 13:40:01,001 INFO [train_asr.py:1253] (3/4) Epoch 25, validation: loss=0.05896, simple_loss=0.05155, pruned_loss=0.00512, audio_tagging_loss=0.02806, over 4681554.00 frames. 2023-11-22 13:40:01,002 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 13:40:38,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294600 2023-11-22 13:40:39,174 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.185e+01 8.770e+01 9.296e+01 1.374e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 13:40:46,823 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 13:41:04,629 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6050, loss[loss=0.06738, simple_loss=0.08563, pruned_loss=0.01418, audio_tagging_loss=0.01038, over 15491.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09449, pruned_loss=0.01511, audio_tagging_loss=0.009244, over 3050326.43 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:41:16,953 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.39 vs. limit=15.0 2023-11-22 13:41:25,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1964193.3333333333, ans=0.1 2023-11-22 13:41:35,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1964260.0, ans=0.0 2023-11-22 13:41:42,865 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294650 2023-11-22 13:41:54,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=1964326.6666666667, ans=0.1 2023-11-22 13:42:09,283 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6100, loss[loss=0.05601, simple_loss=0.0749, pruned_loss=0.008367, audio_tagging_loss=0.0102, over 14042.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.0946, pruned_loss=0.01518, audio_tagging_loss=0.00922, over 3051995.26 frames. ], batch size: 52, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:42:46,730 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294700 2023-11-22 13:42:47,802 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.151e+01 8.939e+01 9.703e+01 1.163e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-22 13:42:56,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1964660.0, ans=0.2 2023-11-22 13:43:04,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1964726.6666666667, ans=0.125 2023-11-22 13:43:07,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1964726.6666666667, ans=0.125 2023-11-22 13:43:13,333 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6150, loss[loss=0.09186, simple_loss=0.1283, pruned_loss=0.02124, audio_tagging_loss=0.006457, over 15686.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09483, pruned_loss=0.01508, audio_tagging_loss=0.00919, over 3049640.13 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:43:29,815 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.53 vs. limit=15.0 2023-11-22 13:43:39,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1964926.6666666667, ans=0.125 2023-11-22 13:43:41,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1964926.6666666667, ans=0.0 2023-11-22 13:43:47,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1964926.6666666667, ans=0.2 2023-11-22 13:43:50,867 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294750 2023-11-22 13:43:55,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1964993.3333333333, ans=0.2 2023-11-22 13:44:00,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1964993.3333333333, ans=0.125 2023-11-22 13:44:18,518 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6200, loss[loss=0.07262, simple_loss=0.1015, pruned_loss=0.01293, audio_tagging_loss=0.00894, over 15827.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09392, pruned_loss=0.01508, audio_tagging_loss=0.009304, over 3045844.51 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:44:26,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1965126.6666666667, ans=0.0 2023-11-22 13:44:30,772 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.29 vs. limit=22.5 2023-11-22 13:44:39,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1965193.3333333333, ans=0.1 2023-11-22 13:44:42,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1965193.3333333333, ans=0.0 2023-11-22 13:44:42,545 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.88 vs. limit=22.5 2023-11-22 13:44:53,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1965260.0, ans=0.1 2023-11-22 13:44:56,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294800 2023-11-22 13:44:57,268 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 7.998e+01 8.582e+01 9.232e+01 1.087e+02, threshold=1.716e+02, percent-clipped=0.0 2023-11-22 13:45:00,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1965326.6666666667, ans=0.125 2023-11-22 13:45:00,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1965326.6666666667, ans=0.125 2023-11-22 13:45:01,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1965326.6666666667, ans=0.0 2023-11-22 13:45:06,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1965326.6666666667, ans=0.1 2023-11-22 13:45:08,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.75 vs. limit=12.0 2023-11-22 13:45:17,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1965393.3333333333, ans=0.125 2023-11-22 13:45:23,729 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6250, loss[loss=0.06825, simple_loss=0.09197, pruned_loss=0.0128, audio_tagging_loss=0.009472, over 15456.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09298, pruned_loss=0.01499, audio_tagging_loss=0.009401, over 3044763.50 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:45:27,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1965460.0, ans=0.0 2023-11-22 13:45:29,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1965460.0, ans=0.5 2023-11-22 13:45:42,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.07 vs. limit=15.0 2023-11-22 13:45:54,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1965593.3333333333, ans=0.2 2023-11-22 13:45:56,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1965593.3333333333, ans=0.125 2023-11-22 13:46:01,150 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294850 2023-11-22 13:46:24,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1965726.6666666667, ans=0.2 2023-11-22 13:46:28,191 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6300, loss[loss=0.05013, simple_loss=0.06903, pruned_loss=0.007826, audio_tagging_loss=0.007791, over 16422.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09319, pruned_loss=0.01492, audio_tagging_loss=0.009504, over 3051135.21 frames. ], batch size: 63, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:46:34,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1965793.3333333333, ans=0.125 2023-11-22 13:46:54,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1965926.6666666667, ans=0.2 2023-11-22 13:46:58,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1965926.6666666667, ans=0.0 2023-11-22 13:47:01,235 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0 2023-11-22 13:47:05,596 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294900 2023-11-22 13:47:07,906 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.416e+01 8.996e+01 9.619e+01 1.207e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 13:47:11,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1965993.3333333333, ans=0.2 2023-11-22 13:47:32,522 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6350, loss[loss=0.07689, simple_loss=0.1008, pruned_loss=0.01835, audio_tagging_loss=0.008152, over 16551.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09405, pruned_loss=0.01503, audio_tagging_loss=0.009498, over 3048298.42 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 16.0 2023-11-22 13:47:41,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1966126.6666666667, ans=10.0 2023-11-22 13:47:52,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1966193.3333333333, ans=0.1 2023-11-22 13:48:09,585 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.96 vs. limit=22.5 2023-11-22 13:48:10,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 294950 2023-11-22 13:48:38,021 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6400, loss[loss=0.09418, simple_loss=0.1242, pruned_loss=0.02455, audio_tagging_loss=0.007531, over 14869.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09329, pruned_loss=0.01488, audio_tagging_loss=0.009516, over 3042245.67 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:48:41,579 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.17 vs. limit=10.0 2023-11-22 13:48:42,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1966460.0, ans=0.0 2023-11-22 13:48:44,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1966460.0, ans=0.0 2023-11-22 13:48:45,970 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 13:48:50,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.55 vs. limit=22.5 2023-11-22 13:48:55,191 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=15.0 2023-11-22 13:49:00,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1966526.6666666667, ans=0.07 2023-11-22 13:49:03,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1966593.3333333333, ans=0.125 2023-11-22 13:49:11,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1966593.3333333333, ans=0.1 2023-11-22 13:49:13,290 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.91 vs. limit=15.0 2023-11-22 13:49:15,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295000 2023-11-22 13:49:17,964 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.103e+01 8.926e+01 9.580e+01 1.211e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-22 13:49:43,151 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6450, loss[loss=0.06765, simple_loss=0.07477, pruned_loss=0.01745, audio_tagging_loss=0.01281, over 14085.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09347, pruned_loss=0.01494, audio_tagging_loss=0.009535, over 3035970.38 frames. ], batch size: 53, lr: 2.73e-03, grad_scale: 32.0 2023-11-22 13:50:00,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1966860.0, ans=0.125 2023-11-22 13:50:01,926 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.39 vs. limit=12.0 2023-11-22 13:50:19,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1966926.6666666667, ans=0.1 2023-11-22 13:50:20,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295050 2023-11-22 13:50:30,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1966993.3333333333, ans=0.1 2023-11-22 13:50:35,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1967060.0, ans=0.1 2023-11-22 13:50:44,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1967060.0, ans=0.125 2023-11-22 13:50:47,828 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6500, loss[loss=0.06885, simple_loss=0.09785, pruned_loss=0.01165, audio_tagging_loss=0.008273, over 15833.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09383, pruned_loss=0.01489, audio_tagging_loss=0.009485, over 3036369.49 frames. ], batch size: 60, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:51:03,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1967193.3333333333, ans=0.0 2023-11-22 13:51:12,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1967260.0, ans=0.0 2023-11-22 13:51:13,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1967260.0, ans=0.125 2023-11-22 13:51:21,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1967260.0, ans=0.0 2023-11-22 13:51:25,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295100 2023-11-22 13:51:27,965 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.137e+01 8.907e+01 9.781e+01 1.377e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-22 13:51:51,251 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6550, loss[loss=0.08541, simple_loss=0.1059, pruned_loss=0.02061, audio_tagging_loss=0.01184, over 15766.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09419, pruned_loss=0.01496, audio_tagging_loss=0.009274, over 3044320.55 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:52:13,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1967526.6666666667, ans=0.0 2023-11-22 13:52:22,461 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.64 vs. limit=6.0 2023-11-22 13:52:23,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1967593.3333333333, ans=0.0 2023-11-22 13:52:29,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295150 2023-11-22 13:52:36,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1967660.0, ans=0.07 2023-11-22 13:52:56,803 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6600, loss[loss=0.05649, simple_loss=0.07299, pruned_loss=0.01028, audio_tagging_loss=0.00972, over 15660.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09319, pruned_loss=0.01492, audio_tagging_loss=0.00932, over 3036433.16 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:53:00,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1967793.3333333333, ans=0.0 2023-11-22 13:53:17,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1967860.0, ans=0.125 2023-11-22 13:53:34,444 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295200 2023-11-22 13:53:36,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1967993.3333333333, ans=0.0 2023-11-22 13:53:37,131 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.628e+01 8.017e+01 8.525e+01 9.445e+01 1.151e+02, threshold=1.705e+02, percent-clipped=0.0 2023-11-22 13:53:41,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1967993.3333333333, ans=0.0 2023-11-22 13:53:41,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1967993.3333333333, ans=0.2 2023-11-22 13:53:49,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1968060.0, ans=0.07 2023-11-22 13:53:56,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1968060.0, ans=0.125 2023-11-22 13:54:02,039 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6650, loss[loss=0.06113, simple_loss=0.07727, pruned_loss=0.01283, audio_tagging_loss=0.009664, over 16239.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09379, pruned_loss=0.01508, audio_tagging_loss=0.009173, over 3039931.38 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:54:08,788 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.06 vs. limit=12.0 2023-11-22 13:54:24,105 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.35 vs. limit=15.0 2023-11-22 13:54:25,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1968193.3333333333, ans=0.0 2023-11-22 13:54:39,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295250 2023-11-22 13:54:47,078 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.10 vs. limit=22.5 2023-11-22 13:54:55,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1968393.3333333333, ans=0.1 2023-11-22 13:55:05,851 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6700, loss[loss=0.09902, simple_loss=0.1317, pruned_loss=0.02688, audio_tagging_loss=0.006305, over 16805.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09364, pruned_loss=0.01513, audio_tagging_loss=0.009126, over 3034829.51 frames. ], batch size: 62, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:55:11,224 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.02 vs. limit=15.0 2023-11-22 13:55:12,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1968460.0, ans=0.125 2023-11-22 13:55:16,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1968460.0, ans=0.07 2023-11-22 13:55:26,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1968526.6666666667, ans=0.1 2023-11-22 13:55:36,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1968593.3333333333, ans=0.125 2023-11-22 13:55:43,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1968593.3333333333, ans=0.125 2023-11-22 13:55:44,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295300 2023-11-22 13:55:45,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1968660.0, ans=0.125 2023-11-22 13:55:46,367 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.891e+01 8.344e+01 8.911e+01 9.672e+01 1.240e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-22 13:56:06,614 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 13:56:11,669 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6750, loss[loss=0.06679, simple_loss=0.09175, pruned_loss=0.01109, audio_tagging_loss=0.009818, over 14899.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09371, pruned_loss=0.01514, audio_tagging_loss=0.009128, over 3035687.67 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:56:29,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1968860.0, ans=0.1 2023-11-22 13:56:29,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1968860.0, ans=0.07 2023-11-22 13:56:48,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295350 2023-11-22 13:57:15,708 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6800, loss[loss=0.06252, simple_loss=0.08388, pruned_loss=0.01099, audio_tagging_loss=0.009599, over 15440.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09316, pruned_loss=0.01501, audio_tagging_loss=0.009128, over 3034672.53 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:57:24,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1969126.6666666667, ans=0.125 2023-11-22 13:57:25,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1969126.6666666667, ans=0.0 2023-11-22 13:57:53,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295400 2023-11-22 13:57:56,106 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.834e+01 8.041e+01 8.623e+01 9.531e+01 1.254e+02, threshold=1.725e+02, percent-clipped=0.0 2023-11-22 13:58:13,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1969393.3333333333, ans=0.1 2023-11-22 13:58:20,161 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6850, loss[loss=0.06741, simple_loss=0.08922, pruned_loss=0.01457, audio_tagging_loss=0.008231, over 15698.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09348, pruned_loss=0.01509, audio_tagging_loss=0.009113, over 3040785.85 frames. ], batch size: 60, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:58:20,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.61 vs. limit=22.5 2023-11-22 13:58:31,260 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.78 vs. limit=22.5 2023-11-22 13:58:42,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1969526.6666666667, ans=0.2 2023-11-22 13:58:43,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1969526.6666666667, ans=0.125 2023-11-22 13:58:57,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295450 2023-11-22 13:59:21,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1969726.6666666667, ans=0.2 2023-11-22 13:59:21,644 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.91 vs. limit=15.0 2023-11-22 13:59:24,638 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6900, loss[loss=0.05499, simple_loss=0.07314, pruned_loss=0.009992, audio_tagging_loss=0.008429, over 15513.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09351, pruned_loss=0.01492, audio_tagging_loss=0.009066, over 3041926.29 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 13:59:33,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=1969793.3333333333, ans=0.2 2023-11-22 13:59:35,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=1969793.3333333333, ans=0.2 2023-11-22 14:00:02,180 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295500 2023-11-22 14:00:02,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.03 vs. limit=15.0 2023-11-22 14:00:04,526 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.094e+01 8.795e+01 9.359e+01 1.337e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 14:00:12,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.68 vs. limit=15.0 2023-11-22 14:00:14,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1969993.3333333333, ans=0.125 2023-11-22 14:00:15,059 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 14:00:30,096 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 6950, loss[loss=0.07162, simple_loss=0.09411, pruned_loss=0.01433, audio_tagging_loss=0.01023, over 15360.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09313, pruned_loss=0.01485, audio_tagging_loss=0.009159, over 3044374.37 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:00:52,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1970193.3333333333, ans=0.0 2023-11-22 14:01:07,007 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295550 2023-11-22 14:01:12,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1970326.6666666667, ans=0.0 2023-11-22 14:01:33,786 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7000, loss[loss=0.05234, simple_loss=0.05728, pruned_loss=0.009985, audio_tagging_loss=0.01372, over 15113.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09265, pruned_loss=0.01464, audio_tagging_loss=0.009253, over 3044509.07 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:01:37,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1970460.0, ans=0.125 2023-11-22 14:01:42,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1970460.0, ans=0.1 2023-11-22 14:02:01,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1970593.3333333333, ans=0.1 2023-11-22 14:02:12,243 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295600 2023-11-22 14:02:14,858 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.282e+01 8.833e+01 9.554e+01 1.299e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 14:02:32,624 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.81 vs. limit=6.0 2023-11-22 14:02:38,747 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7050, loss[loss=0.07963, simple_loss=0.1049, pruned_loss=0.0211, audio_tagging_loss=0.006096, over 16335.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09273, pruned_loss=0.01477, audio_tagging_loss=0.009369, over 3047871.23 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:02:38,962 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 14:02:49,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1970793.3333333333, ans=0.09899494936611666 2023-11-22 14:02:54,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1970860.0, ans=0.125 2023-11-22 14:03:08,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=1970926.6666666667, ans=15.0 2023-11-22 14:03:15,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1970926.6666666667, ans=0.1 2023-11-22 14:03:16,322 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295650 2023-11-22 14:03:26,336 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.96 vs. limit=22.5 2023-11-22 14:03:31,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1971060.0, ans=0.0 2023-11-22 14:03:35,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1971060.0, ans=0.125 2023-11-22 14:03:43,545 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7100, loss[loss=0.05774, simple_loss=0.08039, pruned_loss=0.008637, audio_tagging_loss=0.008909, over 15442.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09275, pruned_loss=0.01488, audio_tagging_loss=0.009486, over 3050856.87 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:04:05,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.35 vs. limit=15.0 2023-11-22 14:04:08,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1971260.0, ans=0.125 2023-11-22 14:04:10,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1971260.0, ans=0.0 2023-11-22 14:04:11,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1971260.0, ans=0.125 2023-11-22 14:04:12,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1971260.0, ans=0.125 2023-11-22 14:04:21,111 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295700 2023-11-22 14:04:24,048 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.394e+01 8.961e+01 9.557e+01 1.218e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-22 14:04:25,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1971326.6666666667, ans=0.125 2023-11-22 14:04:39,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1971393.3333333333, ans=0.125 2023-11-22 14:04:47,938 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7150, loss[loss=0.07292, simple_loss=0.1045, pruned_loss=0.01329, audio_tagging_loss=0.007365, over 15798.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09272, pruned_loss=0.01475, audio_tagging_loss=0.009491, over 3055723.48 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:04:48,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1971460.0, ans=0.05 2023-11-22 14:04:55,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1971460.0, ans=0.0 2023-11-22 14:04:59,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1971526.6666666667, ans=0.125 2023-11-22 14:05:04,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1971526.6666666667, ans=0.125 2023-11-22 14:05:09,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1971526.6666666667, ans=0.0 2023-11-22 14:05:24,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1971593.3333333333, ans=0.125 2023-11-22 14:05:26,293 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295750 2023-11-22 14:05:29,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1971660.0, ans=0.125 2023-11-22 14:05:52,777 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7200, loss[loss=0.06616, simple_loss=0.08581, pruned_loss=0.008226, audio_tagging_loss=0.01503, over 14745.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09302, pruned_loss=0.01488, audio_tagging_loss=0.009542, over 3053522.54 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:05:58,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1971793.3333333333, ans=0.125 2023-11-22 14:06:00,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1971793.3333333333, ans=0.0 2023-11-22 14:06:03,603 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 14:06:03,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1971793.3333333333, ans=0.2 2023-11-22 14:06:30,654 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295800 2023-11-22 14:06:33,416 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.672e+01 8.039e+01 8.548e+01 9.116e+01 1.085e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-22 14:06:35,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1971993.3333333333, ans=0.125 2023-11-22 14:06:45,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1972060.0, ans=0.125 2023-11-22 14:06:48,421 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.05 vs. limit=22.5 2023-11-22 14:06:58,317 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7250, loss[loss=0.08752, simple_loss=0.1131, pruned_loss=0.01936, audio_tagging_loss=0.01162, over 14563.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09407, pruned_loss=0.01494, audio_tagging_loss=0.009522, over 3054822.72 frames. ], batch size: 53, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:07:06,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1972126.6666666667, ans=0.025 2023-11-22 14:07:07,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1972126.6666666667, ans=0.125 2023-11-22 14:07:32,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1972260.0, ans=0.0 2023-11-22 14:07:36,050 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295850 2023-11-22 14:07:45,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1972326.6666666667, ans=0.125 2023-11-22 14:07:49,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1972393.3333333333, ans=0.125 2023-11-22 14:07:49,524 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.76 vs. limit=10.0 2023-11-22 14:08:01,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1972393.3333333333, ans=0.125 2023-11-22 14:08:03,633 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7300, loss[loss=0.06187, simple_loss=0.08248, pruned_loss=0.01353, audio_tagging_loss=0.007104, over 15485.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09448, pruned_loss=0.01523, audio_tagging_loss=0.00944, over 3055247.91 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:08:20,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1972526.6666666667, ans=0.0 2023-11-22 14:08:25,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1972526.6666666667, ans=0.125 2023-11-22 14:08:36,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1972593.3333333333, ans=0.125 2023-11-22 14:08:40,760 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295900 2023-11-22 14:08:44,843 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.276e+01 8.894e+01 9.525e+01 1.269e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-22 14:08:51,411 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.69 vs. limit=15.0 2023-11-22 14:08:52,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.13 vs. limit=15.0 2023-11-22 14:09:08,260 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7350, loss[loss=0.05368, simple_loss=0.07136, pruned_loss=0.009347, audio_tagging_loss=0.008654, over 14677.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09498, pruned_loss=0.01522, audio_tagging_loss=0.009234, over 3051469.86 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:09:09,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1972793.3333333333, ans=0.125 2023-11-22 14:09:35,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1972926.6666666667, ans=0.2 2023-11-22 14:09:45,377 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 295950 2023-11-22 14:09:57,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1972993.3333333333, ans=0.2 2023-11-22 14:10:12,120 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7400, loss[loss=0.08575, simple_loss=0.1142, pruned_loss=0.0228, audio_tagging_loss=0.005853, over 14291.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09455, pruned_loss=0.01518, audio_tagging_loss=0.009155, over 3048740.30 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:10:25,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1973193.3333333333, ans=0.125 2023-11-22 14:10:27,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1973193.3333333333, ans=0.2 2023-11-22 14:10:41,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1973260.0, ans=0.125 2023-11-22 14:10:48,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1973260.0, ans=0.0 2023-11-22 14:10:49,667 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296000 2023-11-22 14:10:51,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1973326.6666666667, ans=0.0 2023-11-22 14:10:56,592 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.500e+01 8.081e+01 8.804e+01 9.585e+01 1.209e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-22 14:11:19,954 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7450, loss[loss=0.05536, simple_loss=0.06983, pruned_loss=0.01003, audio_tagging_loss=0.01042, over 15089.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09424, pruned_loss=0.01533, audio_tagging_loss=0.009236, over 3040391.16 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:11:28,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1973460.0, ans=0.0 2023-11-22 14:11:32,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.81 vs. limit=15.0 2023-11-22 14:11:40,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1973526.6666666667, ans=0.1 2023-11-22 14:11:43,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1973526.6666666667, ans=0.125 2023-11-22 14:11:57,586 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296050 2023-11-22 14:12:03,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1973660.0, ans=0.125 2023-11-22 14:12:20,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1973726.6666666667, ans=0.0 2023-11-22 14:12:24,569 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7500, loss[loss=0.07475, simple_loss=0.1004, pruned_loss=0.01658, audio_tagging_loss=0.007955, over 15112.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09466, pruned_loss=0.01523, audio_tagging_loss=0.009128, over 3044064.06 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:12:27,994 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 14:12:34,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1973793.3333333333, ans=0.125 2023-11-22 14:12:42,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1973860.0, ans=0.125 2023-11-22 14:12:51,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1973926.6666666667, ans=0.07 2023-11-22 14:13:02,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296100 2023-11-22 14:13:02,561 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.51 vs. limit=15.0 2023-11-22 14:13:06,435 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.270e+01 8.171e+01 8.790e+01 9.530e+01 1.185e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 14:13:13,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=12.0 2023-11-22 14:13:29,561 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7550, loss[loss=0.06576, simple_loss=0.08612, pruned_loss=0.0121, audio_tagging_loss=0.01059, over 16390.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09472, pruned_loss=0.01524, audio_tagging_loss=0.009137, over 3052792.53 frames. ], batch size: 63, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:13:37,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1974126.6666666667, ans=0.1 2023-11-22 14:13:40,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1974126.6666666667, ans=0.0 2023-11-22 14:13:45,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1974193.3333333333, ans=0.125 2023-11-22 14:13:59,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.59 vs. limit=22.5 2023-11-22 14:14:01,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1974260.0, ans=0.125 2023-11-22 14:14:07,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296150 2023-11-22 14:14:33,781 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7600, loss[loss=0.06142, simple_loss=0.08071, pruned_loss=0.01031, audio_tagging_loss=0.01075, over 15347.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09417, pruned_loss=0.01511, audio_tagging_loss=0.009138, over 3048640.77 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:14:35,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1974460.0, ans=0.035 2023-11-22 14:14:40,397 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.18 vs. limit=22.5 2023-11-22 14:14:41,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1974460.0, ans=0.2 2023-11-22 14:15:08,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1974593.3333333333, ans=0.2 2023-11-22 14:15:11,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296200 2023-11-22 14:15:15,776 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.584e+01 8.244e+01 8.772e+01 9.413e+01 1.274e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 14:15:21,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1974660.0, ans=0.07 2023-11-22 14:15:28,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1974726.6666666667, ans=0.125 2023-11-22 14:15:31,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1974726.6666666667, ans=0.2 2023-11-22 14:15:35,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1974726.6666666667, ans=0.07 2023-11-22 14:15:39,246 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7650, loss[loss=0.03752, simple_loss=0.04232, pruned_loss=0.005041, audio_tagging_loss=0.01131, over 14549.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09353, pruned_loss=0.01504, audio_tagging_loss=0.009189, over 3048084.07 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:15:47,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1974793.3333333333, ans=0.1 2023-11-22 14:16:11,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1974926.6666666667, ans=0.09899494936611666 2023-11-22 14:16:17,218 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296250 2023-11-22 14:16:17,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1974993.3333333333, ans=0.0 2023-11-22 14:16:22,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1974993.3333333333, ans=0.0 2023-11-22 14:16:44,136 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7700, loss[loss=0.05699, simple_loss=0.0764, pruned_loss=0.01203, audio_tagging_loss=0.006762, over 13904.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09432, pruned_loss=0.01517, audio_tagging_loss=0.009091, over 3050260.76 frames. ], batch size: 53, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:17:06,637 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0 2023-11-22 14:17:21,577 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296300 2023-11-22 14:17:25,651 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.979e+01 8.266e+01 8.985e+01 9.649e+01 1.326e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 14:17:46,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1975393.3333333333, ans=0.2 2023-11-22 14:17:46,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1975393.3333333333, ans=0.07 2023-11-22 14:17:48,833 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7750, loss[loss=0.07927, simple_loss=0.1047, pruned_loss=0.01647, audio_tagging_loss=0.01044, over 15132.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.095, pruned_loss=0.01519, audio_tagging_loss=0.009202, over 3054850.30 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0 2023-11-22 14:18:02,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1975526.6666666667, ans=0.125 2023-11-22 14:18:06,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1975526.6666666667, ans=0.2 2023-11-22 14:18:10,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1975526.6666666667, ans=0.1 2023-11-22 14:18:26,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296350 2023-11-22 14:18:43,697 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.15 vs. limit=15.0 2023-11-22 14:18:52,675 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7800, loss[loss=0.07519, simple_loss=0.09487, pruned_loss=0.01709, audio_tagging_loss=0.01066, over 14786.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09406, pruned_loss=0.01501, audio_tagging_loss=0.009371, over 3053326.80 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:19:09,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1975860.0, ans=0.2 2023-11-22 14:19:25,422 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=11.11 vs. limit=12.0 2023-11-22 14:19:30,992 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296400 2023-11-22 14:19:35,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1975993.3333333333, ans=0.0 2023-11-22 14:19:36,052 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 7.916e+01 8.644e+01 9.515e+01 1.205e+02, threshold=1.729e+02, percent-clipped=0.0 2023-11-22 14:19:46,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1976060.0, ans=0.1 2023-11-22 14:19:53,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1976060.0, ans=0.125 2023-11-22 14:19:53,857 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.93 vs. limit=15.0 2023-11-22 14:19:56,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1976060.0, ans=0.025 2023-11-22 14:19:58,147 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7850, loss[loss=0.05547, simple_loss=0.07195, pruned_loss=0.008997, audio_tagging_loss=0.0105, over 15406.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09402, pruned_loss=0.01497, audio_tagging_loss=0.009464, over 3053566.26 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:20:24,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1976260.0, ans=0.0 2023-11-22 14:20:28,267 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.17 vs. limit=22.5 2023-11-22 14:20:34,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296450 2023-11-22 14:20:54,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1976393.3333333333, ans=0.0 2023-11-22 14:20:57,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1976393.3333333333, ans=0.125 2023-11-22 14:20:58,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1976393.3333333333, ans=0.0 2023-11-22 14:21:02,745 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7900, loss[loss=0.04886, simple_loss=0.05752, pruned_loss=0.006066, audio_tagging_loss=0.01404, over 16768.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09359, pruned_loss=0.01497, audio_tagging_loss=0.009535, over 3056925.00 frames. ], batch size: 65, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:21:25,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1976526.6666666667, ans=0.2 2023-11-22 14:21:37,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1976593.3333333333, ans=0.125 2023-11-22 14:21:39,795 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296500 2023-11-22 14:21:42,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1976660.0, ans=0.0 2023-11-22 14:21:44,985 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.490e+01 8.875e+01 9.857e+01 1.314e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 14:21:57,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1976726.6666666667, ans=0.125 2023-11-22 14:22:06,352 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 7950, loss[loss=0.07028, simple_loss=0.1042, pruned_loss=0.01161, audio_tagging_loss=0.006566, over 15513.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09386, pruned_loss=0.01499, audio_tagging_loss=0.009475, over 3052394.90 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 8.0 2023-11-22 14:22:23,752 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 14:22:33,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1976926.6666666667, ans=0.125 2023-11-22 14:22:44,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296550 2023-11-22 14:22:59,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1977060.0, ans=0.2 2023-11-22 14:23:06,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1977060.0, ans=0.0 2023-11-22 14:23:10,976 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8000, loss[loss=0.05384, simple_loss=0.06554, pruned_loss=0.01166, audio_tagging_loss=0.009404, over 14114.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09372, pruned_loss=0.01516, audio_tagging_loss=0.009536, over 3049009.19 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:23:15,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1977126.6666666667, ans=0.0 2023-11-22 14:23:24,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1977193.3333333333, ans=0.0 2023-11-22 14:23:41,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1977260.0, ans=0.0 2023-11-22 14:23:44,120 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.74 vs. limit=15.0 2023-11-22 14:23:48,277 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296600 2023-11-22 14:23:55,289 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.359e+01 8.469e+01 8.936e+01 9.726e+01 1.360e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-22 14:24:08,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1977393.3333333333, ans=0.0 2023-11-22 14:24:12,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1977393.3333333333, ans=0.0 2023-11-22 14:24:16,317 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8050, loss[loss=0.08996, simple_loss=0.1218, pruned_loss=0.01925, audio_tagging_loss=0.009817, over 15539.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09418, pruned_loss=0.01534, audio_tagging_loss=0.009531, over 3053420.56 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:24:25,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1977460.0, ans=0.0 2023-11-22 14:24:33,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1977526.6666666667, ans=0.125 2023-11-22 14:24:40,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1977593.3333333333, ans=0.0 2023-11-22 14:24:48,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1977593.3333333333, ans=0.125 2023-11-22 14:24:52,943 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296650 2023-11-22 14:24:56,271 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 14:25:02,023 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.51 vs. limit=15.0 2023-11-22 14:25:05,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1977660.0, ans=0.1 2023-11-22 14:25:20,012 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8100, loss[loss=0.08523, simple_loss=0.1163, pruned_loss=0.0184, audio_tagging_loss=0.008667, over 14448.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.09381, pruned_loss=0.01528, audio_tagging_loss=0.009438, over 3051539.30 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:25:21,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1977793.3333333333, ans=0.1 2023-11-22 14:25:25,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1977793.3333333333, ans=0.1 2023-11-22 14:25:50,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1977926.6666666667, ans=0.125 2023-11-22 14:25:51,166 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.45 vs. limit=10.0 2023-11-22 14:25:54,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1977926.6666666667, ans=0.125 2023-11-22 14:25:57,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296700 2023-11-22 14:26:03,390 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.691e+01 8.263e+01 8.996e+01 9.612e+01 1.203e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 14:26:11,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1978060.0, ans=0.05 2023-11-22 14:26:23,604 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8150, loss[loss=0.05421, simple_loss=0.0647, pruned_loss=0.01368, audio_tagging_loss=0.008185, over 14828.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09348, pruned_loss=0.0151, audio_tagging_loss=0.009315, over 3050080.51 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:26:46,456 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.81 vs. limit=10.0 2023-11-22 14:27:00,407 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296750 2023-11-22 14:27:06,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.38 vs. limit=22.5 2023-11-22 14:27:24,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.67 vs. limit=15.0 2023-11-22 14:27:27,875 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8200, loss[loss=0.06788, simple_loss=0.09014, pruned_loss=0.01547, audio_tagging_loss=0.007344, over 14792.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09302, pruned_loss=0.01488, audio_tagging_loss=0.009204, over 3049596.22 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:27:29,139 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 14:27:29,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1978460.0, ans=0.07 2023-11-22 14:27:44,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1978526.6666666667, ans=0.125 2023-11-22 14:27:55,638 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.84 vs. limit=15.0 2023-11-22 14:28:02,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1978593.3333333333, ans=0.1 2023-11-22 14:28:04,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296800 2023-11-22 14:28:07,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1978660.0, ans=0.95 2023-11-22 14:28:10,698 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 7.904e+01 8.816e+01 9.431e+01 1.476e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-22 14:28:32,365 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8250, loss[loss=0.07553, simple_loss=0.09916, pruned_loss=0.01689, audio_tagging_loss=0.009061, over 15787.00 frames. ], tot_loss[loss=0.071, simple_loss=0.0935, pruned_loss=0.01498, audio_tagging_loss=0.009269, over 3050269.78 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:28:51,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1978860.0, ans=0.025 2023-11-22 14:28:59,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1978926.6666666667, ans=0.125 2023-11-22 14:29:04,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1978926.6666666667, ans=0.125 2023-11-22 14:29:08,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1978926.6666666667, ans=0.125 2023-11-22 14:29:09,551 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296850 2023-11-22 14:29:17,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1978993.3333333333, ans=0.1 2023-11-22 14:29:18,853 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 14:29:28,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1979060.0, ans=0.125 2023-11-22 14:29:35,949 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8300, loss[loss=0.09195, simple_loss=0.125, pruned_loss=0.02053, audio_tagging_loss=0.008893, over 14612.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09443, pruned_loss=0.0151, audio_tagging_loss=0.009236, over 3052827.13 frames. ], batch size: 53, lr: 2.72e-03, grad_scale: 8.0 2023-11-22 14:29:53,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1979193.3333333333, ans=0.125 2023-11-22 14:30:13,313 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296900 2023-11-22 14:30:15,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1979326.6666666667, ans=0.0 2023-11-22 14:30:20,390 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.061e+01 8.620e+01 9.639e+01 1.216e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-22 14:30:30,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.15 vs. limit=15.0 2023-11-22 14:30:30,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1979393.3333333333, ans=0.1 2023-11-22 14:30:39,596 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8350, loss[loss=0.07868, simple_loss=0.1044, pruned_loss=0.01829, audio_tagging_loss=0.008179, over 14806.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09335, pruned_loss=0.01506, audio_tagging_loss=0.009257, over 3058630.87 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 8.0 2023-11-22 14:30:41,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1979460.0, ans=0.125 2023-11-22 14:30:49,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1979460.0, ans=0.125 2023-11-22 14:31:17,033 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 296950 2023-11-22 14:31:18,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1979660.0, ans=0.125 2023-11-22 14:31:21,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1979660.0, ans=0.125 2023-11-22 14:31:28,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.03 vs. limit=15.0 2023-11-22 14:31:42,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1979793.3333333333, ans=0.125 2023-11-22 14:31:43,419 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8400, loss[loss=0.08397, simple_loss=0.1072, pruned_loss=0.021, audio_tagging_loss=0.009371, over 14726.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09349, pruned_loss=0.01516, audio_tagging_loss=0.009199, over 3047277.51 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:32:20,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297000 2023-11-22 14:32:21,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1979993.3333333333, ans=0.2 2023-11-22 14:32:26,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1979993.3333333333, ans=0.125 2023-11-22 14:32:29,034 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.093e+01 8.742e+01 9.671e+01 1.399e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 14:32:32,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1979993.3333333333, ans=0.0 2023-11-22 14:32:33,387 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.42 vs. limit=6.0 2023-11-22 14:32:40,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.70 vs. limit=12.0 2023-11-22 14:32:41,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1980060.0, ans=0.5 2023-11-22 14:32:47,275 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8450, loss[loss=0.0814, simple_loss=0.1036, pruned_loss=0.02003, audio_tagging_loss=0.009587, over 14691.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09408, pruned_loss=0.01517, audio_tagging_loss=0.009258, over 3050637.01 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:33:25,269 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297050 2023-11-22 14:33:26,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1980326.6666666667, ans=0.2 2023-11-22 14:33:28,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.73 vs. limit=22.5 2023-11-22 14:33:42,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1980393.3333333333, ans=0.125 2023-11-22 14:33:50,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1980460.0, ans=0.125 2023-11-22 14:33:52,047 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8500, loss[loss=0.06676, simple_loss=0.09181, pruned_loss=0.011, audio_tagging_loss=0.009854, over 15144.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09466, pruned_loss=0.01524, audio_tagging_loss=0.009251, over 3048039.55 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:33:53,969 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.84 vs. limit=10.0 2023-11-22 14:33:57,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1980460.0, ans=0.125 2023-11-22 14:34:19,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1980593.3333333333, ans=0.125 2023-11-22 14:34:28,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1980593.3333333333, ans=0.125 2023-11-22 14:34:29,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297100 2023-11-22 14:34:36,239 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.259e+01 8.972e+01 9.645e+01 1.298e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-22 14:34:51,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1980726.6666666667, ans=0.125 2023-11-22 14:34:55,747 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8550, loss[loss=0.06594, simple_loss=0.08992, pruned_loss=0.01323, audio_tagging_loss=0.007752, over 15088.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09418, pruned_loss=0.01515, audio_tagging_loss=0.00931, over 3038972.86 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:34:56,607 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.91 vs. limit=22.5 2023-11-22 14:35:11,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1980860.0, ans=0.2 2023-11-22 14:35:15,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.38 vs. limit=6.0 2023-11-22 14:35:33,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297150 2023-11-22 14:35:59,841 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8600, loss[loss=0.07862, simple_loss=0.1022, pruned_loss=0.01824, audio_tagging_loss=0.00926, over 16462.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09531, pruned_loss=0.01548, audio_tagging_loss=0.009362, over 3039503.44 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:36:14,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1981193.3333333333, ans=0.2 2023-11-22 14:36:30,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1981260.0, ans=0.1 2023-11-22 14:36:31,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1981260.0, ans=0.0 2023-11-22 14:36:31,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1981260.0, ans=0.0 2023-11-22 14:36:34,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1981260.0, ans=0.1 2023-11-22 14:36:37,021 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297200 2023-11-22 14:36:45,256 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 8.334e+01 9.004e+01 9.545e+01 1.536e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-22 14:36:51,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1981393.3333333333, ans=0.125 2023-11-22 14:37:02,468 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.57 vs. limit=15.0 2023-11-22 14:37:04,295 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8650, loss[loss=0.05911, simple_loss=0.08014, pruned_loss=0.009638, audio_tagging_loss=0.009396, over 14772.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09537, pruned_loss=0.0153, audio_tagging_loss=0.009317, over 3045571.22 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0 2023-11-22 14:37:14,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1981460.0, ans=0.125 2023-11-22 14:37:15,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.67 vs. limit=15.0 2023-11-22 14:37:35,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.28 vs. limit=6.0 2023-11-22 14:37:41,427 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297250 2023-11-22 14:37:49,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1981660.0, ans=0.5 2023-11-22 14:37:54,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1981726.6666666667, ans=0.125 2023-11-22 14:38:02,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1981726.6666666667, ans=0.0 2023-11-22 14:38:08,735 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8700, loss[loss=0.0724, simple_loss=0.09294, pruned_loss=0.01521, audio_tagging_loss=0.01072, over 14951.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09602, pruned_loss=0.01546, audio_tagging_loss=0.009315, over 3036213.06 frames. ], batch size: 59, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:38:36,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1981926.6666666667, ans=0.2 2023-11-22 14:38:45,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297300 2023-11-22 14:38:53,654 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.337e+01 8.929e+01 9.655e+01 1.210e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 14:39:09,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.36 vs. limit=22.5 2023-11-22 14:39:12,688 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8750, loss[loss=0.08149, simple_loss=0.1038, pruned_loss=0.01794, audio_tagging_loss=0.01166, over 14800.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09653, pruned_loss=0.01571, audio_tagging_loss=0.009306, over 3039327.35 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:39:34,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1982193.3333333333, ans=0.1 2023-11-22 14:39:47,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1982260.0, ans=0.0 2023-11-22 14:39:49,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297350 2023-11-22 14:40:04,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1982393.3333333333, ans=0.125 2023-11-22 14:40:04,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1982393.3333333333, ans=0.125 2023-11-22 14:40:07,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1982393.3333333333, ans=0.1 2023-11-22 14:40:16,886 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8800, loss[loss=0.08537, simple_loss=0.1118, pruned_loss=0.02019, audio_tagging_loss=0.009284, over 15550.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09677, pruned_loss=0.01572, audio_tagging_loss=0.00926, over 3044681.33 frames. ], batch size: 59, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:40:36,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1982526.6666666667, ans=0.0 2023-11-22 14:40:40,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1982526.6666666667, ans=0.125 2023-11-22 14:40:53,943 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297400 2023-11-22 14:40:55,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1982660.0, ans=0.05 2023-11-22 14:40:59,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.19 vs. limit=15.0 2023-11-22 14:41:04,593 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.228e+01 8.972e+01 9.857e+01 1.350e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-22 14:41:21,472 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8850, loss[loss=0.04095, simple_loss=0.04693, pruned_loss=0.00598, audio_tagging_loss=0.0115, over 17278.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09629, pruned_loss=0.0156, audio_tagging_loss=0.009343, over 3049984.89 frames. ], batch size: 69, lr: 2.71e-03, grad_scale: 8.0 2023-11-22 14:41:30,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1982793.3333333333, ans=0.125 2023-11-22 14:41:33,594 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 14:41:36,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1982860.0, ans=0.125 2023-11-22 14:41:58,013 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297450 2023-11-22 14:42:07,477 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.42 vs. limit=22.5 2023-11-22 14:42:11,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1983060.0, ans=0.125 2023-11-22 14:42:20,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1983060.0, ans=0.025 2023-11-22 14:42:24,239 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8900, loss[loss=0.07215, simple_loss=0.08373, pruned_loss=0.01872, audio_tagging_loss=0.01157, over 14006.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09526, pruned_loss=0.01529, audio_tagging_loss=0.009337, over 3044348.07 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 8.0 2023-11-22 14:42:34,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1983126.6666666667, ans=0.0 2023-11-22 14:43:01,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297500 2023-11-22 14:43:04,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1983326.6666666667, ans=0.125 2023-11-22 14:43:11,665 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.250e+01 8.702e+01 9.316e+01 1.378e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 14:43:28,583 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 8950, loss[loss=0.07208, simple_loss=0.1026, pruned_loss=0.01545, audio_tagging_loss=0.005331, over 15395.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09607, pruned_loss=0.01548, audio_tagging_loss=0.009211, over 3045657.57 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 8.0 2023-11-22 14:43:34,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1983460.0, ans=0.0 2023-11-22 14:43:53,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1983593.3333333333, ans=0.1 2023-11-22 14:44:04,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297550 2023-11-22 14:44:06,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1983660.0, ans=0.1 2023-11-22 14:44:07,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1983660.0, ans=0.125 2023-11-22 14:44:14,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1983660.0, ans=0.125 2023-11-22 14:44:14,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1983660.0, ans=0.125 2023-11-22 14:44:31,330 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9000, loss[loss=0.06924, simple_loss=0.08, pruned_loss=0.01946, audio_tagging_loss=0.009785, over 15034.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09495, pruned_loss=0.01533, audio_tagging_loss=0.009196, over 3046097.88 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 8.0 2023-11-22 14:44:31,331 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 14:45:11,809 INFO [train_asr.py:1253] (3/4) Epoch 25, validation: loss=0.06003, simple_loss=0.05148, pruned_loss=0.00513, audio_tagging_loss=0.02916, over 4681554.00 frames. 2023-11-22 14:45:11,809 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 14:45:17,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1983793.3333333333, ans=0.2 2023-11-22 14:45:37,037 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 14:45:49,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297600 2023-11-22 14:45:59,589 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.485e+01 9.145e+01 9.654e+01 1.276e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-22 14:46:16,688 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9050, loss[loss=0.1009, simple_loss=0.1244, pruned_loss=0.03015, audio_tagging_loss=0.00857, over 16086.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09553, pruned_loss=0.01545, audio_tagging_loss=0.009089, over 3046501.51 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 8.0 2023-11-22 14:46:19,959 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.73 vs. limit=15.0 2023-11-22 14:46:28,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1984193.3333333333, ans=0.125 2023-11-22 14:46:33,109 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.52 vs. limit=15.0 2023-11-22 14:46:37,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1984193.3333333333, ans=0.125 2023-11-22 14:46:47,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1984260.0, ans=0.0 2023-11-22 14:46:53,079 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297650 2023-11-22 14:46:54,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1984326.6666666667, ans=0.04949747468305833 2023-11-22 14:46:58,152 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.31 vs. limit=22.5 2023-11-22 14:47:03,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1984326.6666666667, ans=0.1 2023-11-22 14:47:09,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1984393.3333333333, ans=0.09899494936611666 2023-11-22 14:47:14,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1984393.3333333333, ans=0.125 2023-11-22 14:47:21,054 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9100, loss[loss=0.06776, simple_loss=0.09617, pruned_loss=0.01143, audio_tagging_loss=0.008248, over 16327.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09426, pruned_loss=0.01509, audio_tagging_loss=0.009034, over 3049415.07 frames. ], batch size: 62, lr: 2.71e-03, grad_scale: 8.0 2023-11-22 14:47:22,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1984460.0, ans=0.125 2023-11-22 14:47:44,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1984593.3333333333, ans=0.1 2023-11-22 14:47:57,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297700 2023-11-22 14:48:07,993 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.161e+01 8.752e+01 9.286e+01 2.425e+02, threshold=1.750e+02, percent-clipped=1.0 2023-11-22 14:48:10,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1984660.0, ans=0.1 2023-11-22 14:48:18,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1984726.6666666667, ans=0.2 2023-11-22 14:48:24,381 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9150, loss[loss=0.08005, simple_loss=0.1006, pruned_loss=0.01897, audio_tagging_loss=0.01079, over 15425.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09468, pruned_loss=0.01522, audio_tagging_loss=0.009097, over 3048971.07 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 8.0 2023-11-22 14:48:52,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1984926.6666666667, ans=0.125 2023-11-22 14:49:02,438 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297750 2023-11-22 14:49:28,676 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9200, loss[loss=0.0871, simple_loss=0.1086, pruned_loss=0.02385, audio_tagging_loss=0.008932, over 15690.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09487, pruned_loss=0.01536, audio_tagging_loss=0.008988, over 3049527.77 frames. ], batch size: 59, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:49:28,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1985126.6666666667, ans=0.125 2023-11-22 14:49:31,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1985126.6666666667, ans=0.09899494936611666 2023-11-22 14:50:03,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1985260.0, ans=0.125 2023-11-22 14:50:03,958 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.54 vs. limit=6.0 2023-11-22 14:50:05,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297800 2023-11-22 14:50:08,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1985326.6666666667, ans=0.2 2023-11-22 14:50:10,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1985326.6666666667, ans=0.125 2023-11-22 14:50:16,264 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.283e+01 9.032e+01 9.826e+01 1.218e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-22 14:50:30,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1985393.3333333333, ans=0.0 2023-11-22 14:50:33,433 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9250, loss[loss=0.06649, simple_loss=0.08768, pruned_loss=0.01352, audio_tagging_loss=0.009138, over 15839.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09425, pruned_loss=0.01506, audio_tagging_loss=0.009029, over 3050448.00 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:50:40,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1985460.0, ans=0.125 2023-11-22 14:50:54,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1985526.6666666667, ans=0.125 2023-11-22 14:51:10,204 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297850 2023-11-22 14:51:30,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1985726.6666666667, ans=0.0 2023-11-22 14:51:37,703 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9300, loss[loss=0.07329, simple_loss=0.1006, pruned_loss=0.01381, audio_tagging_loss=0.009171, over 14940.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09345, pruned_loss=0.01477, audio_tagging_loss=0.009174, over 3050039.83 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:51:41,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1985793.3333333333, ans=0.1 2023-11-22 14:51:49,871 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.04 vs. limit=15.0 2023-11-22 14:52:14,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297900 2023-11-22 14:52:24,527 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.158e+01 8.859e+01 9.535e+01 1.280e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-22 14:52:39,735 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.88 vs. limit=15.0 2023-11-22 14:52:41,521 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9350, loss[loss=0.06437, simple_loss=0.08738, pruned_loss=0.01112, audio_tagging_loss=0.009553, over 15405.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09401, pruned_loss=0.01494, audio_tagging_loss=0.009188, over 3052157.68 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:52:49,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1986126.6666666667, ans=0.125 2023-11-22 14:53:04,498 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.66 vs. limit=10.0 2023-11-22 14:53:15,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1986260.0, ans=0.1 2023-11-22 14:53:20,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 297950 2023-11-22 14:53:27,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1986326.6666666667, ans=0.5 2023-11-22 14:53:48,025 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9400, loss[loss=0.07943, simple_loss=0.106, pruned_loss=0.01715, audio_tagging_loss=0.009278, over 15706.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09343, pruned_loss=0.01497, audio_tagging_loss=0.00929, over 3047812.15 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:53:55,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1986460.0, ans=0.0 2023-11-22 14:54:24,945 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298000 2023-11-22 14:54:35,748 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.116e+01 8.691e+01 9.663e+01 1.322e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-22 14:54:52,085 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 14:54:53,297 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9450, loss[loss=0.05257, simple_loss=0.07252, pruned_loss=0.007209, audio_tagging_loss=0.009107, over 15212.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09343, pruned_loss=0.01494, audio_tagging_loss=0.009397, over 3048544.09 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:55:01,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1986793.3333333333, ans=0.0 2023-11-22 14:55:30,276 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.81 vs. limit=10.0 2023-11-22 14:55:30,964 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298050 2023-11-22 14:55:38,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.14 vs. limit=12.0 2023-11-22 14:55:57,820 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9500, loss[loss=0.05226, simple_loss=0.06793, pruned_loss=0.008321, audio_tagging_loss=0.009975, over 15581.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09445, pruned_loss=0.01519, audio_tagging_loss=0.009367, over 3046540.31 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:56:10,906 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.84 vs. limit=15.0 2023-11-22 14:56:25,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1987260.0, ans=0.125 2023-11-22 14:56:35,396 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298100 2023-11-22 14:56:45,216 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.353e+01 9.126e+01 1.009e+02 1.546e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-22 14:56:50,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.35 vs. limit=15.0 2023-11-22 14:57:02,716 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9550, loss[loss=0.06732, simple_loss=0.08286, pruned_loss=0.01467, audio_tagging_loss=0.01121, over 14486.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09421, pruned_loss=0.01507, audio_tagging_loss=0.009524, over 3046319.13 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 14:57:03,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1987460.0, ans=0.07 2023-11-22 14:57:08,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1987460.0, ans=0.0 2023-11-22 14:57:14,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1987460.0, ans=0.0 2023-11-22 14:57:27,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1987593.3333333333, ans=0.1 2023-11-22 14:57:38,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1987593.3333333333, ans=0.0 2023-11-22 14:57:41,036 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298150 2023-11-22 14:58:09,004 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9600, loss[loss=0.06496, simple_loss=0.08331, pruned_loss=0.01219, audio_tagging_loss=0.01112, over 14777.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09422, pruned_loss=0.01493, audio_tagging_loss=0.009547, over 3053217.52 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 14:58:09,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1987793.3333333333, ans=0.1 2023-11-22 14:58:34,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1987926.6666666667, ans=0.07 2023-11-22 14:58:38,730 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=15.0 2023-11-22 14:58:46,270 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298200 2023-11-22 14:58:46,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1987993.3333333333, ans=0.0 2023-11-22 14:58:57,433 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.183e+01 8.755e+01 9.694e+01 2.318e+02, threshold=1.751e+02, percent-clipped=1.0 2023-11-22 14:59:13,481 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9650, loss[loss=0.04356, simple_loss=0.0535, pruned_loss=0.006801, audio_tagging_loss=0.01001, over 14182.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09411, pruned_loss=0.01488, audio_tagging_loss=0.009524, over 3047269.79 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 14:59:18,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0 2023-11-22 14:59:38,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1988260.0, ans=0.125 2023-11-22 14:59:42,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1988260.0, ans=0.1 2023-11-22 14:59:45,213 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.93 vs. limit=15.0 2023-11-22 14:59:51,324 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298250 2023-11-22 14:59:56,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1988326.6666666667, ans=0.125 2023-11-22 14:59:57,555 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 14:59:58,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1988326.6666666667, ans=10.0 2023-11-22 15:00:17,258 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9700, loss[loss=0.08104, simple_loss=0.1062, pruned_loss=0.01686, audio_tagging_loss=0.0111, over 15217.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09444, pruned_loss=0.01494, audio_tagging_loss=0.009376, over 3043808.73 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:00:38,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1988526.6666666667, ans=0.125 2023-11-22 15:00:49,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1988593.3333333333, ans=0.1 2023-11-22 15:00:51,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1988593.3333333333, ans=0.1 2023-11-22 15:00:55,308 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298300 2023-11-22 15:01:04,934 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.285e+01 8.809e+01 9.678e+01 1.163e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 15:01:09,446 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.83 vs. limit=15.0 2023-11-22 15:01:21,874 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9750, loss[loss=0.07728, simple_loss=0.1069, pruned_loss=0.01552, audio_tagging_loss=0.008306, over 14997.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09415, pruned_loss=0.01484, audio_tagging_loss=0.00933, over 3043583.96 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:01:31,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.70 vs. limit=22.5 2023-11-22 15:01:40,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1988860.0, ans=0.1 2023-11-22 15:01:45,563 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5 2023-11-22 15:01:52,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1988926.6666666667, ans=0.1 2023-11-22 15:01:58,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298350 2023-11-22 15:02:00,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1988993.3333333333, ans=0.125 2023-11-22 15:02:25,382 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9800, loss[loss=0.05098, simple_loss=0.05705, pruned_loss=0.01075, audio_tagging_loss=0.01171, over 16051.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09417, pruned_loss=0.01488, audio_tagging_loss=0.00924, over 3044760.64 frames. ], batch size: 62, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:02:28,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1989126.6666666667, ans=0.125 2023-11-22 15:02:32,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1989126.6666666667, ans=0.125 2023-11-22 15:02:38,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1989193.3333333333, ans=0.0 2023-11-22 15:02:39,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1989193.3333333333, ans=0.125 2023-11-22 15:02:43,173 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 15:03:02,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298400 2023-11-22 15:03:08,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1989326.6666666667, ans=0.125 2023-11-22 15:03:14,269 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.717e+01 7.942e+01 8.704e+01 9.561e+01 1.255e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-22 15:03:20,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1989393.3333333333, ans=0.2 2023-11-22 15:03:23,533 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 15:03:26,724 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.21 vs. limit=15.0 2023-11-22 15:03:29,596 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9850, loss[loss=0.07021, simple_loss=0.08724, pruned_loss=0.01686, audio_tagging_loss=0.009722, over 15443.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09443, pruned_loss=0.01493, audio_tagging_loss=0.009081, over 3048639.45 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:03:41,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1989526.6666666667, ans=0.2 2023-11-22 15:03:43,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff2.min_abs, batch_count=1989526.6666666667, ans=0.1 2023-11-22 15:03:46,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=1989526.6666666667, ans=0.025 2023-11-22 15:03:51,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1989526.6666666667, ans=0.125 2023-11-22 15:04:06,718 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298450 2023-11-22 15:04:08,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1989660.0, ans=0.125 2023-11-22 15:04:28,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1989726.6666666667, ans=0.125 2023-11-22 15:04:33,835 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9900, loss[loss=0.06931, simple_loss=0.1033, pruned_loss=0.01042, audio_tagging_loss=0.007232, over 15418.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.0947, pruned_loss=0.01495, audio_tagging_loss=0.008997, over 3051250.06 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:04:45,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1989860.0, ans=0.125 2023-11-22 15:05:10,768 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298500 2023-11-22 15:05:23,016 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.052e+01 8.333e+01 9.011e+01 9.582e+01 1.423e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-22 15:05:34,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1990060.0, ans=0.1 2023-11-22 15:05:37,734 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 9950, loss[loss=0.06901, simple_loss=0.09487, pruned_loss=0.01613, audio_tagging_loss=0.005443, over 14724.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09499, pruned_loss=0.0152, audio_tagging_loss=0.009083, over 3050694.30 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:05:51,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1990193.3333333333, ans=0.09899494936611666 2023-11-22 15:06:12,141 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.38 vs. limit=10.0 2023-11-22 15:06:13,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1990260.0, ans=0.125 2023-11-22 15:06:13,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1990260.0, ans=0.0 2023-11-22 15:06:15,364 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298550 2023-11-22 15:06:41,340 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.33 vs. limit=15.0 2023-11-22 15:06:42,071 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10000, loss[loss=0.06, simple_loss=0.08177, pruned_loss=0.01014, audio_tagging_loss=0.008981, over 14780.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09487, pruned_loss=0.01522, audio_tagging_loss=0.009037, over 3048580.55 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:06:44,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1990460.0, ans=0.125 2023-11-22 15:06:47,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1990460.0, ans=0.125 2023-11-22 15:06:55,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1990526.6666666667, ans=0.125 2023-11-22 15:07:00,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1990526.6666666667, ans=0.125 2023-11-22 15:07:09,711 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.08 vs. limit=6.0 2023-11-22 15:07:14,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1990593.3333333333, ans=0.0 2023-11-22 15:07:19,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298600 2023-11-22 15:07:25,899 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.84 vs. limit=15.0 2023-11-22 15:07:31,287 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.143e+01 8.830e+01 9.648e+01 3.146e+02, threshold=1.766e+02, percent-clipped=1.0 2023-11-22 15:07:42,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1990726.6666666667, ans=0.125 2023-11-22 15:07:47,382 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10050, loss[loss=0.0593, simple_loss=0.08582, pruned_loss=0.008252, audio_tagging_loss=0.008139, over 14515.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09465, pruned_loss=0.0151, audio_tagging_loss=0.009122, over 3054513.08 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:07:56,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1990793.3333333333, ans=0.125 2023-11-22 15:08:06,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1990860.0, ans=0.125 2023-11-22 15:08:08,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1990860.0, ans=0.125 2023-11-22 15:08:12,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1990926.6666666667, ans=0.2 2023-11-22 15:08:24,244 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298650 2023-11-22 15:08:25,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0 2023-11-22 15:08:30,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.18 vs. limit=15.0 2023-11-22 15:08:36,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1990993.3333333333, ans=0.0 2023-11-22 15:08:51,070 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10100, loss[loss=0.06216, simple_loss=0.08186, pruned_loss=0.01134, audio_tagging_loss=0.009892, over 14538.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09387, pruned_loss=0.01503, audio_tagging_loss=0.009176, over 3049122.81 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:09:28,306 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298700 2023-11-22 15:09:29,049 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0 2023-11-22 15:09:34,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1991326.6666666667, ans=0.2 2023-11-22 15:09:39,087 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.326e+01 8.430e+01 8.908e+01 9.951e+01 1.277e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-22 15:09:41,547 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 15:09:54,826 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10150, loss[loss=0.07748, simple_loss=0.1011, pruned_loss=0.01748, audio_tagging_loss=0.009426, over 13334.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09427, pruned_loss=0.01514, audio_tagging_loss=0.00931, over 3052177.28 frames. ], batch size: 52, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:10:24,415 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 15:10:28,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1991593.3333333333, ans=0.04949747468305833 2023-11-22 15:10:29,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1991593.3333333333, ans=0.2 2023-11-22 15:10:29,572 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 15:10:31,742 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298750 2023-11-22 15:10:58,812 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10200, loss[loss=0.06051, simple_loss=0.06934, pruned_loss=0.01434, audio_tagging_loss=0.0115, over 14740.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09352, pruned_loss=0.01508, audio_tagging_loss=0.009412, over 3058668.89 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:10:59,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1991793.3333333333, ans=0.1 2023-11-22 15:11:21,508 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 15:11:35,987 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298800 2023-11-22 15:11:47,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1991993.3333333333, ans=0.2 2023-11-22 15:11:48,670 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.315e+01 8.823e+01 9.660e+01 1.198e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-22 15:12:02,812 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10250, loss[loss=0.05313, simple_loss=0.0611, pruned_loss=0.00959, audio_tagging_loss=0.01299, over 15526.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09333, pruned_loss=0.01491, audio_tagging_loss=0.009505, over 3055571.44 frames. ], batch size: 61, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:12:32,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1992260.0, ans=0.0 2023-11-22 15:12:33,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1992260.0, ans=0.0 2023-11-22 15:12:36,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1992260.0, ans=0.0 2023-11-22 15:12:40,226 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298850 2023-11-22 15:12:40,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1992326.6666666667, ans=0.0 2023-11-22 15:12:40,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1992326.6666666667, ans=0.0 2023-11-22 15:13:04,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.02 vs. limit=6.0 2023-11-22 15:13:06,226 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10300, loss[loss=0.08591, simple_loss=0.1155, pruned_loss=0.01985, audio_tagging_loss=0.008309, over 15045.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09324, pruned_loss=0.01508, audio_tagging_loss=0.009548, over 3058382.44 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:13:09,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1992460.0, ans=0.2 2023-11-22 15:13:31,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1992593.3333333333, ans=0.1 2023-11-22 15:13:32,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1992593.3333333333, ans=0.125 2023-11-22 15:13:43,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298900 2023-11-22 15:13:45,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1992660.0, ans=0.0 2023-11-22 15:13:55,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1992660.0, ans=0.1 2023-11-22 15:13:55,927 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.205e+01 8.813e+01 9.475e+01 1.270e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-22 15:13:59,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1992726.6666666667, ans=0.1 2023-11-22 15:14:04,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1992726.6666666667, ans=0.125 2023-11-22 15:14:10,541 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10350, loss[loss=0.07468, simple_loss=0.1106, pruned_loss=0.01397, audio_tagging_loss=0.005413, over 14793.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09424, pruned_loss=0.01525, audio_tagging_loss=0.009599, over 3052112.35 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:14:41,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1992926.6666666667, ans=0.025 2023-11-22 15:14:47,109 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 298950 2023-11-22 15:14:49,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1992993.3333333333, ans=0.125 2023-11-22 15:15:09,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1993060.0, ans=0.0 2023-11-22 15:15:14,424 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10400, loss[loss=0.0644, simple_loss=0.07847, pruned_loss=0.01666, audio_tagging_loss=0.008506, over 13948.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09368, pruned_loss=0.01508, audio_tagging_loss=0.009622, over 3047500.94 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:15:23,394 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 15:15:24,925 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.14 vs. limit=6.0 2023-11-22 15:15:25,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1993193.3333333333, ans=0.1 2023-11-22 15:15:34,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1993193.3333333333, ans=0.125 2023-11-22 15:15:48,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.95 vs. limit=15.0 2023-11-22 15:15:51,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299000 2023-11-22 15:15:59,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1993326.6666666667, ans=0.125 2023-11-22 15:16:05,354 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.066e+01 8.819e+01 9.372e+01 1.193e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 15:16:18,115 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10450, loss[loss=0.06487, simple_loss=0.07818, pruned_loss=0.01528, audio_tagging_loss=0.01049, over 14771.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09309, pruned_loss=0.01483, audio_tagging_loss=0.009521, over 3048087.56 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:16:28,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1993460.0, ans=0.1 2023-11-22 15:16:50,898 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.86 vs. limit=15.0 2023-11-22 15:16:55,359 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299050 2023-11-22 15:17:15,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1993726.6666666667, ans=0.0 2023-11-22 15:17:21,638 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10500, loss[loss=0.0581, simple_loss=0.07267, pruned_loss=0.009871, audio_tagging_loss=0.01189, over 14787.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09276, pruned_loss=0.01493, audio_tagging_loss=0.009453, over 3045880.38 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:17:28,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1993793.3333333333, ans=0.125 2023-11-22 15:17:33,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=11.47 vs. limit=15.0 2023-11-22 15:17:39,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.46 vs. limit=15.0 2023-11-22 15:17:44,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1993860.0, ans=0.125 2023-11-22 15:17:53,375 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0 2023-11-22 15:17:58,762 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299100 2023-11-22 15:18:13,204 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.803e+01 8.251e+01 8.969e+01 9.494e+01 1.165e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-22 15:18:18,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1994060.0, ans=0.1 2023-11-22 15:18:25,937 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10550, loss[loss=0.07616, simple_loss=0.0959, pruned_loss=0.0185, audio_tagging_loss=0.009701, over 14935.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.0928, pruned_loss=0.01505, audio_tagging_loss=0.009281, over 3049373.89 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:18:35,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.43 vs. limit=22.5 2023-11-22 15:19:03,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299150 2023-11-22 15:19:04,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1994326.6666666667, ans=0.125 2023-11-22 15:19:07,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1994326.6666666667, ans=0.2 2023-11-22 15:19:08,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1994326.6666666667, ans=0.05 2023-11-22 15:19:10,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1994326.6666666667, ans=0.125 2023-11-22 15:19:10,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1994326.6666666667, ans=0.0 2023-11-22 15:19:13,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1994326.6666666667, ans=0.0 2023-11-22 15:19:29,000 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10600, loss[loss=0.05432, simple_loss=0.06705, pruned_loss=0.01128, audio_tagging_loss=0.00952, over 14320.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09195, pruned_loss=0.0147, audio_tagging_loss=0.009253, over 3045578.04 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:19:43,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.19 vs. limit=12.0 2023-11-22 15:19:56,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1994593.3333333333, ans=0.1 2023-11-22 15:20:06,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299200 2023-11-22 15:20:11,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1994660.0, ans=0.125 2023-11-22 15:20:16,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1994660.0, ans=0.125 2023-11-22 15:20:20,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.21 vs. limit=15.0 2023-11-22 15:20:20,907 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.207e+01 8.203e+01 8.764e+01 9.272e+01 1.300e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-22 15:20:26,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1994726.6666666667, ans=0.0 2023-11-22 15:20:28,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1994726.6666666667, ans=0.1 2023-11-22 15:20:32,956 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10650, loss[loss=0.07787, simple_loss=0.1103, pruned_loss=0.01547, audio_tagging_loss=0.007277, over 15100.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09171, pruned_loss=0.01451, audio_tagging_loss=0.009193, over 3048585.98 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:20:44,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1994793.3333333333, ans=0.05 2023-11-22 15:20:50,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1994860.0, ans=0.1 2023-11-22 15:21:04,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1994926.6666666667, ans=0.125 2023-11-22 15:21:05,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1994926.6666666667, ans=0.125 2023-11-22 15:21:05,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1994926.6666666667, ans=0.95 2023-11-22 15:21:10,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299250 2023-11-22 15:21:19,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1994993.3333333333, ans=0.1 2023-11-22 15:21:34,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.10 vs. limit=15.0 2023-11-22 15:21:36,088 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.25 vs. limit=15.0 2023-11-22 15:21:37,635 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10700, loss[loss=0.1025, simple_loss=0.1458, pruned_loss=0.02234, audio_tagging_loss=0.00721, over 16135.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09218, pruned_loss=0.01454, audio_tagging_loss=0.009244, over 3048706.42 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:21:47,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1995126.6666666667, ans=0.125 2023-11-22 15:22:03,105 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.69 vs. limit=15.0 2023-11-22 15:22:05,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1995260.0, ans=0.0 2023-11-22 15:22:07,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1995260.0, ans=0.0 2023-11-22 15:22:12,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1995260.0, ans=0.0 2023-11-22 15:22:14,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299300 2023-11-22 15:22:14,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1995326.6666666667, ans=0.1 2023-11-22 15:22:25,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1995326.6666666667, ans=0.1 2023-11-22 15:22:28,551 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.250e+01 8.791e+01 9.452e+01 1.180e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 15:22:34,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1995393.3333333333, ans=0.2 2023-11-22 15:22:36,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.01 vs. limit=15.0 2023-11-22 15:22:40,689 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10750, loss[loss=0.06485, simple_loss=0.08518, pruned_loss=0.01211, audio_tagging_loss=0.01016, over 14505.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09298, pruned_loss=0.01473, audio_tagging_loss=0.009191, over 3049320.93 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:22:54,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1995526.6666666667, ans=10.0 2023-11-22 15:22:54,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1995526.6666666667, ans=0.125 2023-11-22 15:23:08,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1995593.3333333333, ans=0.125 2023-11-22 15:23:18,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299350 2023-11-22 15:23:19,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1995660.0, ans=0.0 2023-11-22 15:23:23,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1995660.0, ans=0.2 2023-11-22 15:23:33,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1995726.6666666667, ans=0.125 2023-11-22 15:23:39,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1995726.6666666667, ans=0.0 2023-11-22 15:23:44,109 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10800, loss[loss=0.08183, simple_loss=0.1086, pruned_loss=0.01903, audio_tagging_loss=0.008484, over 14738.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09259, pruned_loss=0.01454, audio_tagging_loss=0.009205, over 3049362.24 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0 2023-11-22 15:23:52,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1995793.3333333333, ans=0.125 2023-11-22 15:23:59,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1995860.0, ans=0.0 2023-11-22 15:24:22,206 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299400 2023-11-22 15:24:36,980 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.241e+01 8.735e+01 9.368e+01 1.325e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 15:24:42,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1996060.0, ans=0.125 2023-11-22 15:24:45,467 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.42 vs. limit=15.0 2023-11-22 15:24:48,814 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.56 vs. limit=10.0 2023-11-22 15:24:49,268 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10850, loss[loss=0.07273, simple_loss=0.09248, pruned_loss=0.01665, audio_tagging_loss=0.009838, over 14871.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09273, pruned_loss=0.0146, audio_tagging_loss=0.009172, over 3044860.39 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0 2023-11-22 15:25:20,263 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0 2023-11-22 15:25:21,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1996260.0, ans=0.07 2023-11-22 15:25:25,934 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299450 2023-11-22 15:25:41,419 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.80 vs. limit=15.0 2023-11-22 15:25:43,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1996393.3333333333, ans=0.0 2023-11-22 15:25:49,802 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 15:25:53,363 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10900, loss[loss=0.07724, simple_loss=0.1075, pruned_loss=0.01466, audio_tagging_loss=0.008825, over 15606.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09412, pruned_loss=0.01483, audio_tagging_loss=0.009154, over 3042589.14 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:25:53,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1996460.0, ans=0.125 2023-11-22 15:25:54,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1996460.0, ans=0.1 2023-11-22 15:26:05,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1996526.6666666667, ans=0.0 2023-11-22 15:26:22,148 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.38 vs. limit=10.0 2023-11-22 15:26:27,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1996593.3333333333, ans=0.125 2023-11-22 15:26:30,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299500 2023-11-22 15:26:45,543 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.01 vs. limit=15.0 2023-11-22 15:26:45,939 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.150e+01 8.424e+01 9.041e+01 9.724e+01 1.556e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-22 15:26:57,230 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 10950, loss[loss=0.06194, simple_loss=0.08733, pruned_loss=0.008752, audio_tagging_loss=0.009524, over 15175.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.0932, pruned_loss=0.01449, audio_tagging_loss=0.009186, over 3048650.67 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:27:01,525 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.64 vs. limit=22.5 2023-11-22 15:27:13,493 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0 2023-11-22 15:27:31,881 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 15:27:34,161 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299550 2023-11-22 15:27:53,680 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.19 vs. limit=22.5 2023-11-22 15:27:56,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1997060.0, ans=0.1 2023-11-22 15:28:01,244 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11000, loss[loss=0.04991, simple_loss=0.06506, pruned_loss=0.00697, audio_tagging_loss=0.01041, over 13917.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09353, pruned_loss=0.01465, audio_tagging_loss=0.009299, over 3050712.05 frames. ], batch size: 55, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:28:02,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1997126.6666666667, ans=0.2 2023-11-22 15:28:06,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1997126.6666666667, ans=0.2 2023-11-22 15:28:11,053 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 15:28:16,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1997193.3333333333, ans=0.0 2023-11-22 15:28:22,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1997193.3333333333, ans=0.0 2023-11-22 15:28:33,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1997260.0, ans=0.1 2023-11-22 15:28:37,965 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299600 2023-11-22 15:28:53,981 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.275e+01 8.821e+01 9.536e+01 1.341e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 15:28:56,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1997393.3333333333, ans=0.2 2023-11-22 15:28:58,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1997393.3333333333, ans=0.5 2023-11-22 15:29:05,401 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11050, loss[loss=0.08255, simple_loss=0.1138, pruned_loss=0.01699, audio_tagging_loss=0.008674, over 15066.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09387, pruned_loss=0.01477, audio_tagging_loss=0.009438, over 3051555.76 frames. ], batch size: 54, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:29:25,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1997526.6666666667, ans=0.0 2023-11-22 15:29:34,653 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 15:29:42,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299650 2023-11-22 15:29:59,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1997726.6666666667, ans=0.2 2023-11-22 15:30:01,557 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.29 vs. limit=6.0 2023-11-22 15:30:06,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1997726.6666666667, ans=0.2 2023-11-22 15:30:08,999 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11100, loss[loss=0.07412, simple_loss=0.09939, pruned_loss=0.01606, audio_tagging_loss=0.00837, over 14692.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09388, pruned_loss=0.01489, audio_tagging_loss=0.00959, over 3048218.82 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:30:27,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1997860.0, ans=0.5 2023-11-22 15:30:45,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1997926.6666666667, ans=0.1 2023-11-22 15:30:45,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299700 2023-11-22 15:31:01,212 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.427e+01 8.934e+01 9.666e+01 1.581e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-22 15:31:12,922 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11150, loss[loss=0.06482, simple_loss=0.081, pruned_loss=0.01313, audio_tagging_loss=0.01119, over 15005.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09411, pruned_loss=0.01493, audio_tagging_loss=0.009582, over 3053284.55 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:31:13,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1998126.6666666667, ans=0.125 2023-11-22 15:31:20,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1998126.6666666667, ans=0.0 2023-11-22 15:31:23,933 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.35 vs. limit=15.0 2023-11-22 15:31:42,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1998260.0, ans=0.125 2023-11-22 15:31:50,003 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299750 2023-11-22 15:31:53,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.07 vs. limit=15.0 2023-11-22 15:32:16,645 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11200, loss[loss=0.07426, simple_loss=0.09555, pruned_loss=0.01386, audio_tagging_loss=0.01263, over 16611.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09357, pruned_loss=0.01474, audio_tagging_loss=0.009727, over 3047243.52 frames. ], batch size: 64, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:32:17,357 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.64 vs. limit=10.0 2023-11-22 15:32:21,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1998460.0, ans=0.1 2023-11-22 15:32:23,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1998460.0, ans=0.2 2023-11-22 15:32:23,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1998460.0, ans=0.2 2023-11-22 15:32:53,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1998593.3333333333, ans=0.0 2023-11-22 15:32:54,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299800 2023-11-22 15:32:58,039 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.65 vs. limit=15.0 2023-11-22 15:33:09,683 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.174e+01 8.782e+01 9.719e+01 1.179e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 15:33:22,088 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11250, loss[loss=0.08112, simple_loss=0.1026, pruned_loss=0.02074, audio_tagging_loss=0.009078, over 15326.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09281, pruned_loss=0.01458, audio_tagging_loss=0.009681, over 3048945.05 frames. ], batch size: 59, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:33:25,064 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.23 vs. limit=6.0 2023-11-22 15:33:28,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1998793.3333333333, ans=0.125 2023-11-22 15:33:36,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1998860.0, ans=0.0 2023-11-22 15:33:37,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1998860.0, ans=0.1 2023-11-22 15:33:45,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1998860.0, ans=0.0 2023-11-22 15:33:56,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1998926.6666666667, ans=0.0 2023-11-22 15:33:58,928 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299850 2023-11-22 15:33:59,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1998993.3333333333, ans=0.125 2023-11-22 15:34:14,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1999060.0, ans=0.0 2023-11-22 15:34:22,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1999060.0, ans=0.025 2023-11-22 15:34:25,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1999126.6666666667, ans=0.1 2023-11-22 15:34:26,193 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11300, loss[loss=0.06389, simple_loss=0.0814, pruned_loss=0.01107, audio_tagging_loss=0.01213, over 14657.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09333, pruned_loss=0.01467, audio_tagging_loss=0.009447, over 3047798.00 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:35:00,143 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.78 vs. limit=15.0 2023-11-22 15:35:03,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299900 2023-11-22 15:35:18,549 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.156e+01 8.491e+01 9.004e+01 9.715e+01 1.251e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-22 15:35:29,977 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11350, loss[loss=0.06125, simple_loss=0.07196, pruned_loss=0.01246, audio_tagging_loss=0.01282, over 14409.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09317, pruned_loss=0.0147, audio_tagging_loss=0.00937, over 3047148.37 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:35:36,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1999460.0, ans=0.125 2023-11-22 15:35:42,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1999526.6666666667, ans=0.0 2023-11-22 15:35:52,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.61 vs. limit=10.0 2023-11-22 15:36:07,127 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 299950 2023-11-22 15:36:11,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1999660.0, ans=0.0 2023-11-22 15:36:33,728 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11400, loss[loss=0.05621, simple_loss=0.0811, pruned_loss=0.008573, audio_tagging_loss=0.007086, over 15408.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09388, pruned_loss=0.01482, audio_tagging_loss=0.009157, over 3040366.28 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:36:47,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1999860.0, ans=0.015 2023-11-22 15:36:55,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1999860.0, ans=0.125 2023-11-22 15:37:10,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300000 2023-11-22 15:37:27,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2000060.0, ans=0.125 2023-11-22 15:37:27,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2000060.0, ans=0.5 2023-11-22 15:37:31,200 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.071e+01 8.858e+01 9.602e+01 1.299e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-22 15:37:38,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0 2023-11-22 15:37:41,010 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11450, loss[loss=0.07522, simple_loss=0.09474, pruned_loss=0.01696, audio_tagging_loss=0.01089, over 15575.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09352, pruned_loss=0.01477, audio_tagging_loss=0.009128, over 3042588.20 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:37:42,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2000126.6666666667, ans=0.125 2023-11-22 15:37:47,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2000126.6666666667, ans=0.2 2023-11-22 15:38:03,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2000193.3333333333, ans=0.125 2023-11-22 15:38:07,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2000260.0, ans=0.1 2023-11-22 15:38:12,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2000260.0, ans=0.125 2023-11-22 15:38:16,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.90 vs. limit=22.5 2023-11-22 15:38:18,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300050 2023-11-22 15:38:26,165 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0 2023-11-22 15:38:31,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2000393.3333333333, ans=0.0 2023-11-22 15:38:33,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2000393.3333333333, ans=0.1 2023-11-22 15:38:40,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.71 vs. limit=12.0 2023-11-22 15:38:44,821 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11500, loss[loss=0.05517, simple_loss=0.07791, pruned_loss=0.006532, audio_tagging_loss=0.009679, over 14425.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09234, pruned_loss=0.01454, audio_tagging_loss=0.009244, over 3037099.19 frames. ], batch size: 54, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:38:45,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2000460.0, ans=0.125 2023-11-22 15:39:10,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2000593.3333333333, ans=0.125 2023-11-22 15:39:15,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2000593.3333333333, ans=0.125 2023-11-22 15:39:21,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300100 2023-11-22 15:39:38,052 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.223e+01 8.870e+01 9.437e+01 1.249e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 15:39:43,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2000726.6666666667, ans=0.0 2023-11-22 15:39:47,762 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11550, loss[loss=0.08894, simple_loss=0.1203, pruned_loss=0.02362, audio_tagging_loss=0.005174, over 14428.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09286, pruned_loss=0.0147, audio_tagging_loss=0.009248, over 3035836.27 frames. ], batch size: 52, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:39:48,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2000793.3333333333, ans=0.0 2023-11-22 15:40:25,321 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300150 2023-11-22 15:40:27,688 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 15:40:29,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2000993.3333333333, ans=0.0 2023-11-22 15:40:32,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2000993.3333333333, ans=0.125 2023-11-22 15:40:36,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2000993.3333333333, ans=0.09899494936611666 2023-11-22 15:40:52,022 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11600, loss[loss=0.07057, simple_loss=0.0964, pruned_loss=0.01325, audio_tagging_loss=0.009124, over 13864.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09386, pruned_loss=0.01489, audio_tagging_loss=0.009197, over 3035572.90 frames. ], batch size: 53, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:41:29,216 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300200 2023-11-22 15:41:36,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2001326.6666666667, ans=0.1 2023-11-22 15:41:46,186 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.408e+01 8.800e+01 9.381e+01 1.504e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 15:41:46,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2001393.3333333333, ans=0.1 2023-11-22 15:41:53,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2001393.3333333333, ans=0.1 2023-11-22 15:41:56,491 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11650, loss[loss=0.06513, simple_loss=0.08354, pruned_loss=0.0122, audio_tagging_loss=0.01116, over 15909.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09379, pruned_loss=0.01493, audio_tagging_loss=0.009217, over 3038049.83 frames. ], batch size: 59, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:42:13,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2001526.6666666667, ans=0.125 2023-11-22 15:42:16,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2001526.6666666667, ans=0.125 2023-11-22 15:42:18,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2001526.6666666667, ans=0.125 2023-11-22 15:42:27,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2001593.3333333333, ans=0.1 2023-11-22 15:42:33,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300250 2023-11-22 15:42:44,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2001660.0, ans=0.0 2023-11-22 15:42:59,629 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11700, loss[loss=0.07805, simple_loss=0.1001, pruned_loss=0.01455, audio_tagging_loss=0.01343, over 16539.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09382, pruned_loss=0.01504, audio_tagging_loss=0.009256, over 3040358.24 frames. ], batch size: 64, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:43:05,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2001793.3333333333, ans=0.125 2023-11-22 15:43:10,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2001793.3333333333, ans=0.125 2023-11-22 15:43:11,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2001860.0, ans=0.0 2023-11-22 15:43:24,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2001926.6666666667, ans=0.125 2023-11-22 15:43:29,295 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=12.0 2023-11-22 15:43:37,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300300 2023-11-22 15:43:37,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2001993.3333333333, ans=0.125 2023-11-22 15:43:41,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2001993.3333333333, ans=0.0 2023-11-22 15:43:44,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2001993.3333333333, ans=0.125 2023-11-22 15:43:45,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2001993.3333333333, ans=0.125 2023-11-22 15:43:53,162 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.806e+01 8.322e+01 8.823e+01 9.489e+01 1.251e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-22 15:44:00,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2002060.0, ans=0.125 2023-11-22 15:44:03,420 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11750, loss[loss=0.06551, simple_loss=0.08531, pruned_loss=0.01304, audio_tagging_loss=0.009809, over 16204.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09399, pruned_loss=0.01507, audio_tagging_loss=0.00928, over 3040211.10 frames. ], batch size: 59, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:44:25,406 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.510e-03 2023-11-22 15:44:39,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300350 2023-11-22 15:44:46,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2002326.6666666667, ans=0.0 2023-11-22 15:44:49,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2002326.6666666667, ans=0.125 2023-11-22 15:45:07,028 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11800, loss[loss=0.05865, simple_loss=0.0784, pruned_loss=0.01144, audio_tagging_loss=0.008008, over 16271.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09378, pruned_loss=0.01493, audio_tagging_loss=0.009327, over 3037831.03 frames. ], batch size: 61, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:45:21,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2002526.6666666667, ans=0.125 2023-11-22 15:45:43,039 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300400 2023-11-22 15:46:01,595 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.204e+01 8.722e+01 9.366e+01 1.185e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-22 15:46:10,186 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11850, loss[loss=0.06726, simple_loss=0.07983, pruned_loss=0.01572, audio_tagging_loss=0.01163, over 16107.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09416, pruned_loss=0.01496, audio_tagging_loss=0.009386, over 3040282.71 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:46:16,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2002793.3333333333, ans=0.125 2023-11-22 15:46:19,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2002793.3333333333, ans=0.1 2023-11-22 15:46:22,320 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.60 vs. limit=22.5 2023-11-22 15:46:47,777 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300450 2023-11-22 15:46:50,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2002993.3333333333, ans=0.125 2023-11-22 15:46:54,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.02 vs. limit=15.0 2023-11-22 15:47:01,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2003060.0, ans=0.025 2023-11-22 15:47:13,754 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11900, loss[loss=0.07608, simple_loss=0.1014, pruned_loss=0.01585, audio_tagging_loss=0.009539, over 15047.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.0937, pruned_loss=0.01502, audio_tagging_loss=0.009581, over 3040124.69 frames. ], batch size: 55, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:47:21,809 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.52 vs. limit=15.0 2023-11-22 15:47:24,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2003126.6666666667, ans=0.0 2023-11-22 15:47:25,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2003193.3333333333, ans=0.0 2023-11-22 15:47:38,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2003260.0, ans=0.04949747468305833 2023-11-22 15:47:42,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.29 vs. limit=22.5 2023-11-22 15:47:50,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300500 2023-11-22 15:47:59,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2003326.6666666667, ans=0.05 2023-11-22 15:48:06,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2003393.3333333333, ans=0.0 2023-11-22 15:48:07,928 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.243e+01 8.822e+01 9.597e+01 1.131e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 15:48:17,683 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 11950, loss[loss=0.07079, simple_loss=0.09493, pruned_loss=0.01325, audio_tagging_loss=0.01008, over 14939.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09429, pruned_loss=0.01511, audio_tagging_loss=0.009554, over 3044339.24 frames. ], batch size: 59, lr: 2.70e-03, grad_scale: 16.0 2023-11-22 15:48:30,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2003526.6666666667, ans=0.125 2023-11-22 15:48:30,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2003526.6666666667, ans=0.05 2023-11-22 15:48:36,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2003526.6666666667, ans=15.0 2023-11-22 15:48:52,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2003593.3333333333, ans=0.2 2023-11-22 15:48:54,163 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300550 2023-11-22 15:49:02,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2003660.0, ans=0.125 2023-11-22 15:49:18,853 INFO [train_asr.py:1221] (3/4) Epoch 25, batch 12000, loss[loss=0.04542, simple_loss=0.04467, pruned_loss=0.006287, audio_tagging_loss=0.0168, over 14151.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09448, pruned_loss=0.01502, audio_tagging_loss=0.009686, over 3045660.96 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 32.0 2023-11-22 15:49:18,853 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 15:49:59,239 INFO [train_asr.py:1253] (3/4) Epoch 25, validation: loss=0.05961, simple_loss=0.05152, pruned_loss=0.005134, audio_tagging_loss=0.02872, over 4681554.00 frames. 2023-11-22 15:49:59,241 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 15:50:00,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2003793.3333333333, ans=0.1 2023-11-22 15:50:19,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2003860.0, ans=0.2 2023-11-22 15:50:20,136 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.23 vs. limit=12.0 2023-11-22 15:51:02,499 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 0, loss[loss=0.09881, simple_loss=0.1219, pruned_loss=0.02197, audio_tagging_loss=0.01591, over 15495.00 frames. ], tot_loss[loss=0.09881, simple_loss=0.1219, pruned_loss=0.02197, audio_tagging_loss=0.01591, over 15495.00 frames. ], batch size: 56, lr: 2.65e-03, grad_scale: 32.0 2023-11-22 15:51:02,500 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 15:51:37,678 INFO [train_asr.py:1253] (3/4) Epoch 26, validation: loss=0.05869, simple_loss=0.05153, pruned_loss=0.005094, audio_tagging_loss=0.02783, over 4681554.00 frames. 2023-11-22 15:51:37,679 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 15:51:38,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.whiten.whitening_limit, batch_count=2003960.0, ans=15.0 2023-11-22 15:51:43,328 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300600 2023-11-22 15:51:47,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2003960.0, ans=10.0 2023-11-22 15:51:48,089 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.49 vs. limit=15.0 2023-11-22 15:51:55,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2004026.6666666667, ans=0.125 2023-11-22 15:52:01,846 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.448e+01 9.337e+01 1.025e+02 1.392e+02, threshold=1.867e+02, percent-clipped=0.0 2023-11-22 15:52:36,027 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.96 vs. limit=12.0 2023-11-22 15:52:43,336 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 50, loss[loss=0.06942, simple_loss=0.08336, pruned_loss=0.01097, audio_tagging_loss=0.01677, over 14626.00 frames. ], tot_loss[loss=0.07775, simple_loss=0.09076, pruned_loss=0.01423, audio_tagging_loss=0.01814, over 685013.32 frames. ], batch size: 55, lr: 2.65e-03, grad_scale: 32.0 2023-11-22 15:52:48,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300650 2023-11-22 15:53:22,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2004493.3333333333, ans=0.125 2023-11-22 15:53:48,097 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 100, loss[loss=0.07261, simple_loss=0.08634, pruned_loss=0.01281, audio_tagging_loss=0.01663, over 14255.00 frames. ], tot_loss[loss=0.07914, simple_loss=0.094, pruned_loss=0.01498, audio_tagging_loss=0.01716, over 1209987.81 frames. ], batch size: 57, lr: 2.65e-03, grad_scale: 32.0 2023-11-22 15:53:52,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2004626.6666666667, ans=0.2 2023-11-22 15:53:53,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300700 2023-11-22 15:54:03,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2004693.3333333333, ans=0.0 2023-11-22 15:54:11,742 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.950e+01 8.718e+01 9.300e+01 1.020e+02 1.184e+02, threshold=1.860e+02, percent-clipped=0.0 2023-11-22 15:54:16,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2004760.0, ans=0.125 2023-11-22 15:54:22,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2004760.0, ans=0.125 2023-11-22 15:54:49,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2004893.3333333333, ans=0.1 2023-11-22 15:54:53,102 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 150, loss[loss=0.1068, simple_loss=0.1351, pruned_loss=0.02675, audio_tagging_loss=0.01247, over 15314.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09427, pruned_loss=0.0149, audio_tagging_loss=0.01534, over 1618387.49 frames. ], batch size: 55, lr: 2.65e-03, grad_scale: 32.0 2023-11-22 15:54:58,098 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300750 2023-11-22 15:55:05,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2005026.6666666667, ans=0.0 2023-11-22 15:55:11,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.22 vs. limit=15.0 2023-11-22 15:55:12,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2005026.6666666667, ans=0.125 2023-11-22 15:55:30,687 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0 2023-11-22 15:55:43,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.05 vs. limit=15.0 2023-11-22 15:55:57,041 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 200, loss[loss=0.07692, simple_loss=0.09853, pruned_loss=0.01584, audio_tagging_loss=0.01181, over 15192.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09491, pruned_loss=0.01504, audio_tagging_loss=0.01356, over 1927775.55 frames. ], batch size: 58, lr: 2.65e-03, grad_scale: 32.0 2023-11-22 15:56:02,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300800 2023-11-22 15:56:20,096 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.352e+01 8.945e+01 1.004e+02 1.668e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-22 15:56:39,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2005493.3333333333, ans=0.0 2023-11-22 15:56:44,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.14 vs. limit=15.0 2023-11-22 15:57:01,857 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 250, loss[loss=0.0599, simple_loss=0.07623, pruned_loss=0.01122, audio_tagging_loss=0.01056, over 15247.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09497, pruned_loss=0.01534, audio_tagging_loss=0.01228, over 2177601.41 frames. ], batch size: 58, lr: 2.65e-03, grad_scale: 32.0 2023-11-22 15:57:06,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300850 2023-11-22 15:57:13,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2005693.3333333333, ans=10.0 2023-11-22 15:57:14,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.94 vs. limit=15.0 2023-11-22 15:57:21,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2005693.3333333333, ans=0.07 2023-11-22 15:57:22,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2005693.3333333333, ans=0.125 2023-11-22 15:57:45,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2005826.6666666667, ans=0.125 2023-11-22 15:57:55,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.26 vs. limit=22.5 2023-11-22 15:58:07,212 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 300, loss[loss=0.0612, simple_loss=0.08192, pruned_loss=0.01177, audio_tagging_loss=0.008474, over 15699.00 frames. ], tot_loss[loss=0.07519, simple_loss=0.0965, pruned_loss=0.01564, audio_tagging_loss=0.01129, over 2372668.41 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 15:58:12,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300900 2023-11-22 15:58:30,065 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.299e+01 8.395e+01 9.138e+01 9.781e+01 1.359e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-22 15:58:35,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2006093.3333333333, ans=0.125 2023-11-22 15:58:39,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2006093.3333333333, ans=0.125 2023-11-22 15:58:42,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2006093.3333333333, ans=0.125 2023-11-22 15:58:46,809 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.84 vs. limit=6.0 2023-11-22 15:58:51,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2006160.0, ans=0.1 2023-11-22 15:58:51,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2006160.0, ans=0.125 2023-11-22 15:59:12,892 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 350, loss[loss=0.08447, simple_loss=0.1186, pruned_loss=0.01939, audio_tagging_loss=0.005757, over 15392.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09609, pruned_loss=0.01534, audio_tagging_loss=0.01062, over 2523491.29 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 15:59:18,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 300950 2023-11-22 15:59:33,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.67 vs. limit=22.5 2023-11-22 15:59:52,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2006493.3333333333, ans=0.0 2023-11-22 16:00:11,885 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:00:17,754 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 400, loss[loss=0.05723, simple_loss=0.06831, pruned_loss=0.0138, audio_tagging_loss=0.00928, over 13799.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09524, pruned_loss=0.0152, audio_tagging_loss=0.01024, over 2632872.53 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:00:20,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2006626.6666666667, ans=0.125 2023-11-22 16:00:22,279 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.64 vs. limit=15.0 2023-11-22 16:00:23,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301000 2023-11-22 16:00:42,278 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.126e+01 8.693e+01 9.257e+01 1.166e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 16:01:00,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2006826.6666666667, ans=0.07 2023-11-22 16:01:01,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2006826.6666666667, ans=0.125 2023-11-22 16:01:23,356 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 450, loss[loss=0.08077, simple_loss=0.1105, pruned_loss=0.01907, audio_tagging_loss=0.006463, over 15352.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09475, pruned_loss=0.01504, audio_tagging_loss=0.009986, over 2719621.89 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:01:29,091 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301050 2023-11-22 16:01:48,763 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.53 vs. limit=22.5 2023-11-22 16:01:53,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2007093.3333333333, ans=0.0 2023-11-22 16:02:02,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.65 vs. limit=15.0 2023-11-22 16:02:20,486 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.66 vs. limit=15.0 2023-11-22 16:02:28,745 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 500, loss[loss=0.06419, simple_loss=0.08655, pruned_loss=0.01366, audio_tagging_loss=0.007254, over 15356.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09309, pruned_loss=0.01465, audio_tagging_loss=0.009877, over 2789428.06 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:02:29,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2007293.3333333333, ans=0.05 2023-11-22 16:02:31,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2007293.3333333333, ans=0.0 2023-11-22 16:02:33,736 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301100 2023-11-22 16:02:50,564 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.306e+01 8.908e+01 9.851e+01 1.431e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-22 16:03:32,321 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 550, loss[loss=0.06924, simple_loss=0.08813, pruned_loss=0.01433, audio_tagging_loss=0.01084, over 15945.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09325, pruned_loss=0.01474, audio_tagging_loss=0.009717, over 2835553.87 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:03:37,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301150 2023-11-22 16:03:44,661 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.67 vs. limit=6.0 2023-11-22 16:04:36,992 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 600, loss[loss=0.0659, simple_loss=0.08096, pruned_loss=0.01697, audio_tagging_loss=0.008451, over 15382.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09377, pruned_loss=0.01479, audio_tagging_loss=0.009554, over 2886801.25 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:04:42,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301200 2023-11-22 16:04:45,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2007960.0, ans=0.0 2023-11-22 16:04:52,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2008026.6666666667, ans=0.0 2023-11-22 16:05:01,750 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.281e+01 8.745e+01 9.297e+01 1.116e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 16:05:01,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2008093.3333333333, ans=0.125 2023-11-22 16:05:05,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.89 vs. limit=12.0 2023-11-22 16:05:27,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2008226.6666666667, ans=0.125 2023-11-22 16:05:35,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2008226.6666666667, ans=0.1 2023-11-22 16:05:38,095 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:05:42,008 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 650, loss[loss=0.05868, simple_loss=0.07999, pruned_loss=0.01041, audio_tagging_loss=0.008278, over 15649.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09296, pruned_loss=0.01464, audio_tagging_loss=0.009624, over 2917961.18 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:05:46,985 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301250 2023-11-22 16:05:49,937 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.80 vs. limit=15.0 2023-11-22 16:06:18,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2008426.6666666667, ans=0.1 2023-11-22 16:06:24,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2008493.3333333333, ans=0.125 2023-11-22 16:06:27,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2008493.3333333333, ans=0.0 2023-11-22 16:06:43,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2008560.0, ans=0.0 2023-11-22 16:06:45,895 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 700, loss[loss=0.05905, simple_loss=0.08232, pruned_loss=0.0105, audio_tagging_loss=0.007388, over 15967.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09269, pruned_loss=0.01462, audio_tagging_loss=0.009674, over 2953975.66 frames. ], batch size: 60, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:06:50,930 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301300 2023-11-22 16:07:11,109 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 8.183e+01 8.760e+01 9.542e+01 1.269e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-22 16:07:24,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2008826.6666666667, ans=0.125 2023-11-22 16:07:28,172 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:07:37,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2008893.3333333333, ans=0.125 2023-11-22 16:07:37,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2008893.3333333333, ans=0.0 2023-11-22 16:07:49,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.94 vs. limit=15.0 2023-11-22 16:07:49,631 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 750, loss[loss=0.06578, simple_loss=0.08702, pruned_loss=0.01232, audio_tagging_loss=0.009948, over 15142.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09286, pruned_loss=0.01472, audio_tagging_loss=0.009563, over 2974850.57 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:07:55,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301350 2023-11-22 16:08:03,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.60 vs. limit=8.0 2023-11-22 16:08:09,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2009026.6666666667, ans=0.125 2023-11-22 16:08:46,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2009226.6666666667, ans=0.125 2023-11-22 16:08:48,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2009226.6666666667, ans=0.2 2023-11-22 16:08:55,394 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 800, loss[loss=0.06853, simple_loss=0.09947, pruned_loss=0.01091, audio_tagging_loss=0.007887, over 15799.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09286, pruned_loss=0.01462, audio_tagging_loss=0.009693, over 2988728.79 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:09:00,858 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301400 2023-11-22 16:09:08,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2009360.0, ans=0.2 2023-11-22 16:09:19,562 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.335e+01 8.906e+01 9.640e+01 1.225e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-22 16:09:46,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2009560.0, ans=0.0 2023-11-22 16:10:00,006 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 850, loss[loss=0.04987, simple_loss=0.06358, pruned_loss=0.007101, audio_tagging_loss=0.01098, over 14679.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09276, pruned_loss=0.01452, audio_tagging_loss=0.009753, over 3006751.85 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:10:04,934 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301450 2023-11-22 16:10:08,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2009626.6666666667, ans=0.025 2023-11-22 16:10:13,389 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:10:18,978 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:10:23,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2009693.3333333333, ans=0.1 2023-11-22 16:10:57,747 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.17 vs. limit=15.0 2023-11-22 16:11:03,345 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 900, loss[loss=0.0761, simple_loss=0.1029, pruned_loss=0.01455, audio_tagging_loss=0.0101, over 15668.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09341, pruned_loss=0.01481, audio_tagging_loss=0.009828, over 3012777.01 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:11:08,964 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301500 2023-11-22 16:11:27,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2010026.6666666667, ans=0.125 2023-11-22 16:11:28,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.648e+01 9.209e+01 1.007e+02 1.462e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-22 16:11:38,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2010093.3333333333, ans=0.0 2023-11-22 16:12:07,509 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 950, loss[loss=0.05932, simple_loss=0.06987, pruned_loss=0.01197, audio_tagging_loss=0.01241, over 15063.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09436, pruned_loss=0.01497, audio_tagging_loss=0.009611, over 3024955.98 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:12:13,787 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301550 2023-11-22 16:12:29,735 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:12:32,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2010426.6666666667, ans=0.125 2023-11-22 16:12:41,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2010426.6666666667, ans=0.125 2023-11-22 16:13:11,458 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1000, loss[loss=0.07162, simple_loss=0.09704, pruned_loss=0.01404, audio_tagging_loss=0.009061, over 16478.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09395, pruned_loss=0.01492, audio_tagging_loss=0.009446, over 3031374.43 frames. ], batch size: 62, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:13:16,284 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301600 2023-11-22 16:13:17,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2010626.6666666667, ans=0.125 2023-11-22 16:13:27,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2010693.3333333333, ans=0.125 2023-11-22 16:13:35,425 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.207e+01 8.822e+01 9.755e+01 1.375e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 16:13:37,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2010760.0, ans=0.0 2023-11-22 16:13:39,672 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 16:13:47,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2010760.0, ans=0.125 2023-11-22 16:13:52,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2010826.6666666667, ans=22.5 2023-11-22 16:14:09,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2010893.3333333333, ans=0.0 2023-11-22 16:14:13,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2010893.3333333333, ans=0.0 2023-11-22 16:14:15,391 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1050, loss[loss=0.05341, simple_loss=0.06786, pruned_loss=0.009393, audio_tagging_loss=0.01008, over 15588.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09269, pruned_loss=0.01464, audio_tagging_loss=0.009495, over 3034485.34 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:14:20,342 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301650 2023-11-22 16:14:39,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2011026.6666666667, ans=0.0 2023-11-22 16:14:47,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2011093.3333333333, ans=0.125 2023-11-22 16:15:10,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2011226.6666666667, ans=0.125 2023-11-22 16:15:18,408 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0 2023-11-22 16:15:20,090 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1100, loss[loss=0.08033, simple_loss=0.1124, pruned_loss=0.01727, audio_tagging_loss=0.006868, over 15182.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.0933, pruned_loss=0.01461, audio_tagging_loss=0.009373, over 3036198.91 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:15:20,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2011293.3333333333, ans=0.04949747468305833 2023-11-22 16:15:23,757 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 16:15:25,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301700 2023-11-22 16:15:29,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2011293.3333333333, ans=0.0 2023-11-22 16:15:30,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.38 vs. limit=15.0 2023-11-22 16:15:31,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2011293.3333333333, ans=0.1 2023-11-22 16:15:32,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2011360.0, ans=0.125 2023-11-22 16:15:37,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2011360.0, ans=0.125 2023-11-22 16:15:43,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2011360.0, ans=0.0 2023-11-22 16:15:44,419 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.766e+01 8.159e+01 8.793e+01 9.276e+01 1.252e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 16:15:48,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2011426.6666666667, ans=0.0 2023-11-22 16:16:04,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.77 vs. limit=15.0 2023-11-22 16:16:24,777 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1150, loss[loss=0.05607, simple_loss=0.06563, pruned_loss=0.01196, audio_tagging_loss=0.0113, over 15997.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09373, pruned_loss=0.01454, audio_tagging_loss=0.009287, over 3041773.97 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:16:27,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2011626.6666666667, ans=0.2 2023-11-22 16:16:29,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301750 2023-11-22 16:16:29,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2011626.6666666667, ans=0.125 2023-11-22 16:16:40,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2011693.3333333333, ans=0.0 2023-11-22 16:16:43,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0 2023-11-22 16:17:02,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2011826.6666666667, ans=0.125 2023-11-22 16:17:22,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2011893.3333333333, ans=0.0 2023-11-22 16:17:28,017 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1200, loss[loss=0.06036, simple_loss=0.08841, pruned_loss=0.009125, audio_tagging_loss=0.007033, over 14222.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09373, pruned_loss=0.01457, audio_tagging_loss=0.009245, over 3041563.29 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:17:33,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301800 2023-11-22 16:17:50,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2012026.6666666667, ans=0.1 2023-11-22 16:17:54,137 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 8.163e+01 8.768e+01 9.430e+01 1.301e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 16:17:55,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2012093.3333333333, ans=0.125 2023-11-22 16:17:56,264 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.55 vs. limit=15.0 2023-11-22 16:18:07,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=15.0 2023-11-22 16:18:10,864 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:18:32,628 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1250, loss[loss=0.0739, simple_loss=0.1116, pruned_loss=0.01168, audio_tagging_loss=0.006407, over 15613.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.0936, pruned_loss=0.01463, audio_tagging_loss=0.009243, over 3038879.23 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:18:34,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2012293.3333333333, ans=0.2 2023-11-22 16:18:37,643 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301850 2023-11-22 16:18:44,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2012360.0, ans=0.0 2023-11-22 16:19:06,319 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.36 vs. limit=15.0 2023-11-22 16:19:06,604 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.43 vs. limit=15.0 2023-11-22 16:19:19,809 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.32 vs. limit=10.0 2023-11-22 16:19:24,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.65 vs. limit=10.0 2023-11-22 16:19:34,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2012560.0, ans=0.95 2023-11-22 16:19:37,029 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1300, loss[loss=0.0941, simple_loss=0.1262, pruned_loss=0.02127, audio_tagging_loss=0.00976, over 15544.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.0931, pruned_loss=0.01437, audio_tagging_loss=0.009251, over 3039594.08 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:19:42,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301900 2023-11-22 16:19:45,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2012626.6666666667, ans=0.0 2023-11-22 16:20:02,339 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.148e+01 8.961e+01 9.729e+01 1.373e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-22 16:20:21,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2012826.6666666667, ans=0.125 2023-11-22 16:20:21,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2012826.6666666667, ans=0.125 2023-11-22 16:20:23,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.36 vs. limit=15.0 2023-11-22 16:20:38,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2012893.3333333333, ans=0.0 2023-11-22 16:20:41,719 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1350, loss[loss=0.08586, simple_loss=0.1191, pruned_loss=0.02, audio_tagging_loss=0.00632, over 15650.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09253, pruned_loss=0.01436, audio_tagging_loss=0.009245, over 3038423.62 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:20:46,782 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 301950 2023-11-22 16:20:51,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2012960.0, ans=0.05 2023-11-22 16:20:56,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2013026.6666666667, ans=0.2 2023-11-22 16:21:02,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2013026.6666666667, ans=0.125 2023-11-22 16:21:20,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2013160.0, ans=0.1 2023-11-22 16:21:29,497 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 16:21:29,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2013160.0, ans=0.0 2023-11-22 16:21:38,401 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:21:42,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2013226.6666666667, ans=0.125 2023-11-22 16:21:46,585 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1400, loss[loss=0.06787, simple_loss=0.08492, pruned_loss=0.01459, audio_tagging_loss=0.01082, over 15366.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09178, pruned_loss=0.01436, audio_tagging_loss=0.00935, over 3041012.35 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:21:49,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2013293.3333333333, ans=0.0 2023-11-22 16:21:51,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302000 2023-11-22 16:21:55,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2013293.3333333333, ans=0.125 2023-11-22 16:22:05,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2013360.0, ans=0.125 2023-11-22 16:22:13,005 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.088e+01 8.833e+01 9.465e+01 1.089e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 16:22:23,983 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0 2023-11-22 16:22:45,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.67 vs. limit=12.0 2023-11-22 16:22:50,767 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1450, loss[loss=0.08068, simple_loss=0.1169, pruned_loss=0.01492, audio_tagging_loss=0.007282, over 15128.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09342, pruned_loss=0.01473, audio_tagging_loss=0.00941, over 3045664.06 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:22:55,909 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302050 2023-11-22 16:23:07,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2013693.3333333333, ans=0.1 2023-11-22 16:23:18,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2013760.0, ans=0.125 2023-11-22 16:23:26,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2013760.0, ans=0.0 2023-11-22 16:23:39,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2013826.6666666667, ans=0.0 2023-11-22 16:23:44,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2013893.3333333333, ans=0.2 2023-11-22 16:23:53,669 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1500, loss[loss=0.05498, simple_loss=0.08298, pruned_loss=0.008248, audio_tagging_loss=0.005238, over 15686.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.0938, pruned_loss=0.0149, audio_tagging_loss=0.009318, over 3043805.10 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:23:59,479 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302100 2023-11-22 16:24:03,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2013960.0, ans=0.125 2023-11-22 16:24:08,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2014026.6666666667, ans=0.0 2023-11-22 16:24:20,919 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.201e+01 8.781e+01 9.408e+01 1.231e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 16:24:33,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2014160.0, ans=0.125 2023-11-22 16:24:36,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2014160.0, ans=0.0 2023-11-22 16:24:42,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2014160.0, ans=0.035 2023-11-22 16:24:44,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2014226.6666666667, ans=0.125 2023-11-22 16:24:52,476 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.42 vs. limit=15.0 2023-11-22 16:24:58,138 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1550, loss[loss=0.07484, simple_loss=0.1085, pruned_loss=0.01286, audio_tagging_loss=0.007707, over 15064.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09357, pruned_loss=0.01483, audio_tagging_loss=0.009389, over 3045619.43 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:25:04,259 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302150 2023-11-22 16:25:25,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2014426.6666666667, ans=0.125 2023-11-22 16:25:41,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2014493.3333333333, ans=0.125 2023-11-22 16:25:52,243 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0 2023-11-22 16:25:59,067 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:26:03,125 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1600, loss[loss=0.06709, simple_loss=0.08357, pruned_loss=0.01568, audio_tagging_loss=0.009627, over 15124.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09351, pruned_loss=0.01502, audio_tagging_loss=0.009543, over 3040728.58 frames. ], batch size: 60, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:26:03,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2014626.6666666667, ans=0.07 2023-11-22 16:26:07,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302200 2023-11-22 16:26:10,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2014626.6666666667, ans=0.0 2023-11-22 16:26:12,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.50 vs. limit=6.0 2023-11-22 16:26:22,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.13 vs. limit=15.0 2023-11-22 16:26:27,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2014760.0, ans=0.0 2023-11-22 16:26:29,464 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.323e+01 8.320e+01 8.949e+01 9.605e+01 1.369e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-22 16:26:41,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.10 vs. limit=22.5 2023-11-22 16:27:02,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.16 vs. limit=15.0 2023-11-22 16:27:04,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0 2023-11-22 16:27:06,581 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1650, loss[loss=0.07955, simple_loss=0.1031, pruned_loss=0.01602, audio_tagging_loss=0.01196, over 15155.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09288, pruned_loss=0.01483, audio_tagging_loss=0.009607, over 3041178.14 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:27:11,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302250 2023-11-22 16:27:14,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2014960.0, ans=0.1 2023-11-22 16:27:33,390 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0 2023-11-22 16:27:58,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.84 vs. limit=15.0 2023-11-22 16:27:59,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2015226.6666666667, ans=0.07 2023-11-22 16:28:00,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2015226.6666666667, ans=0.1 2023-11-22 16:28:10,478 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1700, loss[loss=0.06991, simple_loss=0.09743, pruned_loss=0.01205, audio_tagging_loss=0.009139, over 15007.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09284, pruned_loss=0.01481, audio_tagging_loss=0.009594, over 3038907.75 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:28:13,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2015293.3333333333, ans=0.1 2023-11-22 16:28:14,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2015293.3333333333, ans=0.125 2023-11-22 16:28:16,042 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302300 2023-11-22 16:28:36,857 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.605e+01 8.196e+01 8.694e+01 9.442e+01 1.208e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 16:28:51,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2015493.3333333333, ans=10.0 2023-11-22 16:28:53,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2015493.3333333333, ans=0.0 2023-11-22 16:29:03,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2015560.0, ans=0.1 2023-11-22 16:29:08,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.86 vs. limit=15.0 2023-11-22 16:29:13,281 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1750, loss[loss=0.06705, simple_loss=0.08245, pruned_loss=0.017, audio_tagging_loss=0.008823, over 14642.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09297, pruned_loss=0.01484, audio_tagging_loss=0.00949, over 3035489.83 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:29:18,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302350 2023-11-22 16:29:32,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2015693.3333333333, ans=0.0 2023-11-22 16:29:44,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.54 vs. limit=15.0 2023-11-22 16:29:49,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2015760.0, ans=0.0 2023-11-22 16:30:01,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.93 vs. limit=15.0 2023-11-22 16:30:17,056 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1800, loss[loss=0.08643, simple_loss=0.1209, pruned_loss=0.01791, audio_tagging_loss=0.008094, over 15652.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09359, pruned_loss=0.01476, audio_tagging_loss=0.00942, over 3035619.85 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:30:19,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2015960.0, ans=0.125 2023-11-22 16:30:22,060 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302400 2023-11-22 16:30:42,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2016093.3333333333, ans=0.125 2023-11-22 16:30:43,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2016093.3333333333, ans=0.07 2023-11-22 16:30:44,308 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.100e+01 8.696e+01 9.181e+01 1.134e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 16:30:50,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2016093.3333333333, ans=0.125 2023-11-22 16:31:09,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2016226.6666666667, ans=0.1 2023-11-22 16:31:20,403 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1850, loss[loss=0.08692, simple_loss=0.1147, pruned_loss=0.02078, audio_tagging_loss=0.008791, over 16050.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.0935, pruned_loss=0.01489, audio_tagging_loss=0.009291, over 3027942.03 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:31:25,339 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302450 2023-11-22 16:31:38,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2016360.0, ans=0.1 2023-11-22 16:31:42,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2016360.0, ans=0.1 2023-11-22 16:31:43,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2016360.0, ans=0.0 2023-11-22 16:31:47,802 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.90 vs. limit=22.5 2023-11-22 16:31:49,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2016426.6666666667, ans=0.2 2023-11-22 16:32:04,103 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.22 vs. limit=12.0 2023-11-22 16:32:25,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2016626.6666666667, ans=0.0 2023-11-22 16:32:26,294 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1900, loss[loss=0.06375, simple_loss=0.08618, pruned_loss=0.008686, audio_tagging_loss=0.01197, over 15042.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09354, pruned_loss=0.01473, audio_tagging_loss=0.009231, over 3032039.50 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:32:31,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302500 2023-11-22 16:32:35,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2016626.6666666667, ans=0.1 2023-11-22 16:32:52,810 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.955e+01 8.093e+01 8.828e+01 9.639e+01 1.190e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 16:32:55,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2016760.0, ans=0.125 2023-11-22 16:33:05,598 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.62 vs. limit=22.5 2023-11-22 16:33:16,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2016893.3333333333, ans=0.0 2023-11-22 16:33:18,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2016893.3333333333, ans=0.035 2023-11-22 16:33:23,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.06 vs. limit=12.0 2023-11-22 16:33:29,828 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 1950, loss[loss=0.07827, simple_loss=0.1001, pruned_loss=0.01851, audio_tagging_loss=0.009718, over 15002.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09343, pruned_loss=0.01469, audio_tagging_loss=0.009204, over 3035965.20 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:33:34,851 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302550 2023-11-22 16:34:01,925 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:34:14,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=11.23 vs. limit=15.0 2023-11-22 16:34:19,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2017226.6666666667, ans=0.125 2023-11-22 16:34:32,536 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2000, loss[loss=0.06197, simple_loss=0.08706, pruned_loss=0.01129, audio_tagging_loss=0.007148, over 15134.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09349, pruned_loss=0.01473, audio_tagging_loss=0.009172, over 3035905.63 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0 2023-11-22 16:34:37,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302600 2023-11-22 16:34:55,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2017360.0, ans=0.1 2023-11-22 16:35:00,836 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.236e+01 8.987e+01 9.619e+01 1.204e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 16:35:15,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2017493.3333333333, ans=0.125 2023-11-22 16:35:37,071 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2050, loss[loss=0.06815, simple_loss=0.09025, pruned_loss=0.01442, audio_tagging_loss=0.008603, over 15255.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.0942, pruned_loss=0.01492, audio_tagging_loss=0.009208, over 3037683.68 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 8.0 2023-11-22 16:35:42,625 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302650 2023-11-22 16:35:47,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.92 vs. limit=22.5 2023-11-22 16:35:49,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2017693.3333333333, ans=0.125 2023-11-22 16:35:52,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.38 vs. limit=22.5 2023-11-22 16:35:55,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2017693.3333333333, ans=0.2 2023-11-22 16:36:15,584 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.20 vs. limit=6.0 2023-11-22 16:36:26,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2017893.3333333333, ans=0.125 2023-11-22 16:36:41,106 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2100, loss[loss=0.09493, simple_loss=0.128, pruned_loss=0.02368, audio_tagging_loss=0.007251, over 15054.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09396, pruned_loss=0.01495, audio_tagging_loss=0.009211, over 3040640.01 frames. ], batch size: 53, lr: 2.64e-03, grad_scale: 8.0 2023-11-22 16:36:46,161 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302700 2023-11-22 16:36:52,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2018026.6666666667, ans=0.1 2023-11-22 16:37:03,936 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.08 vs. limit=10.0 2023-11-22 16:37:09,224 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.575e+01 8.367e+01 8.996e+01 9.804e+01 1.259e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 16:37:20,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2018160.0, ans=0.2 2023-11-22 16:37:33,980 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.19 vs. limit=15.0 2023-11-22 16:37:43,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2018293.3333333333, ans=0.0 2023-11-22 16:37:44,291 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2150, loss[loss=0.08099, simple_loss=0.09952, pruned_loss=0.02056, audio_tagging_loss=0.01067, over 14585.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.0936, pruned_loss=0.01484, audio_tagging_loss=0.009241, over 3031217.05 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 8.0 2023-11-22 16:37:45,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2018293.3333333333, ans=0.125 2023-11-22 16:37:49,327 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302750 2023-11-22 16:37:50,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2018293.3333333333, ans=0.125 2023-11-22 16:37:51,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2018293.3333333333, ans=0.1 2023-11-22 16:38:16,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2018426.6666666667, ans=0.125 2023-11-22 16:38:18,977 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.26 vs. limit=15.0 2023-11-22 16:38:23,069 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 16:38:47,927 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2200, loss[loss=0.05676, simple_loss=0.07457, pruned_loss=0.008758, audio_tagging_loss=0.01072, over 14919.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09361, pruned_loss=0.01476, audio_tagging_loss=0.009334, over 3032708.50 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 8.0 2023-11-22 16:38:52,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302800 2023-11-22 16:38:57,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.81 vs. limit=15.0 2023-11-22 16:39:16,790 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 8.391e+01 8.919e+01 9.600e+01 1.144e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 16:39:23,251 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:39:25,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2018826.6666666667, ans=0.0 2023-11-22 16:39:27,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2018826.6666666667, ans=0.125 2023-11-22 16:39:33,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2018826.6666666667, ans=0.125 2023-11-22 16:39:34,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2018826.6666666667, ans=0.1 2023-11-22 16:39:51,946 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2250, loss[loss=0.07652, simple_loss=0.1032, pruned_loss=0.01584, audio_tagging_loss=0.009106, over 15143.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09361, pruned_loss=0.01486, audio_tagging_loss=0.009337, over 3030759.74 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 8.0 2023-11-22 16:39:56,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302850 2023-11-22 16:40:16,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2019093.3333333333, ans=0.125 2023-11-22 16:40:25,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2019093.3333333333, ans=0.125 2023-11-22 16:40:29,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2019160.0, ans=0.0 2023-11-22 16:40:49,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2019226.6666666667, ans=0.07 2023-11-22 16:40:54,902 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2300, loss[loss=0.07521, simple_loss=0.1097, pruned_loss=0.0136, audio_tagging_loss=0.006751, over 16066.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09363, pruned_loss=0.01483, audio_tagging_loss=0.009387, over 3034633.48 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 8.0 2023-11-22 16:40:59,875 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302900 2023-11-22 16:41:24,527 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.135e+01 8.659e+01 9.382e+01 1.572e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-22 16:41:39,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.58 vs. limit=15.0 2023-11-22 16:41:50,892 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 16:41:58,382 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2350, loss[loss=0.06397, simple_loss=0.08748, pruned_loss=0.00991, audio_tagging_loss=0.01033, over 16420.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09388, pruned_loss=0.01483, audio_tagging_loss=0.009452, over 3037158.10 frames. ], batch size: 63, lr: 2.64e-03, grad_scale: 8.0 2023-11-22 16:42:03,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 302950 2023-11-22 16:42:29,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2019760.0, ans=0.1 2023-11-22 16:42:39,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2019826.6666666667, ans=0.0 2023-11-22 16:42:42,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.06 vs. limit=22.5 2023-11-22 16:43:02,283 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2400, loss[loss=0.06342, simple_loss=0.08012, pruned_loss=0.01127, audio_tagging_loss=0.01209, over 15217.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.09447, pruned_loss=0.01493, audio_tagging_loss=0.009479, over 3032863.68 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:43:07,858 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303000 2023-11-22 16:43:19,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2020026.6666666667, ans=0.125 2023-11-22 16:43:22,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.36 vs. limit=15.0 2023-11-22 16:43:31,207 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.331e+01 8.278e+01 8.806e+01 9.570e+01 1.233e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-22 16:43:31,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2020093.3333333333, ans=0.1 2023-11-22 16:43:39,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2020160.0, ans=0.125 2023-11-22 16:43:40,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2020160.0, ans=0.0 2023-11-22 16:44:05,441 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2450, loss[loss=0.06632, simple_loss=0.08048, pruned_loss=0.01487, audio_tagging_loss=0.01121, over 15544.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09454, pruned_loss=0.01486, audio_tagging_loss=0.009417, over 3035289.83 frames. ], batch size: 63, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:44:10,423 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303050 2023-11-22 16:44:27,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2020360.0, ans=0.125 2023-11-22 16:44:28,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2020360.0, ans=0.2 2023-11-22 16:44:29,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2020426.6666666667, ans=0.2 2023-11-22 16:44:31,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.08 vs. limit=15.0 2023-11-22 16:44:34,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2020426.6666666667, ans=0.125 2023-11-22 16:44:34,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2020426.6666666667, ans=0.0 2023-11-22 16:44:42,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2020493.3333333333, ans=0.0 2023-11-22 16:44:47,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2020493.3333333333, ans=0.1 2023-11-22 16:44:48,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2020493.3333333333, ans=0.0 2023-11-22 16:45:08,146 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2500, loss[loss=0.06492, simple_loss=0.08507, pruned_loss=0.009045, audio_tagging_loss=0.01334, over 14927.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09446, pruned_loss=0.01485, audio_tagging_loss=0.009463, over 3041325.63 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:45:12,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2020626.6666666667, ans=0.125 2023-11-22 16:45:13,805 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303100 2023-11-22 16:45:31,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2020693.3333333333, ans=0.125 2023-11-22 16:45:37,911 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.127e+01 8.699e+01 9.305e+01 1.342e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 16:45:38,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2020760.0, ans=10.0 2023-11-22 16:45:49,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2020826.6666666667, ans=0.1 2023-11-22 16:45:52,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2020826.6666666667, ans=0.2 2023-11-22 16:46:12,049 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2550, loss[loss=0.05404, simple_loss=0.06307, pruned_loss=0.01063, audio_tagging_loss=0.01188, over 15364.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09422, pruned_loss=0.01496, audio_tagging_loss=0.009392, over 3044100.94 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 16.0 2023-11-22 16:46:16,907 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303150 2023-11-22 16:46:33,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.17 vs. limit=10.0 2023-11-22 16:46:33,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2021026.6666666667, ans=0.125 2023-11-22 16:46:36,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2021093.3333333333, ans=10.0 2023-11-22 16:46:45,326 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.18 vs. limit=15.0 2023-11-22 16:46:52,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2021160.0, ans=0.0 2023-11-22 16:46:56,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2021160.0, ans=0.125 2023-11-22 16:47:02,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2021226.6666666667, ans=0.125 2023-11-22 16:47:14,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2021293.3333333333, ans=0.125 2023-11-22 16:47:15,974 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2600, loss[loss=0.06026, simple_loss=0.08287, pruned_loss=0.01021, audio_tagging_loss=0.008618, over 15120.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.0933, pruned_loss=0.01483, audio_tagging_loss=0.009235, over 3037228.79 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:47:20,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303200 2023-11-22 16:47:21,039 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:47:26,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2021293.3333333333, ans=0.0 2023-11-22 16:47:27,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2021360.0, ans=0.2 2023-11-22 16:47:27,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2021360.0, ans=0.2 2023-11-22 16:47:44,771 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 8.118e+01 8.704e+01 9.634e+01 1.548e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-22 16:48:00,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2021493.3333333333, ans=0.125 2023-11-22 16:48:18,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2021626.6666666667, ans=0.0 2023-11-22 16:48:19,353 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2650, loss[loss=0.06568, simple_loss=0.08673, pruned_loss=0.01245, audio_tagging_loss=0.009865, over 15165.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09317, pruned_loss=0.01487, audio_tagging_loss=0.009149, over 3038644.59 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:48:23,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2021626.6666666667, ans=0.125 2023-11-22 16:48:24,377 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303250 2023-11-22 16:48:37,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2021693.3333333333, ans=0.0 2023-11-22 16:48:45,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2021760.0, ans=0.1 2023-11-22 16:49:09,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2021893.3333333333, ans=0.0 2023-11-22 16:49:23,151 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2700, loss[loss=0.09598, simple_loss=0.1242, pruned_loss=0.02451, audio_tagging_loss=0.009375, over 16175.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09376, pruned_loss=0.01515, audio_tagging_loss=0.009134, over 3041121.49 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:49:28,816 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303300 2023-11-22 16:49:29,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.49 vs. limit=15.0 2023-11-22 16:49:52,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2022093.3333333333, ans=0.125 2023-11-22 16:49:53,746 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.226e+01 8.902e+01 9.755e+01 1.486e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-22 16:50:26,595 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2750, loss[loss=0.06737, simple_loss=0.09191, pruned_loss=0.01193, audio_tagging_loss=0.009483, over 14424.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09298, pruned_loss=0.01503, audio_tagging_loss=0.009149, over 3045507.27 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 8.0 2023-11-22 16:50:31,482 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303350 2023-11-22 16:50:56,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2022426.6666666667, ans=0.125 2023-11-22 16:50:59,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2022426.6666666667, ans=0.5 2023-11-22 16:51:21,015 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 16:51:30,242 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2800, loss[loss=0.07467, simple_loss=0.09972, pruned_loss=0.01587, audio_tagging_loss=0.00894, over 15120.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09237, pruned_loss=0.01477, audio_tagging_loss=0.00922, over 3042483.74 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:51:35,412 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303400 2023-11-22 16:51:44,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2022693.3333333333, ans=0.0 2023-11-22 16:51:50,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2022693.3333333333, ans=0.0 2023-11-22 16:51:55,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2022760.0, ans=0.125 2023-11-22 16:51:59,713 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.45 vs. limit=15.0 2023-11-22 16:52:00,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2022760.0, ans=0.0 2023-11-22 16:52:01,425 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.550e+01 8.038e+01 8.757e+01 9.417e+01 1.241e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-22 16:52:06,596 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:52:15,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2022826.6666666667, ans=0.0 2023-11-22 16:52:28,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2022893.3333333333, ans=0.125 2023-11-22 16:52:34,473 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2850, loss[loss=0.08822, simple_loss=0.1284, pruned_loss=0.0192, audio_tagging_loss=0.004793, over 15031.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.0923, pruned_loss=0.01464, audio_tagging_loss=0.009226, over 3039639.05 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:52:40,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303450 2023-11-22 16:52:50,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2023026.6666666667, ans=0.1 2023-11-22 16:53:07,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2023093.3333333333, ans=0.125 2023-11-22 16:53:17,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2023160.0, ans=0.0 2023-11-22 16:53:28,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2023226.6666666667, ans=0.1 2023-11-22 16:53:30,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2023226.6666666667, ans=0.1 2023-11-22 16:53:37,519 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2900, loss[loss=0.07167, simple_loss=0.0945, pruned_loss=0.01399, audio_tagging_loss=0.01043, over 14267.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09253, pruned_loss=0.01476, audio_tagging_loss=0.009214, over 3035238.83 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:53:42,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303500 2023-11-22 16:53:44,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2023293.3333333333, ans=0.0 2023-11-22 16:53:56,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2023360.0, ans=0.0 2023-11-22 16:54:05,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2023426.6666666667, ans=0.0 2023-11-22 16:54:07,705 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.261e+01 8.993e+01 9.849e+01 1.244e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 16:54:20,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2023493.3333333333, ans=0.0 2023-11-22 16:54:21,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2023493.3333333333, ans=0.125 2023-11-22 16:54:25,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.42 vs. limit=15.0 2023-11-22 16:54:26,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2023560.0, ans=0.125 2023-11-22 16:54:29,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.43 vs. limit=22.5 2023-11-22 16:54:29,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.80 vs. limit=15.0 2023-11-22 16:54:32,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2023560.0, ans=0.09899494936611666 2023-11-22 16:54:39,788 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 2950, loss[loss=0.06203, simple_loss=0.08135, pruned_loss=0.009905, audio_tagging_loss=0.01145, over 14583.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09407, pruned_loss=0.01506, audio_tagging_loss=0.009219, over 3042128.47 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:54:40,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2023626.6666666667, ans=0.0 2023-11-22 16:54:45,357 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303550 2023-11-22 16:55:34,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2023893.3333333333, ans=0.125 2023-11-22 16:55:43,882 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3000, loss[loss=0.08631, simple_loss=0.1199, pruned_loss=0.01708, audio_tagging_loss=0.009277, over 15631.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09474, pruned_loss=0.0152, audio_tagging_loss=0.009308, over 3052724.77 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:55:43,883 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 16:56:20,523 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2591, 4.2054, 4.4503, 4.4320], device='cuda:3') 2023-11-22 16:56:24,238 INFO [train_asr.py:1253] (3/4) Epoch 26, validation: loss=0.05863, simple_loss=0.05148, pruned_loss=0.005087, audio_tagging_loss=0.0278, over 4681554.00 frames. 2023-11-22 16:56:24,239 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 16:56:28,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.84 vs. limit=15.0 2023-11-22 16:56:29,262 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303600 2023-11-22 16:56:45,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2024026.6666666667, ans=0.0 2023-11-22 16:56:47,604 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.70 vs. limit=15.0 2023-11-22 16:56:55,186 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.309e+01 8.958e+01 9.598e+01 2.915e+02, threshold=1.792e+02, percent-clipped=1.0 2023-11-22 16:57:02,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2024160.0, ans=0.1 2023-11-22 16:57:11,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2024160.0, ans=0.0 2023-11-22 16:57:14,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2024226.6666666667, ans=0.0 2023-11-22 16:57:15,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2024226.6666666667, ans=0.125 2023-11-22 16:57:27,858 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3050, loss[loss=0.09519, simple_loss=0.1354, pruned_loss=0.02068, audio_tagging_loss=0.006802, over 15530.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09479, pruned_loss=0.01517, audio_tagging_loss=0.009415, over 3043615.50 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:57:32,978 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303650 2023-11-22 16:57:36,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2024293.3333333333, ans=0.125 2023-11-22 16:57:56,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2024426.6666666667, ans=0.125 2023-11-22 16:57:57,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2024426.6666666667, ans=0.125 2023-11-22 16:57:58,341 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 16:58:00,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2024426.6666666667, ans=0.1 2023-11-22 16:58:06,780 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 16:58:11,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2024493.3333333333, ans=0.125 2023-11-22 16:58:21,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2024560.0, ans=0.0 2023-11-22 16:58:33,078 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3100, loss[loss=0.07572, simple_loss=0.1021, pruned_loss=0.01547, audio_tagging_loss=0.009179, over 15787.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09619, pruned_loss=0.01546, audio_tagging_loss=0.009396, over 3047397.51 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:58:38,687 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303700 2023-11-22 16:59:03,007 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.458e+01 8.373e+01 8.955e+01 9.319e+01 1.385e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-22 16:59:12,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.59 vs. limit=22.5 2023-11-22 16:59:35,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2024960.0, ans=0.125 2023-11-22 16:59:36,714 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3150, loss[loss=0.05451, simple_loss=0.06369, pruned_loss=0.0123, audio_tagging_loss=0.01037, over 13991.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.09525, pruned_loss=0.01526, audio_tagging_loss=0.009507, over 3043414.34 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 16:59:41,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303750 2023-11-22 16:59:55,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2025026.6666666667, ans=0.1 2023-11-22 17:00:12,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2025093.3333333333, ans=0.0 2023-11-22 17:00:13,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.35 vs. limit=22.5 2023-11-22 17:00:31,496 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.91 vs. limit=15.0 2023-11-22 17:00:39,351 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3200, loss[loss=0.07456, simple_loss=0.1045, pruned_loss=0.01364, audio_tagging_loss=0.00866, over 15811.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09489, pruned_loss=0.01518, audio_tagging_loss=0.009541, over 3048782.87 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:00:44,343 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303800 2023-11-22 17:00:56,973 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 17:01:03,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2025360.0, ans=0.0 2023-11-22 17:01:06,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2025426.6666666667, ans=0.2 2023-11-22 17:01:08,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2025426.6666666667, ans=0.125 2023-11-22 17:01:10,783 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.434e+01 8.209e+01 8.808e+01 9.580e+01 1.231e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 17:01:23,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2025493.3333333333, ans=0.1 2023-11-22 17:01:30,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.78 vs. limit=15.0 2023-11-22 17:01:32,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2025560.0, ans=0.1 2023-11-22 17:01:42,694 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3250, loss[loss=0.07336, simple_loss=0.08441, pruned_loss=0.01987, audio_tagging_loss=0.01128, over 15683.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09407, pruned_loss=0.01494, audio_tagging_loss=0.009649, over 3049474.94 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:01:47,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2025626.6666666667, ans=0.125 2023-11-22 17:01:48,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303850 2023-11-22 17:01:55,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2025693.3333333333, ans=0.125 2023-11-22 17:01:59,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2025693.3333333333, ans=0.0 2023-11-22 17:02:02,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2025693.3333333333, ans=0.2 2023-11-22 17:02:22,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2025826.6666666667, ans=0.0 2023-11-22 17:02:27,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2025826.6666666667, ans=0.125 2023-11-22 17:02:37,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2025893.3333333333, ans=0.5 2023-11-22 17:02:37,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2025893.3333333333, ans=0.125 2023-11-22 17:02:45,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.10 vs. limit=12.0 2023-11-22 17:02:46,302 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3300, loss[loss=0.07613, simple_loss=0.103, pruned_loss=0.01536, audio_tagging_loss=0.009286, over 17585.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09422, pruned_loss=0.01501, audio_tagging_loss=0.009673, over 3051545.24 frames. ], batch size: 65, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:02:51,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303900 2023-11-22 17:03:03,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2026026.6666666667, ans=0.0 2023-11-22 17:03:16,785 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.797e+01 8.259e+01 8.793e+01 9.669e+01 1.578e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 17:03:49,229 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3350, loss[loss=0.04999, simple_loss=0.06971, pruned_loss=0.005426, audio_tagging_loss=0.009704, over 14776.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09405, pruned_loss=0.0149, audio_tagging_loss=0.009654, over 3052251.69 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:03:54,270 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 303950 2023-11-22 17:03:54,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.47 vs. limit=22.5 2023-11-22 17:04:27,519 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.33 vs. limit=15.0 2023-11-22 17:04:29,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2026493.3333333333, ans=0.125 2023-11-22 17:04:43,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2026560.0, ans=0.2 2023-11-22 17:04:45,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2026560.0, ans=0.125 2023-11-22 17:04:46,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2026560.0, ans=10.0 2023-11-22 17:04:50,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2026626.6666666667, ans=0.2 2023-11-22 17:04:51,532 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3400, loss[loss=0.08037, simple_loss=0.1108, pruned_loss=0.01624, audio_tagging_loss=0.008733, over 15196.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09426, pruned_loss=0.01491, audio_tagging_loss=0.00948, over 3044185.41 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:04:56,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304000 2023-11-22 17:05:15,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2026693.3333333333, ans=0.0 2023-11-22 17:05:16,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2026693.3333333333, ans=0.95 2023-11-22 17:05:18,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2026693.3333333333, ans=0.05 2023-11-22 17:05:18,283 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.41 vs. limit=12.0 2023-11-22 17:05:26,259 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.152e+01 8.823e+01 9.410e+01 1.182e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-22 17:05:47,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2026893.3333333333, ans=0.2 2023-11-22 17:05:57,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2026960.0, ans=0.04949747468305833 2023-11-22 17:05:58,785 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3450, loss[loss=0.05057, simple_loss=0.06379, pruned_loss=0.007789, audio_tagging_loss=0.01089, over 16459.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.0937, pruned_loss=0.01482, audio_tagging_loss=0.009347, over 3043955.89 frames. ], batch size: 66, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:05:59,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2026960.0, ans=0.125 2023-11-22 17:06:03,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304050 2023-11-22 17:06:05,467 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.22 vs. limit=15.0 2023-11-22 17:06:08,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2026960.0, ans=0.1 2023-11-22 17:06:24,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2027093.3333333333, ans=0.0 2023-11-22 17:06:24,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2027093.3333333333, ans=0.1 2023-11-22 17:06:50,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2027226.6666666667, ans=0.125 2023-11-22 17:06:52,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2027226.6666666667, ans=0.125 2023-11-22 17:07:01,445 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3500, loss[loss=0.0623, simple_loss=0.07949, pruned_loss=0.01144, audio_tagging_loss=0.01112, over 14435.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09349, pruned_loss=0.01479, audio_tagging_loss=0.009334, over 3045229.47 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:07:06,411 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304100 2023-11-22 17:07:10,615 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.45 vs. limit=15.0 2023-11-22 17:07:10,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.48 vs. limit=15.0 2023-11-22 17:07:30,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2027426.6666666667, ans=0.125 2023-11-22 17:07:33,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.704e+01 8.467e+01 8.969e+01 9.686e+01 1.239e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-22 17:07:36,544 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 17:07:45,875 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0 2023-11-22 17:07:46,884 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.77 vs. limit=15.0 2023-11-22 17:07:47,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2027493.3333333333, ans=0.0 2023-11-22 17:08:04,508 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3550, loss[loss=0.06904, simple_loss=0.09613, pruned_loss=0.01219, audio_tagging_loss=0.008782, over 15358.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.0938, pruned_loss=0.01485, audio_tagging_loss=0.009239, over 3047473.98 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:08:06,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2027626.6666666667, ans=0.125 2023-11-22 17:08:10,123 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304150 2023-11-22 17:08:13,027 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0 2023-11-22 17:08:15,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2027626.6666666667, ans=0.125 2023-11-22 17:08:19,460 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 17:08:30,335 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.04 vs. limit=10.0 2023-11-22 17:09:08,657 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3600, loss[loss=0.07097, simple_loss=0.09673, pruned_loss=0.01466, audio_tagging_loss=0.007942, over 15070.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09342, pruned_loss=0.01487, audio_tagging_loss=0.009237, over 3046778.37 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:09:14,327 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304200 2023-11-22 17:09:23,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2028026.6666666667, ans=0.2 2023-11-22 17:09:29,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.39 vs. limit=15.0 2023-11-22 17:09:41,224 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.567e+01 8.177e+01 8.783e+01 9.582e+01 1.117e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 17:09:46,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.74 vs. limit=15.0 2023-11-22 17:09:51,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=2028160.0, ans=0.02 2023-11-22 17:09:53,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2028160.0, ans=0.125 2023-11-22 17:10:03,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2028226.6666666667, ans=0.1 2023-11-22 17:10:11,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2028226.6666666667, ans=0.035 2023-11-22 17:10:13,382 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3650, loss[loss=0.0709, simple_loss=0.07496, pruned_loss=0.02232, audio_tagging_loss=0.0111, over 14758.00 frames. ], tot_loss[loss=0.072, simple_loss=0.0951, pruned_loss=0.01534, audio_tagging_loss=0.009116, over 3050494.90 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:10:18,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304250 2023-11-22 17:10:35,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2028360.0, ans=0.1 2023-11-22 17:10:50,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2028493.3333333333, ans=0.0 2023-11-22 17:11:05,283 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.74 vs. limit=12.0 2023-11-22 17:11:07,749 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.02 vs. limit=15.0 2023-11-22 17:11:16,820 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3700, loss[loss=0.08048, simple_loss=0.1041, pruned_loss=0.02019, audio_tagging_loss=0.008233, over 15564.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09657, pruned_loss=0.01573, audio_tagging_loss=0.009012, over 3055924.85 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:11:20,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2028626.6666666667, ans=0.125 2023-11-22 17:11:21,769 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304300 2023-11-22 17:11:30,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2028693.3333333333, ans=0.1 2023-11-22 17:11:31,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2028693.3333333333, ans=0.1 2023-11-22 17:11:35,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2028693.3333333333, ans=0.2 2023-11-22 17:11:50,449 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.264e+01 8.888e+01 9.594e+01 1.594e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 17:12:05,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.02 vs. limit=22.5 2023-11-22 17:12:18,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2028893.3333333333, ans=0.125 2023-11-22 17:12:21,540 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3750, loss[loss=0.06092, simple_loss=0.07437, pruned_loss=0.01384, audio_tagging_loss=0.009894, over 15665.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09664, pruned_loss=0.01562, audio_tagging_loss=0.009001, over 3062347.68 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:12:26,510 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304350 2023-11-22 17:12:40,549 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 17:12:40,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2029026.6666666667, ans=0.125 2023-11-22 17:12:50,075 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 17:12:52,877 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.08 vs. limit=12.0 2023-11-22 17:13:00,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2029160.0, ans=0.125 2023-11-22 17:13:05,484 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 17:13:05,908 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.85 vs. limit=15.0 2023-11-22 17:13:13,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2029226.6666666667, ans=0.2 2023-11-22 17:13:14,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2029226.6666666667, ans=0.0 2023-11-22 17:13:17,456 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 17:13:25,468 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3800, loss[loss=0.07163, simple_loss=0.1027, pruned_loss=0.01195, audio_tagging_loss=0.008337, over 13928.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.097, pruned_loss=0.01562, audio_tagging_loss=0.00901, over 3056231.83 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:13:29,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2029293.3333333333, ans=0.1 2023-11-22 17:13:30,574 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304400 2023-11-22 17:13:48,309 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.31 vs. limit=15.0 2023-11-22 17:13:53,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2029426.6666666667, ans=0.2 2023-11-22 17:14:00,216 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.081e+01 8.257e+01 8.903e+01 9.664e+01 1.355e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-22 17:14:17,769 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.20 vs. limit=12.0 2023-11-22 17:14:18,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2029560.0, ans=0.2 2023-11-22 17:14:30,803 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3850, loss[loss=0.06409, simple_loss=0.07476, pruned_loss=0.0122, audio_tagging_loss=0.0145, over 14980.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.0962, pruned_loss=0.01546, audio_tagging_loss=0.009139, over 3055762.83 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:14:32,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2029626.6666666667, ans=0.125 2023-11-22 17:14:35,750 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304450 2023-11-22 17:14:42,602 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.45 vs. limit=10.0 2023-11-22 17:15:07,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2029760.0, ans=0.0 2023-11-22 17:15:10,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.80 vs. limit=10.0 2023-11-22 17:15:20,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2029826.6666666667, ans=0.125 2023-11-22 17:15:23,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.15 vs. limit=22.5 2023-11-22 17:15:23,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2029893.3333333333, ans=0.125 2023-11-22 17:15:32,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2029893.3333333333, ans=0.1 2023-11-22 17:15:35,615 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3900, loss[loss=0.06761, simple_loss=0.09507, pruned_loss=0.01086, audio_tagging_loss=0.009221, over 16078.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09534, pruned_loss=0.01522, audio_tagging_loss=0.009245, over 3057135.25 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:15:41,111 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304500 2023-11-22 17:16:00,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2030093.3333333333, ans=0.125 2023-11-22 17:16:06,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2030093.3333333333, ans=0.1 2023-11-22 17:16:08,784 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.197e+01 8.799e+01 9.518e+01 1.700e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 17:16:26,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2030226.6666666667, ans=0.2 2023-11-22 17:16:38,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2030293.3333333333, ans=0.125 2023-11-22 17:16:39,729 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 3950, loss[loss=0.07377, simple_loss=0.1098, pruned_loss=0.01183, audio_tagging_loss=0.007025, over 15221.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09473, pruned_loss=0.01511, audio_tagging_loss=0.009381, over 3059629.52 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:16:44,844 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304550 2023-11-22 17:16:50,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2030293.3333333333, ans=0.0 2023-11-22 17:17:00,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2030360.0, ans=0.2 2023-11-22 17:17:13,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.39 vs. limit=15.0 2023-11-22 17:17:28,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2030493.3333333333, ans=0.0 2023-11-22 17:17:43,617 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4000, loss[loss=0.06738, simple_loss=0.09161, pruned_loss=0.01208, audio_tagging_loss=0.009497, over 15180.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09505, pruned_loss=0.01512, audio_tagging_loss=0.009428, over 3053739.18 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:17:48,568 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304600 2023-11-22 17:17:52,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2030626.6666666667, ans=0.1 2023-11-22 17:17:53,302 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.32 vs. limit=15.0 2023-11-22 17:17:56,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2030693.3333333333, ans=0.0 2023-11-22 17:18:17,468 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.393e+01 9.096e+01 9.757e+01 1.242e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-22 17:18:35,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2030893.3333333333, ans=0.0 2023-11-22 17:18:48,603 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4050, loss[loss=0.1062, simple_loss=0.1443, pruned_loss=0.02828, audio_tagging_loss=0.005837, over 15254.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.0945, pruned_loss=0.01514, audio_tagging_loss=0.009503, over 3046721.26 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:18:52,378 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 17:18:54,288 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304650 2023-11-22 17:19:06,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2031026.6666666667, ans=0.1 2023-11-22 17:19:08,500 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0 2023-11-22 17:19:10,859 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0 2023-11-22 17:19:11,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2031026.6666666667, ans=0.1 2023-11-22 17:19:37,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2031160.0, ans=0.025 2023-11-22 17:19:52,423 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4100, loss[loss=0.06951, simple_loss=0.09848, pruned_loss=0.01231, audio_tagging_loss=0.007964, over 14938.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09517, pruned_loss=0.01528, audio_tagging_loss=0.009546, over 3050829.24 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:19:55,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2031293.3333333333, ans=0.125 2023-11-22 17:19:57,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304700 2023-11-22 17:19:57,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2031293.3333333333, ans=0.5 2023-11-22 17:20:23,699 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.24 vs. limit=15.0 2023-11-22 17:20:25,426 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.231e+01 8.289e+01 8.885e+01 9.485e+01 1.195e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 17:20:25,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2031426.6666666667, ans=0.0 2023-11-22 17:20:27,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2031426.6666666667, ans=0.125 2023-11-22 17:20:42,145 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.07 vs. limit=15.0 2023-11-22 17:20:56,267 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4150, loss[loss=0.07484, simple_loss=0.1072, pruned_loss=0.01413, audio_tagging_loss=0.007135, over 14913.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09455, pruned_loss=0.01501, audio_tagging_loss=0.009487, over 3046401.94 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:21:01,816 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304750 2023-11-22 17:21:04,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.04 vs. limit=8.0 2023-11-22 17:21:15,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2031693.3333333333, ans=0.125 2023-11-22 17:21:31,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2031760.0, ans=0.0 2023-11-22 17:21:40,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2031826.6666666667, ans=0.07 2023-11-22 17:21:43,417 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 17:21:46,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2031893.3333333333, ans=0.0 2023-11-22 17:22:00,327 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4200, loss[loss=0.06246, simple_loss=0.08178, pruned_loss=0.01039, audio_tagging_loss=0.01117, over 15077.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09462, pruned_loss=0.01492, audio_tagging_loss=0.009304, over 3051702.10 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:22:05,833 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304800 2023-11-22 17:22:26,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2032093.3333333333, ans=0.2 2023-11-22 17:22:34,568 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.472e+01 8.973e+01 9.606e+01 1.148e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-22 17:22:34,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2032093.3333333333, ans=0.125 2023-11-22 17:22:49,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2032160.0, ans=0.1 2023-11-22 17:23:04,795 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4250, loss[loss=0.079, simple_loss=0.09914, pruned_loss=0.02214, audio_tagging_loss=0.007289, over 15207.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.0942, pruned_loss=0.01492, audio_tagging_loss=0.009213, over 3052362.24 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:23:08,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2032293.3333333333, ans=0.0 2023-11-22 17:23:09,741 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304850 2023-11-22 17:23:23,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2032360.0, ans=0.1 2023-11-22 17:23:24,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2032360.0, ans=0.5 2023-11-22 17:23:33,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2032426.6666666667, ans=0.1 2023-11-22 17:23:50,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2032493.3333333333, ans=0.1 2023-11-22 17:24:08,354 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4300, loss[loss=0.06263, simple_loss=0.08187, pruned_loss=0.01406, audio_tagging_loss=0.007631, over 14027.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09449, pruned_loss=0.01499, audio_tagging_loss=0.009156, over 3052216.65 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:24:13,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304900 2023-11-22 17:24:14,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2032626.6666666667, ans=0.125 2023-11-22 17:24:28,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.88 vs. limit=15.0 2023-11-22 17:24:30,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2032693.3333333333, ans=0.0 2023-11-22 17:24:39,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2032760.0, ans=0.1 2023-11-22 17:24:44,060 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.878e+01 8.402e+01 8.884e+01 9.738e+01 1.155e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 17:24:49,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2032826.6666666667, ans=0.1 2023-11-22 17:24:50,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2032826.6666666667, ans=0.125 2023-11-22 17:25:04,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2032893.3333333333, ans=0.2 2023-11-22 17:25:11,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.81 vs. limit=15.0 2023-11-22 17:25:13,352 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4350, loss[loss=0.06415, simple_loss=0.08916, pruned_loss=0.01113, audio_tagging_loss=0.00844, over 16029.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09376, pruned_loss=0.01482, audio_tagging_loss=0.009162, over 3052369.99 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:25:16,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.81 vs. limit=12.0 2023-11-22 17:25:19,508 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 304950 2023-11-22 17:25:19,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2032960.0, ans=0.0 2023-11-22 17:25:29,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2033026.6666666667, ans=0.05 2023-11-22 17:25:35,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.32 vs. limit=22.5 2023-11-22 17:26:00,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2033160.0, ans=0.1 2023-11-22 17:26:15,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2033226.6666666667, ans=0.0 2023-11-22 17:26:18,670 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4400, loss[loss=0.07276, simple_loss=0.09571, pruned_loss=0.01518, audio_tagging_loss=0.009728, over 15195.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09377, pruned_loss=0.01485, audio_tagging_loss=0.009161, over 3050127.32 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:26:23,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305000 2023-11-22 17:26:41,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.09 vs. limit=12.0 2023-11-22 17:26:42,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.52 vs. limit=15.0 2023-11-22 17:26:52,029 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.342e+01 8.115e+01 8.926e+01 9.630e+01 1.276e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-22 17:27:03,284 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.46 vs. limit=12.0 2023-11-22 17:27:05,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.30 vs. limit=22.5 2023-11-22 17:27:09,644 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.30 vs. limit=15.0 2023-11-22 17:27:22,549 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4450, loss[loss=0.07356, simple_loss=0.1081, pruned_loss=0.01484, audio_tagging_loss=0.004664, over 14940.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09402, pruned_loss=0.01481, audio_tagging_loss=0.009127, over 3051357.25 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:27:23,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.93 vs. limit=15.0 2023-11-22 17:27:27,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305050 2023-11-22 17:28:25,536 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4500, loss[loss=0.08077, simple_loss=0.1142, pruned_loss=0.01686, audio_tagging_loss=0.006815, over 15267.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09456, pruned_loss=0.01485, audio_tagging_loss=0.009149, over 3051398.08 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:28:31,149 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305100 2023-11-22 17:29:01,030 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.181e+01 8.834e+01 9.479e+01 1.227e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 17:29:09,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2034160.0, ans=0.125 2023-11-22 17:29:30,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2034293.3333333333, ans=0.125 2023-11-22 17:29:31,645 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4550, loss[loss=0.04953, simple_loss=0.06248, pruned_loss=0.007948, audio_tagging_loss=0.01034, over 15995.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09458, pruned_loss=0.01486, audio_tagging_loss=0.009139, over 3047771.77 frames. ], batch size: 61, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:29:36,700 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305150 2023-11-22 17:29:52,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2034360.0, ans=0.125 2023-11-22 17:30:12,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.74 vs. limit=15.0 2023-11-22 17:30:13,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2034493.3333333333, ans=0.125 2023-11-22 17:30:21,217 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 17:30:28,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2034560.0, ans=0.125 2023-11-22 17:30:34,631 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4600, loss[loss=0.07337, simple_loss=0.09923, pruned_loss=0.0148, audio_tagging_loss=0.008952, over 15517.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09388, pruned_loss=0.01483, audio_tagging_loss=0.009286, over 3048548.89 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:30:39,743 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305200 2023-11-22 17:30:42,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2034626.6666666667, ans=0.125 2023-11-22 17:31:06,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2034760.0, ans=0.1 2023-11-22 17:31:11,479 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.052e+01 8.212e+01 8.747e+01 9.317e+01 1.226e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 17:31:37,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2034960.0, ans=0.0 2023-11-22 17:31:38,391 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4650, loss[loss=0.08542, simple_loss=0.1101, pruned_loss=0.02198, audio_tagging_loss=0.008404, over 14673.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.09423, pruned_loss=0.015, audio_tagging_loss=0.00936, over 3048888.45 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:31:43,268 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305250 2023-11-22 17:31:51,269 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.81 vs. limit=10.0 2023-11-22 17:31:53,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten.whitening_limit, batch_count=2035026.6666666667, ans=22.5 2023-11-22 17:32:06,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2035093.3333333333, ans=0.125 2023-11-22 17:32:18,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2035160.0, ans=0.1 2023-11-22 17:32:28,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.80 vs. limit=15.0 2023-11-22 17:32:32,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.98 vs. limit=12.0 2023-11-22 17:32:43,681 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4700, loss[loss=0.07931, simple_loss=0.103, pruned_loss=0.01724, audio_tagging_loss=0.01058, over 15567.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09369, pruned_loss=0.01487, audio_tagging_loss=0.009406, over 3051983.13 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:32:47,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.10 vs. limit=15.0 2023-11-22 17:32:49,363 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305300 2023-11-22 17:32:49,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2035293.3333333333, ans=0.0 2023-11-22 17:33:10,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2035426.6666666667, ans=22.5 2023-11-22 17:33:13,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2035426.6666666667, ans=0.0 2023-11-22 17:33:14,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2035426.6666666667, ans=0.0 2023-11-22 17:33:18,415 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.537e+01 8.112e+01 8.766e+01 9.442e+01 1.103e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-22 17:33:22,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2035493.3333333333, ans=0.0 2023-11-22 17:33:28,792 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.21 vs. limit=22.5 2023-11-22 17:33:47,866 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4750, loss[loss=0.07177, simple_loss=0.0994, pruned_loss=0.01357, audio_tagging_loss=0.008499, over 15195.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09322, pruned_loss=0.01462, audio_tagging_loss=0.009367, over 3052213.21 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:33:52,957 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305350 2023-11-22 17:34:06,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2035693.3333333333, ans=0.0 2023-11-22 17:34:30,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2035826.6666666667, ans=0.1 2023-11-22 17:34:51,775 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4800, loss[loss=0.09151, simple_loss=0.1181, pruned_loss=0.02319, audio_tagging_loss=0.009254, over 14332.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09369, pruned_loss=0.01489, audio_tagging_loss=0.009488, over 3058121.67 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 32.0 2023-11-22 17:34:53,796 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.63 vs. limit=6.0 2023-11-22 17:34:56,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305400 2023-11-22 17:34:59,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2035960.0, ans=0.2 2023-11-22 17:35:04,376 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.49 vs. limit=15.0 2023-11-22 17:35:16,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2036026.6666666667, ans=0.0 2023-11-22 17:35:28,860 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.281e+01 8.887e+01 9.672e+01 1.150e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 17:35:37,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2036160.0, ans=0.2 2023-11-22 17:35:38,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2036160.0, ans=0.2 2023-11-22 17:35:57,039 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4850, loss[loss=0.06814, simple_loss=0.08003, pruned_loss=0.01514, audio_tagging_loss=0.01299, over 16389.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09488, pruned_loss=0.01506, audio_tagging_loss=0.009452, over 3058808.65 frames. ], batch size: 63, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:36:02,692 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305450 2023-11-22 17:36:15,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2036360.0, ans=0.0 2023-11-22 17:36:59,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2036626.6666666667, ans=0.125 2023-11-22 17:37:01,312 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4900, loss[loss=0.0866, simple_loss=0.1115, pruned_loss=0.02402, audio_tagging_loss=0.006841, over 15503.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09428, pruned_loss=0.01489, audio_tagging_loss=0.009434, over 3054718.31 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0 2023-11-22 17:37:05,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2036626.6666666667, ans=0.125 2023-11-22 17:37:06,343 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305500 2023-11-22 17:37:09,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2036626.6666666667, ans=0.125 2023-11-22 17:37:10,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2036626.6666666667, ans=0.125 2023-11-22 17:37:16,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2036693.3333333333, ans=0.125 2023-11-22 17:37:24,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2036760.0, ans=0.125 2023-11-22 17:37:26,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0 2023-11-22 17:37:38,236 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.464e+01 8.092e+01 8.921e+01 9.687e+01 1.263e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 17:37:39,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2036826.6666666667, ans=0.125 2023-11-22 17:37:42,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2036826.6666666667, ans=0.125 2023-11-22 17:38:02,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2036893.3333333333, ans=0.125 2023-11-22 17:38:04,876 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 4950, loss[loss=0.07082, simple_loss=0.1003, pruned_loss=0.01384, audio_tagging_loss=0.006813, over 15774.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09344, pruned_loss=0.01487, audio_tagging_loss=0.009336, over 3038199.76 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:38:09,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305550 2023-11-22 17:38:14,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2036960.0, ans=0.125 2023-11-22 17:38:18,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2037026.6666666667, ans=0.125 2023-11-22 17:38:23,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2037026.6666666667, ans=0.0 2023-11-22 17:38:25,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.68 vs. limit=15.0 2023-11-22 17:38:26,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2037026.6666666667, ans=0.125 2023-11-22 17:39:10,219 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5000, loss[loss=0.09457, simple_loss=0.1319, pruned_loss=0.02169, audio_tagging_loss=0.006929, over 15590.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09213, pruned_loss=0.01465, audio_tagging_loss=0.00932, over 3039621.15 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:39:12,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2037293.3333333333, ans=0.0 2023-11-22 17:39:15,792 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305600 2023-11-22 17:39:39,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.15 vs. limit=15.0 2023-11-22 17:39:47,352 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.198e+01 8.797e+01 9.538e+01 1.250e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 17:39:55,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2037493.3333333333, ans=0.0 2023-11-22 17:39:57,405 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.07 vs. limit=15.0 2023-11-22 17:40:14,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2037626.6666666667, ans=0.04949747468305833 2023-11-22 17:40:15,617 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5050, loss[loss=0.07737, simple_loss=0.09989, pruned_loss=0.01866, audio_tagging_loss=0.008767, over 14190.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09346, pruned_loss=0.01495, audio_tagging_loss=0.009164, over 3039043.78 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:40:21,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305650 2023-11-22 17:40:47,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2037693.3333333333, ans=0.2 2023-11-22 17:40:53,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2037760.0, ans=0.0 2023-11-22 17:41:46,544 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5100, loss[loss=0.06816, simple_loss=0.09459, pruned_loss=0.01229, audio_tagging_loss=0.008578, over 14621.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09342, pruned_loss=0.01495, audio_tagging_loss=0.009032, over 3033942.28 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 8.0 2023-11-22 17:41:54,026 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305700 2023-11-22 17:41:59,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2037960.0, ans=0.125 2023-11-22 17:42:05,704 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.63 vs. limit=15.0 2023-11-22 17:42:14,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2038026.6666666667, ans=0.0 2023-11-22 17:42:16,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2038026.6666666667, ans=0.015 2023-11-22 17:42:38,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2038093.3333333333, ans=0.2 2023-11-22 17:42:41,740 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.755e+01 8.235e+01 8.784e+01 9.418e+01 1.512e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 17:43:06,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2038226.6666666667, ans=0.0 2023-11-22 17:43:14,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2038226.6666666667, ans=0.0 2023-11-22 17:43:19,588 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5150, loss[loss=0.08556, simple_loss=0.119, pruned_loss=0.01798, audio_tagging_loss=0.008104, over 14717.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09356, pruned_loss=0.0149, audio_tagging_loss=0.009105, over 3028059.11 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 8.0 2023-11-22 17:43:23,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2038293.3333333333, ans=0.2 2023-11-22 17:43:27,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305750 2023-11-22 17:44:07,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2038426.6666666667, ans=0.0 2023-11-22 17:44:07,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2038426.6666666667, ans=0.2 2023-11-22 17:44:24,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2038493.3333333333, ans=0.0 2023-11-22 17:44:39,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2038560.0, ans=0.125 2023-11-22 17:44:46,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2038560.0, ans=0.125 2023-11-22 17:44:49,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.66 vs. limit=15.0 2023-11-22 17:44:52,037 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5200, loss[loss=0.05377, simple_loss=0.06774, pruned_loss=0.01133, audio_tagging_loss=0.008566, over 14956.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09304, pruned_loss=0.01491, audio_tagging_loss=0.009105, over 3024592.28 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:44:52,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2038626.6666666667, ans=0.125 2023-11-22 17:44:59,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305800 2023-11-22 17:45:32,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2038760.0, ans=0.1 2023-11-22 17:45:48,140 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.015e+01 8.193e+01 8.836e+01 9.389e+01 1.241e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 17:46:16,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.03 vs. limit=15.0 2023-11-22 17:46:21,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2038893.3333333333, ans=0.1 2023-11-22 17:46:23,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2038960.0, ans=0.125 2023-11-22 17:46:24,763 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5250, loss[loss=0.07851, simple_loss=0.1122, pruned_loss=0.01443, audio_tagging_loss=0.007973, over 15428.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09421, pruned_loss=0.01512, audio_tagging_loss=0.00904, over 3034367.96 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:46:26,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2038960.0, ans=0.125 2023-11-22 17:46:32,225 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305850 2023-11-22 17:47:18,953 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.24 vs. limit=22.5 2023-11-22 17:47:56,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2039293.3333333333, ans=0.125 2023-11-22 17:47:57,883 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5300, loss[loss=0.06045, simple_loss=0.08349, pruned_loss=0.008711, audio_tagging_loss=0.009993, over 15736.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09396, pruned_loss=0.01506, audio_tagging_loss=0.009091, over 3032199.47 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:48:04,100 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.05 vs. limit=22.5 2023-11-22 17:48:05,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305900 2023-11-22 17:48:05,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2039293.3333333333, ans=0.125 2023-11-22 17:48:35,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.57 vs. limit=15.0 2023-11-22 17:48:53,023 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.119e+01 8.483e+01 8.855e+01 9.483e+01 1.164e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-22 17:49:16,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2039560.0, ans=0.07 2023-11-22 17:49:30,737 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5350, loss[loss=0.06628, simple_loss=0.09057, pruned_loss=0.01269, audio_tagging_loss=0.008308, over 15276.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09462, pruned_loss=0.01521, audio_tagging_loss=0.009083, over 3028921.23 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:49:36,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2039626.6666666667, ans=0.125 2023-11-22 17:49:38,167 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 305950 2023-11-22 17:50:07,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2039760.0, ans=0.0 2023-11-22 17:50:38,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2039826.6666666667, ans=0.2 2023-11-22 17:51:03,652 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5400, loss[loss=0.08545, simple_loss=0.1152, pruned_loss=0.02152, audio_tagging_loss=0.006305, over 16174.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09411, pruned_loss=0.01493, audio_tagging_loss=0.009113, over 3035703.46 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:51:11,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306000 2023-11-22 17:51:13,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2039960.0, ans=0.0 2023-11-22 17:51:17,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.11 vs. limit=15.0 2023-11-22 17:51:52,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2040093.3333333333, ans=0.0 2023-11-22 17:51:59,431 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.288e+01 8.843e+01 9.633e+01 1.187e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 17:51:59,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2040160.0, ans=0.125 2023-11-22 17:52:27,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2040226.6666666667, ans=0.95 2023-11-22 17:52:36,547 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5450, loss[loss=0.07106, simple_loss=0.08676, pruned_loss=0.01653, audio_tagging_loss=0.01116, over 15146.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09401, pruned_loss=0.01485, audio_tagging_loss=0.009233, over 3034015.80 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:52:44,756 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306050 2023-11-22 17:52:52,749 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.56 vs. limit=15.0 2023-11-22 17:53:25,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2040426.6666666667, ans=0.125 2023-11-22 17:53:32,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2040493.3333333333, ans=0.1 2023-11-22 17:54:10,068 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5500, loss[loss=0.06483, simple_loss=0.09568, pruned_loss=0.01163, audio_tagging_loss=0.005363, over 15741.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.09446, pruned_loss=0.01498, audio_tagging_loss=0.009286, over 3040072.46 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:54:17,293 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306100 2023-11-22 17:54:20,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2040626.6666666667, ans=0.0 2023-11-22 17:54:20,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2040626.6666666667, ans=0.2 2023-11-22 17:54:47,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2040760.0, ans=0.0 2023-11-22 17:54:56,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0 2023-11-22 17:55:00,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2040760.0, ans=0.0 2023-11-22 17:55:05,353 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.963e+01 8.451e+01 8.930e+01 9.689e+01 1.282e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 17:55:17,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2040826.6666666667, ans=0.0 2023-11-22 17:55:25,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.95 vs. limit=12.0 2023-11-22 17:55:42,889 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5550, loss[loss=0.08488, simple_loss=0.1169, pruned_loss=0.01863, audio_tagging_loss=0.007793, over 14814.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09459, pruned_loss=0.0151, audio_tagging_loss=0.009327, over 3042957.73 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:55:45,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2040960.0, ans=0.0 2023-11-22 17:55:50,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306150 2023-11-22 17:56:05,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2041026.6666666667, ans=0.125 2023-11-22 17:56:14,979 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.92 vs. limit=22.5 2023-11-22 17:56:40,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2041160.0, ans=0.09899494936611666 2023-11-22 17:57:08,384 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.89 vs. limit=15.0 2023-11-22 17:57:14,861 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5600, loss[loss=0.09098, simple_loss=0.1114, pruned_loss=0.02605, audio_tagging_loss=0.009247, over 15960.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09479, pruned_loss=0.01504, audio_tagging_loss=0.009346, over 3041774.05 frames. ], batch size: 61, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 17:57:22,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306200 2023-11-22 17:57:30,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2041293.3333333333, ans=0.2 2023-11-22 17:57:41,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2041360.0, ans=0.1 2023-11-22 17:58:11,312 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.138e+01 8.696e+01 9.555e+01 1.151e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 17:58:19,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2041493.3333333333, ans=0.2 2023-11-22 17:58:20,017 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 17:58:21,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2041493.3333333333, ans=0.0 2023-11-22 17:58:31,786 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 17:58:38,241 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5650, loss[loss=0.06933, simple_loss=0.09591, pruned_loss=0.01245, audio_tagging_loss=0.008924, over 15808.00 frames. ], tot_loss[loss=0.072, simple_loss=0.095, pruned_loss=0.01509, audio_tagging_loss=0.009406, over 3045459.97 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 17:58:43,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306250 2023-11-22 17:59:18,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2041826.6666666667, ans=0.125 2023-11-22 17:59:24,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2041826.6666666667, ans=0.0 2023-11-22 17:59:29,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2041893.3333333333, ans=0.0 2023-11-22 17:59:42,497 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5700, loss[loss=0.06258, simple_loss=0.08376, pruned_loss=0.01111, audio_tagging_loss=0.009595, over 16205.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09402, pruned_loss=0.01479, audio_tagging_loss=0.009454, over 3044360.67 frames. ], batch size: 62, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 17:59:47,621 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306300 2023-11-22 17:59:53,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2042026.6666666667, ans=0.0 2023-11-22 18:00:18,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2042093.3333333333, ans=0.1 2023-11-22 18:00:22,596 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.203e+01 8.769e+01 9.418e+01 1.173e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 18:00:46,246 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5750, loss[loss=0.05674, simple_loss=0.06596, pruned_loss=0.01351, audio_tagging_loss=0.01025, over 13630.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09314, pruned_loss=0.01461, audio_tagging_loss=0.009424, over 3037942.74 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:00:51,435 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306350 2023-11-22 18:01:17,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2042426.6666666667, ans=0.0 2023-11-22 18:01:25,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2042493.3333333333, ans=0.125 2023-11-22 18:01:28,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=15.0 2023-11-22 18:01:51,690 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5800, loss[loss=0.08243, simple_loss=0.1147, pruned_loss=0.01751, audio_tagging_loss=0.007592, over 14973.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09393, pruned_loss=0.01464, audio_tagging_loss=0.009277, over 3049078.56 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:01:57,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306400 2023-11-22 18:02:05,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2042693.3333333333, ans=0.125 2023-11-22 18:02:31,134 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.101e+01 8.686e+01 9.623e+01 1.169e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-22 18:02:56,990 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5850, loss[loss=0.07824, simple_loss=0.1038, pruned_loss=0.01638, audio_tagging_loss=0.00998, over 15866.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09335, pruned_loss=0.01456, audio_tagging_loss=0.0093, over 3051389.72 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:03:02,057 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306450 2023-11-22 18:03:26,804 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0 2023-11-22 18:03:29,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2043093.3333333333, ans=0.1 2023-11-22 18:03:35,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2043160.0, ans=0.125 2023-11-22 18:03:35,515 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.05 vs. limit=15.0 2023-11-22 18:03:42,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.84 vs. limit=15.0 2023-11-22 18:04:06,862 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5900, loss[loss=0.0658, simple_loss=0.08308, pruned_loss=0.01691, audio_tagging_loss=0.007349, over 15469.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09407, pruned_loss=0.01468, audio_tagging_loss=0.009266, over 3055517.90 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:04:09,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2043293.3333333333, ans=0.1 2023-11-22 18:04:09,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2043293.3333333333, ans=0.125 2023-11-22 18:04:13,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306500 2023-11-22 18:04:42,941 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.52 vs. limit=15.0 2023-11-22 18:04:44,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2043426.6666666667, ans=0.125 2023-11-22 18:04:50,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2043426.6666666667, ans=0.04949747468305833 2023-11-22 18:05:02,417 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.183e+01 8.699e+01 9.524e+01 1.128e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 18:05:10,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.62 vs. limit=22.5 2023-11-22 18:05:27,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2043560.0, ans=0.1 2023-11-22 18:05:35,779 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 5950, loss[loss=0.07201, simple_loss=0.09827, pruned_loss=0.01292, audio_tagging_loss=0.009951, over 14695.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09362, pruned_loss=0.01471, audio_tagging_loss=0.009169, over 3052055.78 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:05:43,024 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306550 2023-11-22 18:05:48,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2043626.6666666667, ans=0.2 2023-11-22 18:06:22,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2043760.0, ans=0.2 2023-11-22 18:06:29,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2043826.6666666667, ans=0.95 2023-11-22 18:07:05,383 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6000, loss[loss=0.09809, simple_loss=0.1332, pruned_loss=0.0236, audio_tagging_loss=0.007869, over 15714.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09342, pruned_loss=0.01455, audio_tagging_loss=0.009091, over 3051392.80 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:07:05,385 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 18:07:21,599 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.3301, 5.1066, 4.5483, 5.0125], device='cuda:3') 2023-11-22 18:07:56,206 INFO [train_asr.py:1253] (3/4) Epoch 26, validation: loss=0.05819, simple_loss=0.05149, pruned_loss=0.005105, audio_tagging_loss=0.02734, over 4681554.00 frames. 2023-11-22 18:07:56,207 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 18:08:01,692 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306600 2023-11-22 18:08:03,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2043960.0, ans=0.125 2023-11-22 18:08:12,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.59 vs. limit=15.0 2023-11-22 18:08:17,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2044026.6666666667, ans=0.125 2023-11-22 18:08:36,923 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.220e+01 8.809e+01 9.457e+01 1.359e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 18:08:37,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2044160.0, ans=0.2 2023-11-22 18:08:42,600 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.64 vs. limit=15.0 2023-11-22 18:08:43,041 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 18:08:45,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2044160.0, ans=0.07 2023-11-22 18:08:48,173 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:09:01,886 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6050, loss[loss=0.07715, simple_loss=0.1145, pruned_loss=0.01253, audio_tagging_loss=0.007376, over 15919.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.0927, pruned_loss=0.0144, audio_tagging_loss=0.009105, over 3049184.20 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:09:06,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306650 2023-11-22 18:09:17,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2044360.0, ans=0.125 2023-11-22 18:09:42,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2044493.3333333333, ans=0.125 2023-11-22 18:10:04,706 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6100, loss[loss=0.07297, simple_loss=0.09308, pruned_loss=0.0141, audio_tagging_loss=0.01233, over 15145.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09256, pruned_loss=0.01436, audio_tagging_loss=0.009155, over 3041174.55 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:10:10,220 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306700 2023-11-22 18:10:20,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2044693.3333333333, ans=0.07 2023-11-22 18:10:25,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2044693.3333333333, ans=0.1 2023-11-22 18:10:28,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2044760.0, ans=0.0 2023-11-22 18:10:41,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0 2023-11-22 18:10:42,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2044826.6666666667, ans=0.025 2023-11-22 18:10:46,194 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.417e+01 8.381e+01 8.850e+01 9.348e+01 1.200e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-22 18:10:59,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2044893.3333333333, ans=0.2 2023-11-22 18:11:08,985 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6150, loss[loss=0.0699, simple_loss=0.08738, pruned_loss=0.0129, audio_tagging_loss=0.01331, over 15572.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09312, pruned_loss=0.0147, audio_tagging_loss=0.009186, over 3037017.74 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:11:13,855 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306750 2023-11-22 18:11:14,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2044960.0, ans=0.2 2023-11-22 18:11:16,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2044960.0, ans=0.0 2023-11-22 18:11:17,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2044960.0, ans=0.125 2023-11-22 18:11:21,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2045026.6666666667, ans=0.1 2023-11-22 18:11:49,624 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.96 vs. limit=15.0 2023-11-22 18:12:13,092 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6200, loss[loss=0.07672, simple_loss=0.1102, pruned_loss=0.01187, audio_tagging_loss=0.009752, over 14754.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.093, pruned_loss=0.01463, audio_tagging_loss=0.009201, over 3036924.57 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:12:18,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306800 2023-11-22 18:12:32,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2045360.0, ans=0.0 2023-11-22 18:12:53,120 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.91 vs. limit=15.0 2023-11-22 18:12:53,572 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.167e+01 8.049e+01 8.588e+01 9.515e+01 1.238e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-22 18:13:17,029 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6250, loss[loss=0.08196, simple_loss=0.1126, pruned_loss=0.01553, audio_tagging_loss=0.01015, over 16070.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.093, pruned_loss=0.01466, audio_tagging_loss=0.0093, over 3047136.02 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:13:17,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2045626.6666666667, ans=0.5 2023-11-22 18:13:21,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2045626.6666666667, ans=0.125 2023-11-22 18:13:22,083 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306850 2023-11-22 18:13:22,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.90 vs. limit=15.0 2023-11-22 18:13:36,153 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:13:42,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2045760.0, ans=0.2 2023-11-22 18:13:48,271 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0 2023-11-22 18:14:21,615 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6300, loss[loss=0.05896, simple_loss=0.06318, pruned_loss=0.01572, audio_tagging_loss=0.01164, over 14487.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09349, pruned_loss=0.01463, audio_tagging_loss=0.009303, over 3049794.45 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:14:22,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2045960.0, ans=0.015 2023-11-22 18:14:26,534 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306900 2023-11-22 18:14:35,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2046026.6666666667, ans=0.1 2023-11-22 18:14:39,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2046026.6666666667, ans=0.95 2023-11-22 18:14:48,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2046093.3333333333, ans=0.0 2023-11-22 18:14:51,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2046093.3333333333, ans=0.1 2023-11-22 18:14:59,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2046160.0, ans=0.0 2023-11-22 18:15:02,394 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.359e+01 8.414e+01 9.060e+01 9.921e+01 1.444e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-22 18:15:08,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2046160.0, ans=0.0 2023-11-22 18:15:25,750 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6350, loss[loss=0.08689, simple_loss=0.111, pruned_loss=0.02119, audio_tagging_loss=0.01021, over 14388.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09352, pruned_loss=0.01465, audio_tagging_loss=0.009449, over 3050579.12 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:15:27,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2046293.3333333333, ans=0.5 2023-11-22 18:15:30,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2046293.3333333333, ans=0.125 2023-11-22 18:15:31,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 306950 2023-11-22 18:15:33,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.18 vs. limit=22.5 2023-11-22 18:15:44,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2046360.0, ans=0.0 2023-11-22 18:15:57,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2046426.6666666667, ans=0.125 2023-11-22 18:16:07,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2046493.3333333333, ans=0.2 2023-11-22 18:16:29,690 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6400, loss[loss=0.07222, simple_loss=0.1001, pruned_loss=0.01573, audio_tagging_loss=0.006425, over 14904.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09337, pruned_loss=0.0146, audio_tagging_loss=0.009568, over 3051528.17 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:16:34,588 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307000 2023-11-22 18:16:40,381 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.73 vs. limit=12.0 2023-11-22 18:16:50,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2046693.3333333333, ans=0.0 2023-11-22 18:16:56,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2046760.0, ans=0.1 2023-11-22 18:16:57,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2046760.0, ans=0.0 2023-11-22 18:17:03,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.65 vs. limit=6.0 2023-11-22 18:17:10,982 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.820e+01 8.418e+01 9.159e+01 1.026e+02 1.241e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-22 18:17:33,072 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6450, loss[loss=0.0778, simple_loss=0.09453, pruned_loss=0.01998, audio_tagging_loss=0.01055, over 14984.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09308, pruned_loss=0.01465, audio_tagging_loss=0.009651, over 3050639.36 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:17:38,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307050 2023-11-22 18:17:42,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2046960.0, ans=0.1 2023-11-22 18:18:36,792 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6500, loss[loss=0.06974, simple_loss=0.09002, pruned_loss=0.01724, audio_tagging_loss=0.007495, over 14461.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09372, pruned_loss=0.01479, audio_tagging_loss=0.009519, over 3049131.03 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:18:42,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307100 2023-11-22 18:18:52,967 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:19:11,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2047426.6666666667, ans=0.2 2023-11-22 18:19:19,259 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.186e+01 8.231e+01 8.746e+01 9.661e+01 1.657e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 18:19:39,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2047560.0, ans=0.0 2023-11-22 18:19:41,396 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6550, loss[loss=0.0734, simple_loss=0.09527, pruned_loss=0.01743, audio_tagging_loss=0.00834, over 14414.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09356, pruned_loss=0.01471, audio_tagging_loss=0.009438, over 3049007.75 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:19:42,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2047626.6666666667, ans=0.125 2023-11-22 18:19:43,374 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0 2023-11-22 18:19:44,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2047626.6666666667, ans=0.1 2023-11-22 18:19:46,417 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307150 2023-11-22 18:19:52,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2047693.3333333333, ans=0.125 2023-11-22 18:19:56,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2047693.3333333333, ans=0.0 2023-11-22 18:19:56,425 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:19:57,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2047693.3333333333, ans=0.0 2023-11-22 18:20:19,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2047826.6666666667, ans=10.0 2023-11-22 18:20:34,009 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:20:43,127 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.43 vs. limit=15.0 2023-11-22 18:20:44,975 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6600, loss[loss=0.0743, simple_loss=0.09988, pruned_loss=0.01631, audio_tagging_loss=0.008051, over 14471.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09366, pruned_loss=0.01456, audio_tagging_loss=0.009362, over 3042219.63 frames. ], batch size: 53, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:20:50,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307200 2023-11-22 18:20:53,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0 2023-11-22 18:21:10,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2048026.6666666667, ans=0.0 2023-11-22 18:21:18,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2048093.3333333333, ans=0.2 2023-11-22 18:21:19,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2048093.3333333333, ans=0.5 2023-11-22 18:21:22,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2048093.3333333333, ans=0.125 2023-11-22 18:21:27,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.316e+01 8.739e+01 9.394e+01 1.270e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 18:21:36,496 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:21:37,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=19.92 vs. limit=22.5 2023-11-22 18:21:39,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2048226.6666666667, ans=0.035 2023-11-22 18:21:42,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2048226.6666666667, ans=0.0 2023-11-22 18:21:49,012 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6650, loss[loss=0.07619, simple_loss=0.09953, pruned_loss=0.01739, audio_tagging_loss=0.009029, over 14098.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09272, pruned_loss=0.01457, audio_tagging_loss=0.009319, over 3040598.26 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:21:49,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2048293.3333333333, ans=0.125 2023-11-22 18:21:54,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307250 2023-11-22 18:22:17,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2048426.6666666667, ans=0.125 2023-11-22 18:22:18,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.89 vs. limit=15.0 2023-11-22 18:22:27,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2048493.3333333333, ans=0.0 2023-11-22 18:22:30,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2048493.3333333333, ans=0.125 2023-11-22 18:22:32,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2048493.3333333333, ans=0.125 2023-11-22 18:22:43,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2048560.0, ans=0.2 2023-11-22 18:22:46,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2048560.0, ans=0.125 2023-11-22 18:22:53,625 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6700, loss[loss=0.06007, simple_loss=0.0869, pruned_loss=0.009425, audio_tagging_loss=0.007196, over 15436.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09233, pruned_loss=0.01452, audio_tagging_loss=0.009229, over 3042411.63 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:22:58,514 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307300 2023-11-22 18:23:18,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2048760.0, ans=0.125 2023-11-22 18:23:21,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2048760.0, ans=0.125 2023-11-22 18:23:35,468 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.221e+01 9.103e+01 9.902e+01 1.404e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-22 18:23:38,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2048826.6666666667, ans=0.125 2023-11-22 18:23:38,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2048826.6666666667, ans=0.2 2023-11-22 18:23:54,707 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.70 vs. limit=10.0 2023-11-22 18:23:56,348 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6750, loss[loss=0.07356, simple_loss=0.1078, pruned_loss=0.01311, audio_tagging_loss=0.006558, over 14846.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09301, pruned_loss=0.01455, audio_tagging_loss=0.009262, over 3045554.21 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:24:01,400 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307350 2023-11-22 18:24:07,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2049026.6666666667, ans=0.125 2023-11-22 18:24:09,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.26 vs. limit=15.0 2023-11-22 18:24:25,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2049093.3333333333, ans=0.0 2023-11-22 18:24:32,107 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0 2023-11-22 18:24:37,791 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:24:45,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2049160.0, ans=0.1 2023-11-22 18:24:52,582 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.57 vs. limit=15.0 2023-11-22 18:24:57,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2049226.6666666667, ans=0.125 2023-11-22 18:24:59,208 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6800, loss[loss=0.08538, simple_loss=0.1124, pruned_loss=0.02337, audio_tagging_loss=0.005794, over 15755.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09426, pruned_loss=0.01475, audio_tagging_loss=0.009197, over 3053843.94 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:25:04,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307400 2023-11-22 18:25:08,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2049293.3333333333, ans=0.125 2023-11-22 18:25:40,525 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.331e+01 8.948e+01 9.524e+01 1.247e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-22 18:25:45,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2049493.3333333333, ans=0.2 2023-11-22 18:25:55,689 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.69 vs. limit=15.0 2023-11-22 18:26:03,618 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6850, loss[loss=0.07249, simple_loss=0.09681, pruned_loss=0.01676, audio_tagging_loss=0.007326, over 16317.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09386, pruned_loss=0.0147, audio_tagging_loss=0.009279, over 3049122.65 frames. ], batch size: 63, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:26:03,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2049626.6666666667, ans=0.2 2023-11-22 18:26:08,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307450 2023-11-22 18:26:11,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2049626.6666666667, ans=0.2 2023-11-22 18:26:22,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0 2023-11-22 18:26:38,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2049826.6666666667, ans=0.04949747468305833 2023-11-22 18:26:55,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2049893.3333333333, ans=0.125 2023-11-22 18:27:06,610 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6900, loss[loss=0.08326, simple_loss=0.1078, pruned_loss=0.02008, audio_tagging_loss=0.009292, over 14563.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09391, pruned_loss=0.01481, audio_tagging_loss=0.00924, over 3050784.88 frames. ], batch size: 53, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:27:11,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307500 2023-11-22 18:27:12,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2049960.0, ans=0.1 2023-11-22 18:27:27,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2050026.6666666667, ans=0.2 2023-11-22 18:27:49,683 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.175e+01 8.211e+01 8.819e+01 9.704e+01 1.350e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 18:27:56,108 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 18:27:59,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2050226.6666666667, ans=0.125 2023-11-22 18:28:05,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2050226.6666666667, ans=0.0 2023-11-22 18:28:11,158 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 6950, loss[loss=0.05152, simple_loss=0.06038, pruned_loss=0.01038, audio_tagging_loss=0.01095, over 15875.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09444, pruned_loss=0.01502, audio_tagging_loss=0.009196, over 3050241.89 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:28:16,388 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307550 2023-11-22 18:28:22,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2050293.3333333333, ans=0.125 2023-11-22 18:28:24,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2050360.0, ans=0.125 2023-11-22 18:28:48,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2050426.6666666667, ans=0.0 2023-11-22 18:29:03,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2050560.0, ans=0.125 2023-11-22 18:29:17,063 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=12.0 2023-11-22 18:29:17,597 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7000, loss[loss=0.08413, simple_loss=0.1196, pruned_loss=0.01805, audio_tagging_loss=0.006276, over 14490.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09326, pruned_loss=0.01482, audio_tagging_loss=0.009241, over 3040791.72 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:29:20,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2050626.6666666667, ans=0.125 2023-11-22 18:29:23,224 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307600 2023-11-22 18:29:26,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.29 vs. limit=15.0 2023-11-22 18:29:27,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2050626.6666666667, ans=0.125 2023-11-22 18:29:52,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2050760.0, ans=0.2 2023-11-22 18:29:59,743 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 8.302e+01 8.887e+01 9.651e+01 1.249e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 18:30:22,297 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7050, loss[loss=0.07664, simple_loss=0.1027, pruned_loss=0.01484, audio_tagging_loss=0.01046, over 15172.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09278, pruned_loss=0.01467, audio_tagging_loss=0.009333, over 3042521.03 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:30:23,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2050960.0, ans=0.1 2023-11-22 18:30:26,674 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=14.65 vs. limit=15.0 2023-11-22 18:30:27,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307650 2023-11-22 18:30:36,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2051026.6666666667, ans=0.125 2023-11-22 18:31:03,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2051160.0, ans=0.05 2023-11-22 18:31:16,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2051226.6666666667, ans=0.125 2023-11-22 18:31:27,264 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7100, loss[loss=0.07299, simple_loss=0.0962, pruned_loss=0.0166, audio_tagging_loss=0.008293, over 15478.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09322, pruned_loss=0.01475, audio_tagging_loss=0.009368, over 3044283.90 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:31:32,411 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307700 2023-11-22 18:32:11,640 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.336e+01 9.020e+01 9.845e+01 2.040e+02, threshold=1.804e+02, percent-clipped=1.0 2023-11-22 18:32:32,726 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7150, loss[loss=0.096, simple_loss=0.1224, pruned_loss=0.02675, audio_tagging_loss=0.008051, over 15612.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09333, pruned_loss=0.01478, audio_tagging_loss=0.009378, over 3041162.42 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0 2023-11-22 18:32:38,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307750 2023-11-22 18:32:51,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2051693.3333333333, ans=0.125 2023-11-22 18:32:53,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2051693.3333333333, ans=0.125 2023-11-22 18:32:57,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2051760.0, ans=0.0 2023-11-22 18:33:01,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2051760.0, ans=0.125 2023-11-22 18:33:22,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.05 vs. limit=15.0 2023-11-22 18:33:29,818 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:33:36,862 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7200, loss[loss=0.08834, simple_loss=0.1153, pruned_loss=0.02192, audio_tagging_loss=0.00878, over 14136.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09372, pruned_loss=0.01482, audio_tagging_loss=0.009374, over 3041269.75 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 32.0 2023-11-22 18:33:42,469 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307800 2023-11-22 18:33:47,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2051960.0, ans=0.125 2023-11-22 18:33:48,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.36 vs. limit=15.0 2023-11-22 18:33:56,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2052026.6666666667, ans=0.125 2023-11-22 18:34:10,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2052093.3333333333, ans=0.125 2023-11-22 18:34:18,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2052160.0, ans=0.125 2023-11-22 18:34:20,787 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.282e+01 8.868e+01 9.678e+01 1.341e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 18:34:25,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2052160.0, ans=0.125 2023-11-22 18:34:29,706 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.96 vs. limit=8.0 2023-11-22 18:34:30,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2052226.6666666667, ans=0.1 2023-11-22 18:34:33,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.68 vs. limit=15.0 2023-11-22 18:34:41,001 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7250, loss[loss=0.04861, simple_loss=0.06427, pruned_loss=0.006313, audio_tagging_loss=0.01017, over 14456.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09346, pruned_loss=0.01461, audio_tagging_loss=0.00942, over 3041568.84 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:34:46,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307850 2023-11-22 18:34:59,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.62 vs. limit=22.5 2023-11-22 18:35:20,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2052493.3333333333, ans=0.1 2023-11-22 18:35:26,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2052493.3333333333, ans=0.125 2023-11-22 18:35:43,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.07 vs. limit=15.0 2023-11-22 18:35:45,798 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7300, loss[loss=0.05482, simple_loss=0.06195, pruned_loss=0.01095, audio_tagging_loss=0.0129, over 14364.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09352, pruned_loss=0.01466, audio_tagging_loss=0.009333, over 3041547.91 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:35:51,260 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307900 2023-11-22 18:36:28,174 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.001e+01 8.613e+01 9.353e+01 1.295e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-22 18:36:49,238 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7350, loss[loss=0.07218, simple_loss=0.0914, pruned_loss=0.01866, audio_tagging_loss=0.007825, over 15653.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09331, pruned_loss=0.01483, audio_tagging_loss=0.00933, over 3055322.29 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:36:54,158 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 307950 2023-11-22 18:36:54,571 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.59 vs. limit=15.0 2023-11-22 18:37:08,656 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.32 vs. limit=15.0 2023-11-22 18:37:42,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0 2023-11-22 18:37:53,624 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7400, loss[loss=0.05623, simple_loss=0.07339, pruned_loss=0.008529, audio_tagging_loss=0.01101, over 13475.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09369, pruned_loss=0.01488, audio_tagging_loss=0.009198, over 3048590.74 frames. ], batch size: 52, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:37:58,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308000 2023-11-22 18:38:15,800 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:38:20,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2053360.0, ans=0.2 2023-11-22 18:38:33,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2053426.6666666667, ans=0.2 2023-11-22 18:38:40,999 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.189e+01 8.818e+01 9.680e+01 1.265e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 18:38:44,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2053493.3333333333, ans=0.1 2023-11-22 18:38:48,272 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:38:54,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2053560.0, ans=0.015 2023-11-22 18:39:02,138 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7450, loss[loss=0.08749, simple_loss=0.1173, pruned_loss=0.02196, audio_tagging_loss=0.006862, over 15876.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09413, pruned_loss=0.01499, audio_tagging_loss=0.009102, over 3054448.75 frames. ], batch size: 60, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:39:08,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308050 2023-11-22 18:39:16,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2053693.3333333333, ans=0.0 2023-11-22 18:39:30,727 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:39:42,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2053826.6666666667, ans=0.0 2023-11-22 18:40:07,066 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7500, loss[loss=0.06271, simple_loss=0.07817, pruned_loss=0.01528, audio_tagging_loss=0.00834, over 14305.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09399, pruned_loss=0.01495, audio_tagging_loss=0.009124, over 3049911.64 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:40:12,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308100 2023-11-22 18:40:22,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.93 vs. limit=22.5 2023-11-22 18:40:23,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2054026.6666666667, ans=0.125 2023-11-22 18:40:52,212 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.656e+01 8.267e+01 8.775e+01 9.244e+01 1.175e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-22 18:40:55,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.40 vs. limit=22.5 2023-11-22 18:41:10,634 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7550, loss[loss=0.07537, simple_loss=0.1027, pruned_loss=0.01501, audio_tagging_loss=0.00903, over 14547.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09363, pruned_loss=0.01494, audio_tagging_loss=0.009086, over 3045567.31 frames. ], batch size: 53, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:41:16,163 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308150 2023-11-22 18:41:18,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2054293.3333333333, ans=0.1 2023-11-22 18:41:36,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2054426.6666666667, ans=0.05 2023-11-22 18:41:41,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2054426.6666666667, ans=0.125 2023-11-22 18:41:58,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2054493.3333333333, ans=0.1 2023-11-22 18:42:02,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2054560.0, ans=0.125 2023-11-22 18:42:11,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2054560.0, ans=0.125 2023-11-22 18:42:14,488 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7600, loss[loss=0.07517, simple_loss=0.1001, pruned_loss=0.01607, audio_tagging_loss=0.009023, over 14655.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09428, pruned_loss=0.01498, audio_tagging_loss=0.009093, over 3055073.74 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:42:20,780 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308200 2023-11-22 18:42:22,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2054626.6666666667, ans=0.0 2023-11-22 18:42:38,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2054693.3333333333, ans=0.035 2023-11-22 18:42:52,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.28 vs. limit=15.0 2023-11-22 18:42:59,731 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 7.973e+01 8.478e+01 9.439e+01 1.214e+02, threshold=1.696e+02, percent-clipped=0.0 2023-11-22 18:43:02,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2054826.6666666667, ans=0.0 2023-11-22 18:43:20,423 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7650, loss[loss=0.06928, simple_loss=0.09703, pruned_loss=0.01434, audio_tagging_loss=0.006417, over 16164.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09337, pruned_loss=0.01481, audio_tagging_loss=0.009104, over 3050150.00 frames. ], batch size: 63, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:43:22,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2054960.0, ans=0.0 2023-11-22 18:43:25,659 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308250 2023-11-22 18:43:32,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.57 vs. limit=22.5 2023-11-22 18:43:54,027 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.24 vs. limit=15.0 2023-11-22 18:43:59,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2055160.0, ans=0.0 2023-11-22 18:44:18,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2055226.6666666667, ans=0.0 2023-11-22 18:44:19,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=2055226.6666666667, ans=0.02 2023-11-22 18:44:23,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2055293.3333333333, ans=0.0 2023-11-22 18:44:24,544 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7700, loss[loss=0.08239, simple_loss=0.1191, pruned_loss=0.01512, audio_tagging_loss=0.007697, over 16181.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09352, pruned_loss=0.01474, audio_tagging_loss=0.009133, over 3051950.02 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:44:29,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308300 2023-11-22 18:44:55,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2055426.6666666667, ans=0.125 2023-11-22 18:45:01,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.60 vs. limit=22.5 2023-11-22 18:45:02,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.38 vs. limit=12.0 2023-11-22 18:45:03,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2055493.3333333333, ans=0.2 2023-11-22 18:45:10,229 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.353e+01 9.009e+01 9.552e+01 1.417e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-22 18:45:29,287 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7750, loss[loss=0.05915, simple_loss=0.08363, pruned_loss=0.008335, audio_tagging_loss=0.008998, over 15567.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09301, pruned_loss=0.01469, audio_tagging_loss=0.009245, over 3050586.34 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:45:34,793 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308350 2023-11-22 18:46:02,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0 2023-11-22 18:46:04,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2055760.0, ans=0.0 2023-11-22 18:46:20,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2055893.3333333333, ans=0.125 2023-11-22 18:46:34,448 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7800, loss[loss=0.06934, simple_loss=0.08994, pruned_loss=0.01585, audio_tagging_loss=0.008514, over 14161.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09296, pruned_loss=0.01464, audio_tagging_loss=0.009312, over 3039504.49 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:46:34,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2055960.0, ans=0.05 2023-11-22 18:46:35,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2055960.0, ans=0.125 2023-11-22 18:46:39,341 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308400 2023-11-22 18:46:58,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2056093.3333333333, ans=0.1 2023-11-22 18:46:58,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2056093.3333333333, ans=0.125 2023-11-22 18:47:07,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2056093.3333333333, ans=0.1 2023-11-22 18:47:09,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2056093.3333333333, ans=0.125 2023-11-22 18:47:15,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2056160.0, ans=0.125 2023-11-22 18:47:21,060 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.003e+01 8.337e+01 8.829e+01 9.514e+01 1.389e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 18:47:27,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2056226.6666666667, ans=0.0 2023-11-22 18:47:38,097 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7850, loss[loss=0.07618, simple_loss=0.09683, pruned_loss=0.01819, audio_tagging_loss=0.00958, over 14805.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09251, pruned_loss=0.01462, audio_tagging_loss=0.009413, over 3039002.04 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:47:39,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2056293.3333333333, ans=0.0 2023-11-22 18:47:43,186 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308450 2023-11-22 18:48:01,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.97 vs. limit=15.0 2023-11-22 18:48:04,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=12.0 2023-11-22 18:48:06,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2056426.6666666667, ans=0.125 2023-11-22 18:48:09,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2056426.6666666667, ans=0.125 2023-11-22 18:48:10,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2056426.6666666667, ans=15.0 2023-11-22 18:48:13,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2056426.6666666667, ans=0.2 2023-11-22 18:48:27,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2056493.3333333333, ans=0.125 2023-11-22 18:48:41,471 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7900, loss[loss=0.06965, simple_loss=0.09621, pruned_loss=0.01358, audio_tagging_loss=0.007958, over 16203.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09281, pruned_loss=0.01466, audio_tagging_loss=0.009478, over 3035918.09 frames. ], batch size: 64, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:48:46,546 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308500 2023-11-22 18:48:46,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2056626.6666666667, ans=0.125 2023-11-22 18:49:27,225 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.747e+01 8.185e+01 8.852e+01 9.450e+01 1.416e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-22 18:49:28,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2056826.6666666667, ans=0.0 2023-11-22 18:49:40,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2056893.3333333333, ans=0.125 2023-11-22 18:49:46,623 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 7950, loss[loss=0.06237, simple_loss=0.0876, pruned_loss=0.009703, audio_tagging_loss=0.008864, over 15645.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09186, pruned_loss=0.01433, audio_tagging_loss=0.009554, over 3044595.76 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:49:51,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308550 2023-11-22 18:50:00,022 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 18:50:01,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2057026.6666666667, ans=0.04949747468305833 2023-11-22 18:50:11,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2057093.3333333333, ans=0.1 2023-11-22 18:50:36,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2057226.6666666667, ans=0.125 2023-11-22 18:50:49,761 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8000, loss[loss=0.07173, simple_loss=0.09914, pruned_loss=0.0142, audio_tagging_loss=0.007958, over 15721.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09163, pruned_loss=0.01417, audio_tagging_loss=0.009615, over 3044538.61 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:50:54,928 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308600 2023-11-22 18:50:59,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.98 vs. limit=15.0 2023-11-22 18:51:00,663 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.89 vs. limit=22.5 2023-11-22 18:51:36,564 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 8.209e+01 8.705e+01 9.523e+01 1.226e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-22 18:51:38,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2057493.3333333333, ans=0.125 2023-11-22 18:51:39,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2057493.3333333333, ans=0.04949747468305833 2023-11-22 18:51:53,699 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8050, loss[loss=0.08653, simple_loss=0.1244, pruned_loss=0.01727, audio_tagging_loss=0.007069, over 14645.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09198, pruned_loss=0.01421, audio_tagging_loss=0.00964, over 3049497.42 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:51:58,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308650 2023-11-22 18:52:26,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2057760.0, ans=0.125 2023-11-22 18:52:37,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2057826.6666666667, ans=0.1 2023-11-22 18:52:41,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2057826.6666666667, ans=0.0 2023-11-22 18:52:59,051 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8100, loss[loss=0.08663, simple_loss=0.1218, pruned_loss=0.02016, audio_tagging_loss=0.005566, over 16418.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09272, pruned_loss=0.01437, audio_tagging_loss=0.009464, over 3041620.09 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:53:04,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308700 2023-11-22 18:53:04,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2057960.0, ans=0.1 2023-11-22 18:53:15,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2058026.6666666667, ans=0.0 2023-11-22 18:53:45,846 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.339e+01 8.206e+01 8.841e+01 9.625e+01 1.237e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-22 18:53:46,727 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.50 vs. limit=12.0 2023-11-22 18:54:03,362 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8150, loss[loss=0.05528, simple_loss=0.07111, pruned_loss=0.01107, audio_tagging_loss=0.008654, over 15128.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09212, pruned_loss=0.01408, audio_tagging_loss=0.009345, over 3044799.73 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:54:08,436 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308750 2023-11-22 18:54:13,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2058293.3333333333, ans=0.125 2023-11-22 18:54:18,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2058360.0, ans=0.125 2023-11-22 18:54:52,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2058493.3333333333, ans=0.125 2023-11-22 18:55:07,313 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8200, loss[loss=0.05821, simple_loss=0.07155, pruned_loss=0.01085, audio_tagging_loss=0.01159, over 15409.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.092, pruned_loss=0.01413, audio_tagging_loss=0.009262, over 3038879.44 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:55:07,327 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 18:55:12,428 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308800 2023-11-22 18:55:29,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2058693.3333333333, ans=0.125 2023-11-22 18:55:35,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2058760.0, ans=0.125 2023-11-22 18:55:40,531 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.09 vs. limit=22.5 2023-11-22 18:55:42,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.97 vs. limit=10.0 2023-11-22 18:55:48,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2058826.6666666667, ans=0.0 2023-11-22 18:55:51,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2058826.6666666667, ans=0.125 2023-11-22 18:55:55,511 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.302e+01 8.890e+01 9.670e+01 1.288e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 18:56:11,849 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8250, loss[loss=0.06871, simple_loss=0.09116, pruned_loss=0.01383, audio_tagging_loss=0.009308, over 15181.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09215, pruned_loss=0.01428, audio_tagging_loss=0.009266, over 3036939.07 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:56:17,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308850 2023-11-22 18:56:21,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.94 vs. limit=15.0 2023-11-22 18:56:21,564 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.46 vs. limit=15.0 2023-11-22 18:56:26,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2059026.6666666667, ans=0.0 2023-11-22 18:56:35,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2059026.6666666667, ans=0.09899494936611666 2023-11-22 18:56:36,173 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:56:56,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2059160.0, ans=0.125 2023-11-22 18:57:04,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2059226.6666666667, ans=0.0 2023-11-22 18:57:04,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2059226.6666666667, ans=0.1 2023-11-22 18:57:10,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2059226.6666666667, ans=0.125 2023-11-22 18:57:10,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2059226.6666666667, ans=0.125 2023-11-22 18:57:16,296 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8300, loss[loss=0.06331, simple_loss=0.0833, pruned_loss=0.01359, audio_tagging_loss=0.008069, over 14808.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09239, pruned_loss=0.01421, audio_tagging_loss=0.009258, over 3036775.82 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:57:16,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2059293.3333333333, ans=0.0 2023-11-22 18:57:21,466 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308900 2023-11-22 18:57:29,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.99 vs. limit=15.0 2023-11-22 18:57:45,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2059426.6666666667, ans=0.2 2023-11-22 18:57:47,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.23 vs. limit=10.0 2023-11-22 18:58:03,823 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.260e+01 8.357e+01 8.920e+01 9.669e+01 1.328e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 18:58:20,135 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8350, loss[loss=0.06513, simple_loss=0.08812, pruned_loss=0.01087, audio_tagging_loss=0.0102, over 15120.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09259, pruned_loss=0.01436, audio_tagging_loss=0.009279, over 3041905.53 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 18:58:25,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 308950 2023-11-22 18:58:31,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2059693.3333333333, ans=0.0 2023-11-22 18:58:32,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2059693.3333333333, ans=0.125 2023-11-22 18:58:46,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2059760.0, ans=0.0 2023-11-22 18:58:55,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2059760.0, ans=0.125 2023-11-22 18:59:09,325 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 18:59:22,777 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8400, loss[loss=0.06332, simple_loss=0.08383, pruned_loss=0.01273, audio_tagging_loss=0.008671, over 15195.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09189, pruned_loss=0.01417, audio_tagging_loss=0.009258, over 3049436.97 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 18:59:26,413 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.55 vs. limit=15.0 2023-11-22 18:59:29,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309000 2023-11-22 18:59:33,522 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.74 vs. limit=22.5 2023-11-22 18:59:39,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2060026.6666666667, ans=0.125 2023-11-22 18:59:53,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2060093.3333333333, ans=0.0 2023-11-22 18:59:57,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2060093.3333333333, ans=0.0 2023-11-22 19:00:10,223 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.438e+01 8.273e+01 8.789e+01 9.898e+01 1.441e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-22 19:00:22,481 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.72 vs. limit=10.0 2023-11-22 19:00:25,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2060226.6666666667, ans=0.125 2023-11-22 19:00:28,043 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8450, loss[loss=0.06191, simple_loss=0.08657, pruned_loss=0.01334, audio_tagging_loss=0.005283, over 16195.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09205, pruned_loss=0.01417, audio_tagging_loss=0.009221, over 3056392.64 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:00:33,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309050 2023-11-22 19:00:45,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.13 vs. limit=6.0 2023-11-22 19:00:47,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2060360.0, ans=0.125 2023-11-22 19:01:10,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2060493.3333333333, ans=0.2 2023-11-22 19:01:11,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2060493.3333333333, ans=0.0 2023-11-22 19:01:17,301 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.33 vs. limit=15.0 2023-11-22 19:01:19,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2060560.0, ans=0.0 2023-11-22 19:01:31,974 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8500, loss[loss=0.07503, simple_loss=0.08306, pruned_loss=0.02177, audio_tagging_loss=0.01174, over 14457.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09303, pruned_loss=0.0146, audio_tagging_loss=0.009227, over 3049305.08 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:01:34,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2060626.6666666667, ans=0.04949747468305833 2023-11-22 19:01:36,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309100 2023-11-22 19:01:39,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2060626.6666666667, ans=0.125 2023-11-22 19:01:42,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2060626.6666666667, ans=0.0 2023-11-22 19:01:49,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2060693.3333333333, ans=0.125 2023-11-22 19:02:19,382 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.798e+01 8.300e+01 8.864e+01 9.381e+01 1.141e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-22 19:02:24,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2060893.3333333333, ans=0.1 2023-11-22 19:02:35,143 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8550, loss[loss=0.06127, simple_loss=0.08284, pruned_loss=0.01358, audio_tagging_loss=0.006274, over 15038.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09376, pruned_loss=0.0148, audio_tagging_loss=0.009205, over 3051226.00 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:02:35,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2060960.0, ans=0.125 2023-11-22 19:02:38,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten.whitening_limit, batch_count=2060960.0, ans=22.5 2023-11-22 19:02:41,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309150 2023-11-22 19:03:12,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2061093.3333333333, ans=0.1 2023-11-22 19:03:18,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2061160.0, ans=0.0 2023-11-22 19:03:19,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2061160.0, ans=0.125 2023-11-22 19:03:20,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2061160.0, ans=0.0 2023-11-22 19:03:31,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2061226.6666666667, ans=0.2 2023-11-22 19:03:40,644 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8600, loss[loss=0.0964, simple_loss=0.1292, pruned_loss=0.02042, audio_tagging_loss=0.01138, over 16051.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09372, pruned_loss=0.01472, audio_tagging_loss=0.009239, over 3044543.03 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:03:45,533 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309200 2023-11-22 19:03:54,873 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.78 vs. limit=22.5 2023-11-22 19:04:01,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2061360.0, ans=0.125 2023-11-22 19:04:27,939 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.278e+01 9.087e+01 9.783e+01 1.157e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-22 19:04:43,833 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8650, loss[loss=0.04867, simple_loss=0.0555, pruned_loss=0.007453, audio_tagging_loss=0.01347, over 15562.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.0932, pruned_loss=0.01453, audio_tagging_loss=0.009412, over 3045688.44 frames. ], batch size: 62, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:04:49,424 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309250 2023-11-22 19:05:16,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2061760.0, ans=0.1 2023-11-22 19:05:19,557 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.03 vs. limit=22.5 2023-11-22 19:05:44,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2061893.3333333333, ans=0.125 2023-11-22 19:05:46,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2061960.0, ans=0.125 2023-11-22 19:05:47,814 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8700, loss[loss=0.04936, simple_loss=0.06522, pruned_loss=0.005326, audio_tagging_loss=0.01142, over 15254.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09414, pruned_loss=0.01479, audio_tagging_loss=0.009414, over 3047740.12 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:05:53,017 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309300 2023-11-22 19:06:36,750 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.774e+01 8.375e+01 8.886e+01 9.746e+01 1.312e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 19:06:40,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2062226.6666666667, ans=0.125 2023-11-22 19:06:51,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2062293.3333333333, ans=0.0 2023-11-22 19:06:52,430 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8750, loss[loss=0.08278, simple_loss=0.1089, pruned_loss=0.01878, audio_tagging_loss=0.009562, over 15206.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.094, pruned_loss=0.01469, audio_tagging_loss=0.009473, over 3048663.13 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:06:57,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309350 2023-11-22 19:06:58,914 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:07:33,014 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:07:41,200 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.67 vs. limit=15.0 2023-11-22 19:07:55,117 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8800, loss[loss=0.06511, simple_loss=0.08566, pruned_loss=0.009539, audio_tagging_loss=0.01275, over 16275.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09426, pruned_loss=0.01479, audio_tagging_loss=0.009601, over 3054828.48 frames. ], batch size: 65, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:08:00,169 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309400 2023-11-22 19:08:10,046 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.02 vs. limit=22.5 2023-11-22 19:08:20,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2062760.0, ans=0.125 2023-11-22 19:08:27,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.84 vs. limit=12.0 2023-11-22 19:08:36,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2062826.6666666667, ans=10.0 2023-11-22 19:08:43,646 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.328e+01 8.323e+01 8.895e+01 9.421e+01 1.402e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-22 19:08:44,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0 2023-11-22 19:08:58,982 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8850, loss[loss=0.06822, simple_loss=0.08547, pruned_loss=0.01572, audio_tagging_loss=0.009764, over 15225.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09349, pruned_loss=0.01469, audio_tagging_loss=0.009646, over 3042198.72 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:09:03,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309450 2023-11-22 19:09:05,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2062960.0, ans=0.0 2023-11-22 19:09:06,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2062960.0, ans=0.125 2023-11-22 19:09:10,421 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:09:11,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.24 vs. limit=22.5 2023-11-22 19:09:22,451 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.47 vs. limit=15.0 2023-11-22 19:09:49,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2063226.6666666667, ans=0.1 2023-11-22 19:09:51,046 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:10:02,979 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8900, loss[loss=0.06789, simple_loss=0.09385, pruned_loss=0.0118, audio_tagging_loss=0.009163, over 15259.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09372, pruned_loss=0.01479, audio_tagging_loss=0.009556, over 3042352.46 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:10:07,926 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309500 2023-11-22 19:10:10,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2063293.3333333333, ans=0.125 2023-11-22 19:10:15,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2063360.0, ans=0.0 2023-11-22 19:10:21,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2063360.0, ans=0.0 2023-11-22 19:10:22,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2063360.0, ans=0.0 2023-11-22 19:10:26,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=15.0 2023-11-22 19:10:29,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.35 vs. limit=15.0 2023-11-22 19:10:48,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2063493.3333333333, ans=0.125 2023-11-22 19:10:50,963 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.152e+01 8.729e+01 9.572e+01 1.153e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-22 19:10:58,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2063560.0, ans=0.125 2023-11-22 19:11:05,625 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 8950, loss[loss=0.06462, simple_loss=0.0902, pruned_loss=0.009763, audio_tagging_loss=0.009755, over 15370.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09369, pruned_loss=0.0147, audio_tagging_loss=0.009347, over 3039560.58 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:11:10,651 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309550 2023-11-22 19:11:17,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2063693.3333333333, ans=0.125 2023-11-22 19:11:39,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2063760.0, ans=0.125 2023-11-22 19:11:41,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2063760.0, ans=0.0 2023-11-22 19:11:58,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2063893.3333333333, ans=0.05 2023-11-22 19:12:07,968 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9000, loss[loss=0.07894, simple_loss=0.09849, pruned_loss=0.02082, audio_tagging_loss=0.008881, over 15635.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09345, pruned_loss=0.01476, audio_tagging_loss=0.00929, over 3044534.29 frames. ], batch size: 60, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:12:07,969 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 19:12:45,250 INFO [train_asr.py:1253] (3/4) Epoch 26, validation: loss=0.0595, simple_loss=0.05137, pruned_loss=0.00505, audio_tagging_loss=0.02877, over 4681554.00 frames. 2023-11-22 19:12:45,251 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 19:12:50,188 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309600 2023-11-22 19:12:50,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2063960.0, ans=0.0 2023-11-22 19:12:53,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2063960.0, ans=0.1 2023-11-22 19:12:55,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2063960.0, ans=0.125 2023-11-22 19:12:58,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064026.6666666667, ans=0.1 2023-11-22 19:12:59,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064026.6666666667, ans=0.1 2023-11-22 19:13:04,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2064026.6666666667, ans=0.125 2023-11-22 19:13:32,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2064160.0, ans=0.0 2023-11-22 19:13:34,773 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.292e+01 8.198e+01 8.954e+01 9.788e+01 1.376e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-22 19:13:48,344 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9050, loss[loss=0.06592, simple_loss=0.09426, pruned_loss=0.01195, audio_tagging_loss=0.006833, over 15581.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09378, pruned_loss=0.01471, audio_tagging_loss=0.009103, over 3046319.17 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:13:51,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2064293.3333333333, ans=0.0 2023-11-22 19:13:53,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309650 2023-11-22 19:14:01,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2064360.0, ans=0.0 2023-11-22 19:14:07,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2064360.0, ans=0.07 2023-11-22 19:14:09,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064360.0, ans=0.1 2023-11-22 19:14:10,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2064360.0, ans=0.2 2023-11-22 19:14:23,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2064426.6666666667, ans=0.07 2023-11-22 19:14:24,255 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:14:26,020 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.66 vs. limit=15.0 2023-11-22 19:14:29,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2064493.3333333333, ans=0.125 2023-11-22 19:14:43,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0 2023-11-22 19:14:50,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2064626.6666666667, ans=0.04949747468305833 2023-11-22 19:14:50,839 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9100, loss[loss=0.06481, simple_loss=0.07786, pruned_loss=0.01584, audio_tagging_loss=0.01004, over 14775.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09405, pruned_loss=0.0147, audio_tagging_loss=0.008958, over 3042260.68 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:14:55,771 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309700 2023-11-22 19:14:57,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2064626.6666666667, ans=0.125 2023-11-22 19:15:17,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2064760.0, ans=0.0 2023-11-22 19:15:20,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2064760.0, ans=0.0 2023-11-22 19:15:21,489 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:15:22,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064760.0, ans=0.1 2023-11-22 19:15:26,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=12.0 2023-11-22 19:15:36,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2064826.6666666667, ans=0.125 2023-11-22 19:15:39,606 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.178e+01 9.169e+01 9.914e+01 1.442e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-22 19:15:54,986 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9150, loss[loss=0.06911, simple_loss=0.09495, pruned_loss=0.01453, audio_tagging_loss=0.007101, over 15649.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09436, pruned_loss=0.01496, audio_tagging_loss=0.008961, over 3044093.42 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:15:58,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2064960.0, ans=0.0 2023-11-22 19:15:59,950 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309750 2023-11-22 19:16:05,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2064960.0, ans=0.0 2023-11-22 19:16:11,209 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:16:54,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2065226.6666666667, ans=0.07 2023-11-22 19:16:54,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2065226.6666666667, ans=0.125 2023-11-22 19:16:57,884 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9200, loss[loss=0.07893, simple_loss=0.106, pruned_loss=0.01925, audio_tagging_loss=0.006652, over 15270.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09385, pruned_loss=0.01491, audio_tagging_loss=0.008954, over 3039842.85 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0 2023-11-22 19:17:02,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309800 2023-11-22 19:17:05,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2065293.3333333333, ans=0.2 2023-11-22 19:17:20,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2065360.0, ans=0.1 2023-11-22 19:17:43,602 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.68 vs. limit=12.0 2023-11-22 19:17:44,873 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.41 vs. limit=10.0 2023-11-22 19:17:48,101 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.424e+01 8.996e+01 9.911e+01 1.165e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 19:17:51,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=15.0 2023-11-22 19:18:00,307 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9250, loss[loss=0.05172, simple_loss=0.06727, pruned_loss=0.01054, audio_tagging_loss=0.007551, over 14577.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09356, pruned_loss=0.01481, audio_tagging_loss=0.009039, over 3046661.97 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:18:01,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2065626.6666666667, ans=0.125 2023-11-22 19:18:05,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309850 2023-11-22 19:18:06,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.46 vs. limit=22.5 2023-11-22 19:18:07,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2065626.6666666667, ans=0.125 2023-11-22 19:18:09,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2065626.6666666667, ans=0.0 2023-11-22 19:18:09,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2065626.6666666667, ans=0.125 2023-11-22 19:18:23,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2065693.3333333333, ans=0.1 2023-11-22 19:18:23,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=2065693.3333333333, ans=0.1 2023-11-22 19:18:26,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2065760.0, ans=0.0 2023-11-22 19:18:26,396 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.51 vs. limit=15.0 2023-11-22 19:18:32,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=9.98 vs. limit=22.5 2023-11-22 19:18:51,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2065893.3333333333, ans=0.0 2023-11-22 19:19:04,402 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9300, loss[loss=0.08345, simple_loss=0.1064, pruned_loss=0.02121, audio_tagging_loss=0.009068, over 15020.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09277, pruned_loss=0.01454, audio_tagging_loss=0.009178, over 3051169.62 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:19:09,957 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309900 2023-11-22 19:19:37,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2066093.3333333333, ans=0.0 2023-11-22 19:19:55,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.43 vs. limit=10.0 2023-11-22 19:19:56,385 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 7.933e+01 8.549e+01 9.311e+01 1.177e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-22 19:19:59,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2066226.6666666667, ans=0.1 2023-11-22 19:19:59,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2066226.6666666667, ans=0.125 2023-11-22 19:20:09,199 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9350, loss[loss=0.09218, simple_loss=0.1224, pruned_loss=0.02002, audio_tagging_loss=0.01095, over 15687.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09216, pruned_loss=0.0143, audio_tagging_loss=0.009176, over 3049121.36 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:20:11,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2066293.3333333333, ans=0.05 2023-11-22 19:20:11,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2066293.3333333333, ans=0.0 2023-11-22 19:20:14,068 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 309950 2023-11-22 19:21:01,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2066560.0, ans=0.1 2023-11-22 19:21:04,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2066560.0, ans=0.0 2023-11-22 19:21:11,941 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9400, loss[loss=0.05416, simple_loss=0.06214, pruned_loss=0.01245, audio_tagging_loss=0.01064, over 14493.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09265, pruned_loss=0.01442, audio_tagging_loss=0.009315, over 3049378.17 frames. ], batch size: 55, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:21:16,948 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310000 2023-11-22 19:21:19,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2066626.6666666667, ans=0.125 2023-11-22 19:21:24,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2066693.3333333333, ans=0.125 2023-11-22 19:21:35,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2066693.3333333333, ans=0.125 2023-11-22 19:21:35,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2066693.3333333333, ans=0.125 2023-11-22 19:21:38,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2066760.0, ans=0.125 2023-11-22 19:21:41,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2066760.0, ans=0.0 2023-11-22 19:21:43,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2066760.0, ans=0.0 2023-11-22 19:21:46,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2066760.0, ans=0.125 2023-11-22 19:22:03,896 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.065e+01 8.287e+01 9.048e+01 9.651e+01 1.161e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-22 19:22:12,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2066893.3333333333, ans=0.0 2023-11-22 19:22:13,662 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:22:15,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2066960.0, ans=0.05 2023-11-22 19:22:16,563 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9450, loss[loss=0.06077, simple_loss=0.07847, pruned_loss=0.01115, audio_tagging_loss=0.01038, over 14839.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09349, pruned_loss=0.01459, audio_tagging_loss=0.009421, over 3043969.10 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:22:19,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2066960.0, ans=0.0 2023-11-22 19:22:22,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310050 2023-11-22 19:22:36,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2067026.6666666667, ans=0.1 2023-11-22 19:22:49,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2067093.3333333333, ans=0.125 2023-11-22 19:22:49,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2067093.3333333333, ans=0.0 2023-11-22 19:22:51,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2067093.3333333333, ans=0.0 2023-11-22 19:22:58,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2067160.0, ans=0.07 2023-11-22 19:23:18,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2067226.6666666667, ans=0.125 2023-11-22 19:23:20,771 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9500, loss[loss=0.06153, simple_loss=0.07272, pruned_loss=0.01426, audio_tagging_loss=0.01091, over 15233.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09283, pruned_loss=0.01448, audio_tagging_loss=0.009439, over 3043659.06 frames. ], batch size: 61, lr: 2.61e-03, grad_scale: 8.0 2023-11-22 19:23:26,362 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310100 2023-11-22 19:23:29,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.11 vs. limit=12.0 2023-11-22 19:23:50,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2067426.6666666667, ans=0.2 2023-11-22 19:24:12,954 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.401e+01 8.999e+01 9.659e+01 1.129e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-22 19:24:20,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.68 vs. limit=22.5 2023-11-22 19:24:24,405 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9550, loss[loss=0.07017, simple_loss=0.08975, pruned_loss=0.01481, audio_tagging_loss=0.01049, over 15268.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09338, pruned_loss=0.01466, audio_tagging_loss=0.009488, over 3031197.96 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 8.0 2023-11-22 19:24:29,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310150 2023-11-22 19:24:38,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2067693.3333333333, ans=0.125 2023-11-22 19:24:39,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2067693.3333333333, ans=0.0 2023-11-22 19:25:06,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2067826.6666666667, ans=10.0 2023-11-22 19:25:09,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2067826.6666666667, ans=0.125 2023-11-22 19:25:27,156 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9600, loss[loss=0.06686, simple_loss=0.08646, pruned_loss=0.01465, audio_tagging_loss=0.008988, over 15414.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.0937, pruned_loss=0.01474, audio_tagging_loss=0.009545, over 3036651.37 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0 2023-11-22 19:25:32,803 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310200 2023-11-22 19:25:41,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2068026.6666666667, ans=0.1 2023-11-22 19:25:49,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2068026.6666666667, ans=0.125 2023-11-22 19:25:55,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2068093.3333333333, ans=0.035 2023-11-22 19:26:11,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2068160.0, ans=0.125 2023-11-22 19:26:13,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.38 vs. limit=22.5 2023-11-22 19:26:20,650 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.239e+01 8.821e+01 9.524e+01 1.134e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 19:26:28,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2068226.6666666667, ans=0.125 2023-11-22 19:26:32,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2068293.3333333333, ans=0.125 2023-11-22 19:26:32,856 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9650, loss[loss=0.077, simple_loss=0.1006, pruned_loss=0.01842, audio_tagging_loss=0.008295, over 13984.00 frames. ], tot_loss[loss=0.07164, simple_loss=0.09431, pruned_loss=0.01501, audio_tagging_loss=0.009481, over 3035796.48 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:26:37,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310250 2023-11-22 19:26:46,313 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:26:56,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2068426.6666666667, ans=0.125 2023-11-22 19:27:00,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2068426.6666666667, ans=0.0 2023-11-22 19:27:23,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2068560.0, ans=0.125 2023-11-22 19:27:28,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.17 vs. limit=22.5 2023-11-22 19:27:36,324 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9700, loss[loss=0.07335, simple_loss=0.1044, pruned_loss=0.01483, audio_tagging_loss=0.006312, over 14452.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09423, pruned_loss=0.01497, audio_tagging_loss=0.009253, over 3036567.65 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:27:41,825 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310300 2023-11-22 19:28:04,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2068760.0, ans=0.0 2023-11-22 19:28:21,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2068826.6666666667, ans=0.125 2023-11-22 19:28:27,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2068893.3333333333, ans=0.125 2023-11-22 19:28:27,894 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.10 vs. limit=22.5 2023-11-22 19:28:29,030 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.323e+01 8.859e+01 9.612e+01 1.303e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-22 19:28:39,876 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9750, loss[loss=0.06463, simple_loss=0.08774, pruned_loss=0.01061, audio_tagging_loss=0.01015, over 14233.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09305, pruned_loss=0.01464, audio_tagging_loss=0.009286, over 3037799.56 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:28:43,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2068960.0, ans=0.125 2023-11-22 19:28:45,391 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310350 2023-11-22 19:29:02,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0 2023-11-22 19:29:03,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2069026.6666666667, ans=0.125 2023-11-22 19:29:05,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2069093.3333333333, ans=0.2 2023-11-22 19:29:44,817 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9800, loss[loss=0.07561, simple_loss=0.1046, pruned_loss=0.01606, audio_tagging_loss=0.00723, over 15773.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09324, pruned_loss=0.01467, audio_tagging_loss=0.009328, over 3038727.13 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:29:45,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2069293.3333333333, ans=0.125 2023-11-22 19:29:45,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2069293.3333333333, ans=0.125 2023-11-22 19:29:45,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2069293.3333333333, ans=0.125 2023-11-22 19:29:49,779 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310400 2023-11-22 19:29:52,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2069293.3333333333, ans=0.0 2023-11-22 19:30:00,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2069360.0, ans=0.125 2023-11-22 19:30:07,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2069360.0, ans=0.125 2023-11-22 19:30:37,549 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.460e+01 9.204e+01 9.900e+01 1.495e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-22 19:30:41,219 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:30:45,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2069560.0, ans=0.0 2023-11-22 19:30:48,378 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9850, loss[loss=0.07966, simple_loss=0.1012, pruned_loss=0.01972, audio_tagging_loss=0.009337, over 15362.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09373, pruned_loss=0.01498, audio_tagging_loss=0.009286, over 3045356.62 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 8.0 2023-11-22 19:30:53,393 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310450 2023-11-22 19:31:05,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2069693.3333333333, ans=0.1 2023-11-22 19:31:12,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2069760.0, ans=0.0 2023-11-22 19:31:28,009 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.85 vs. limit=12.0 2023-11-22 19:31:51,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=12.0 2023-11-22 19:31:51,987 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9900, loss[loss=0.05528, simple_loss=0.07618, pruned_loss=0.007402, audio_tagging_loss=0.009788, over 15663.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09301, pruned_loss=0.0147, audio_tagging_loss=0.009202, over 3047781.21 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 8.0 2023-11-22 19:31:56,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310500 2023-11-22 19:31:57,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2069960.0, ans=0.05 2023-11-22 19:32:11,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2070026.6666666667, ans=0.04949747468305833 2023-11-22 19:32:11,215 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:32:13,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2070026.6666666667, ans=0.125 2023-11-22 19:32:25,599 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:32:39,694 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.65 vs. limit=15.0 2023-11-22 19:32:45,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.326e+01 8.797e+01 9.739e+01 1.395e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 19:32:48,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.34 vs. limit=6.0 2023-11-22 19:32:54,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.85 vs. limit=15.0 2023-11-22 19:32:55,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2070293.3333333333, ans=0.0 2023-11-22 19:32:56,581 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 9950, loss[loss=0.06413, simple_loss=0.0757, pruned_loss=0.01549, audio_tagging_loss=0.01079, over 15222.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09218, pruned_loss=0.01438, audio_tagging_loss=0.009251, over 3050693.74 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 8.0 2023-11-22 19:33:01,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310550 2023-11-22 19:33:15,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.23 vs. limit=15.0 2023-11-22 19:33:18,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.40 vs. limit=22.5 2023-11-22 19:33:27,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2070426.6666666667, ans=0.0 2023-11-22 19:33:32,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=11.06 vs. limit=12.0 2023-11-22 19:33:32,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2070493.3333333333, ans=0.0 2023-11-22 19:33:59,661 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10000, loss[loss=0.05589, simple_loss=0.06524, pruned_loss=0.01141, audio_tagging_loss=0.01187, over 14512.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09213, pruned_loss=0.01431, audio_tagging_loss=0.009156, over 3047543.02 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:34:02,879 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.60 vs. limit=15.0 2023-11-22 19:34:04,700 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310600 2023-11-22 19:34:43,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2070826.6666666667, ans=0.05 2023-11-22 19:34:52,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2070893.3333333333, ans=0.1 2023-11-22 19:34:53,495 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.266e+01 8.772e+01 9.539e+01 1.390e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 19:35:03,295 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10050, loss[loss=0.08226, simple_loss=0.1139, pruned_loss=0.0183, audio_tagging_loss=0.007005, over 15647.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09277, pruned_loss=0.01456, audio_tagging_loss=0.009119, over 3047338.20 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:35:08,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310650 2023-11-22 19:35:14,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2070960.0, ans=0.0 2023-11-22 19:35:15,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.00 vs. limit=15.0 2023-11-22 19:35:24,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2071026.6666666667, ans=0.2 2023-11-22 19:35:30,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=15.0 2023-11-22 19:35:35,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2071093.3333333333, ans=0.125 2023-11-22 19:35:46,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2071160.0, ans=0.1 2023-11-22 19:35:57,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2071226.6666666667, ans=0.125 2023-11-22 19:36:08,959 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10100, loss[loss=0.1002, simple_loss=0.1339, pruned_loss=0.02353, audio_tagging_loss=0.009698, over 15626.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09315, pruned_loss=0.01457, audio_tagging_loss=0.00917, over 3055135.87 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:36:14,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310700 2023-11-22 19:36:31,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2071360.0, ans=0.09899494936611666 2023-11-22 19:36:40,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.92 vs. limit=15.0 2023-11-22 19:36:47,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2071493.3333333333, ans=0.125 2023-11-22 19:36:52,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.74 vs. limit=12.0 2023-11-22 19:36:56,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2071493.3333333333, ans=0.125 2023-11-22 19:36:58,550 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:37:02,765 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.311e+01 8.891e+01 9.853e+01 2.191e+02, threshold=1.778e+02, percent-clipped=1.0 2023-11-22 19:37:12,763 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10150, loss[loss=0.06676, simple_loss=0.08567, pruned_loss=0.01349, audio_tagging_loss=0.01044, over 15530.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09449, pruned_loss=0.01482, audio_tagging_loss=0.009233, over 3053158.62 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:37:17,772 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310750 2023-11-22 19:37:41,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2071760.0, ans=0.0 2023-11-22 19:37:41,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2071760.0, ans=0.125 2023-11-22 19:37:41,918 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:37:48,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2071760.0, ans=0.1 2023-11-22 19:38:08,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2071893.3333333333, ans=0.125 2023-11-22 19:38:15,733 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10200, loss[loss=0.05836, simple_loss=0.07367, pruned_loss=0.01072, audio_tagging_loss=0.01081, over 15466.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09374, pruned_loss=0.01477, audio_tagging_loss=0.009285, over 3061476.98 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:38:20,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310800 2023-11-22 19:38:39,611 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:38:41,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2072093.3333333333, ans=0.125 2023-11-22 19:38:53,754 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.25 vs. limit=15.0 2023-11-22 19:38:56,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2072160.0, ans=0.125 2023-11-22 19:39:08,472 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.131e+01 8.876e+01 9.472e+01 1.289e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-22 19:39:19,373 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10250, loss[loss=0.07197, simple_loss=0.08596, pruned_loss=0.01779, audio_tagging_loss=0.01121, over 15001.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09345, pruned_loss=0.01483, audio_tagging_loss=0.009402, over 3054275.58 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:39:24,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2072293.3333333333, ans=0.5 2023-11-22 19:39:25,540 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310850 2023-11-22 19:39:45,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2072426.6666666667, ans=0.2 2023-11-22 19:40:08,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2072493.3333333333, ans=0.125 2023-11-22 19:40:20,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2072560.0, ans=0.1 2023-11-22 19:40:23,794 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10300, loss[loss=0.04961, simple_loss=0.05631, pruned_loss=0.01111, audio_tagging_loss=0.01034, over 14501.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09394, pruned_loss=0.01495, audio_tagging_loss=0.009357, over 3061409.08 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:40:28,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310900 2023-11-22 19:40:33,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.09 vs. limit=8.0 2023-11-22 19:40:34,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2072693.3333333333, ans=0.125 2023-11-22 19:40:39,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2072693.3333333333, ans=0.125 2023-11-22 19:40:39,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2072693.3333333333, ans=0.1 2023-11-22 19:40:49,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2072760.0, ans=0.05 2023-11-22 19:41:02,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2072826.6666666667, ans=0.125 2023-11-22 19:41:06,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2072826.6666666667, ans=0.07 2023-11-22 19:41:16,986 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.279e+01 8.744e+01 9.443e+01 1.295e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 19:41:17,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2072893.3333333333, ans=0.2 2023-11-22 19:41:26,741 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10350, loss[loss=0.07978, simple_loss=0.1072, pruned_loss=0.01683, audio_tagging_loss=0.009334, over 16034.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.0937, pruned_loss=0.01506, audio_tagging_loss=0.009479, over 3057937.73 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:41:31,724 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 310950 2023-11-22 19:41:53,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2073093.3333333333, ans=0.05 2023-11-22 19:42:03,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2073093.3333333333, ans=0.0 2023-11-22 19:42:30,610 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10400, loss[loss=0.09328, simple_loss=0.1297, pruned_loss=0.02092, audio_tagging_loss=0.007515, over 15856.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09313, pruned_loss=0.01487, audio_tagging_loss=0.00965, over 3055858.75 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 19:42:36,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311000 2023-11-22 19:42:51,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2073360.0, ans=0.1 2023-11-22 19:42:55,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=2073360.0, ans=0.1 2023-11-22 19:43:08,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2073493.3333333333, ans=0.125 2023-11-22 19:43:17,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2073493.3333333333, ans=0.125 2023-11-22 19:43:19,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2073493.3333333333, ans=0.125 2023-11-22 19:43:25,682 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.233e+01 8.739e+01 9.492e+01 1.200e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-22 19:43:36,098 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10450, loss[loss=0.068, simple_loss=0.07931, pruned_loss=0.01639, audio_tagging_loss=0.01196, over 15011.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09244, pruned_loss=0.01478, audio_tagging_loss=0.009568, over 3046550.31 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 19:43:36,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2073626.6666666667, ans=0.1 2023-11-22 19:43:41,070 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311050 2023-11-22 19:43:58,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2073693.3333333333, ans=0.125 2023-11-22 19:44:19,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2073826.6666666667, ans=0.2 2023-11-22 19:44:34,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2073893.3333333333, ans=0.125 2023-11-22 19:44:39,311 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10500, loss[loss=0.0949, simple_loss=0.1308, pruned_loss=0.02049, audio_tagging_loss=0.009004, over 14739.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09251, pruned_loss=0.01475, audio_tagging_loss=0.009435, over 3048400.38 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:44:39,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2073960.0, ans=0.1 2023-11-22 19:44:44,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311100 2023-11-22 19:44:51,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2074026.6666666667, ans=0.125 2023-11-22 19:44:51,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2074026.6666666667, ans=0.1 2023-11-22 19:45:09,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2074093.3333333333, ans=0.5 2023-11-22 19:45:20,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2074160.0, ans=0.0 2023-11-22 19:45:26,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2074160.0, ans=0.1 2023-11-22 19:45:33,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.986e+01 7.970e+01 8.549e+01 9.373e+01 1.177e+02, threshold=1.710e+02, percent-clipped=0.0 2023-11-22 19:45:42,245 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10550, loss[loss=0.05378, simple_loss=0.07282, pruned_loss=0.008739, audio_tagging_loss=0.008626, over 16094.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09291, pruned_loss=0.0147, audio_tagging_loss=0.009293, over 3049012.42 frames. ], batch size: 62, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:45:47,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311150 2023-11-22 19:45:52,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2074293.3333333333, ans=0.0 2023-11-22 19:45:59,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2074360.0, ans=0.125 2023-11-22 19:46:01,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2074360.0, ans=0.0 2023-11-22 19:46:10,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2074426.6666666667, ans=0.125 2023-11-22 19:46:14,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2074426.6666666667, ans=0.0 2023-11-22 19:46:20,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2074493.3333333333, ans=0.125 2023-11-22 19:46:47,681 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10600, loss[loss=0.07549, simple_loss=0.1082, pruned_loss=0.01298, audio_tagging_loss=0.008424, over 15525.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09366, pruned_loss=0.01486, audio_tagging_loss=0.0092, over 3050331.92 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:46:53,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311200 2023-11-22 19:46:54,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2074626.6666666667, ans=0.09899494936611666 2023-11-22 19:47:26,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2074826.6666666667, ans=0.125 2023-11-22 19:47:30,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2074826.6666666667, ans=0.125 2023-11-22 19:47:40,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2074893.3333333333, ans=0.125 2023-11-22 19:47:43,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.586e+01 8.272e+01 8.914e+01 9.712e+01 1.198e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-22 19:47:51,583 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10650, loss[loss=0.08543, simple_loss=0.1148, pruned_loss=0.02094, audio_tagging_loss=0.007082, over 14219.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09413, pruned_loss=0.01487, audio_tagging_loss=0.009194, over 3054472.42 frames. ], batch size: 53, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:47:56,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311250 2023-11-22 19:48:13,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2075026.6666666667, ans=0.0 2023-11-22 19:48:35,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2075160.0, ans=0.09899494936611666 2023-11-22 19:48:37,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.85 vs. limit=22.5 2023-11-22 19:48:54,722 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10700, loss[loss=0.06893, simple_loss=0.09666, pruned_loss=0.01432, audio_tagging_loss=0.006271, over 16264.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09356, pruned_loss=0.01478, audio_tagging_loss=0.009111, over 3049399.03 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:49:00,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311300 2023-11-22 19:49:10,589 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.89 vs. limit=15.0 2023-11-22 19:49:15,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2075360.0, ans=0.125 2023-11-22 19:49:15,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.09 vs. limit=6.0 2023-11-22 19:49:31,648 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 19:49:35,963 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.88 vs. limit=15.0 2023-11-22 19:49:36,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2075493.3333333333, ans=0.1 2023-11-22 19:49:45,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2075560.0, ans=0.0 2023-11-22 19:49:48,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2075560.0, ans=0.0 2023-11-22 19:49:50,182 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.233e+01 8.994e+01 9.619e+01 1.201e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 19:50:00,081 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10750, loss[loss=0.0677, simple_loss=0.08542, pruned_loss=0.01436, audio_tagging_loss=0.01062, over 14767.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09328, pruned_loss=0.01477, audio_tagging_loss=0.009119, over 3049605.11 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:50:01,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2075626.6666666667, ans=0.125 2023-11-22 19:50:05,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311350 2023-11-22 19:50:07,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2075626.6666666667, ans=0.1 2023-11-22 19:50:17,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.24 vs. limit=6.0 2023-11-22 19:50:17,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2075693.3333333333, ans=0.0 2023-11-22 19:50:22,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2075693.3333333333, ans=0.125 2023-11-22 19:50:29,408 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.72 vs. limit=15.0 2023-11-22 19:50:57,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2075893.3333333333, ans=0.125 2023-11-22 19:50:58,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2075893.3333333333, ans=0.0 2023-11-22 19:51:04,219 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10800, loss[loss=0.06031, simple_loss=0.06498, pruned_loss=0.01556, audio_tagging_loss=0.01226, over 14539.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09293, pruned_loss=0.01474, audio_tagging_loss=0.009083, over 3049876.74 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 19:51:09,197 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311400 2023-11-22 19:51:23,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2076026.6666666667, ans=0.0 2023-11-22 19:51:42,442 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0 2023-11-22 19:51:59,225 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.263e+01 8.884e+01 9.522e+01 1.111e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 19:52:07,954 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10850, loss[loss=0.06667, simple_loss=0.08718, pruned_loss=0.01319, audio_tagging_loss=0.009894, over 14822.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09213, pruned_loss=0.01469, audio_tagging_loss=0.009227, over 3044720.05 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 19:52:13,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311450 2023-11-22 19:52:52,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2076493.3333333333, ans=0.1 2023-11-22 19:53:07,035 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:53:13,058 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10900, loss[loss=0.09641, simple_loss=0.1188, pruned_loss=0.02757, audio_tagging_loss=0.009417, over 14416.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09231, pruned_loss=0.01465, audio_tagging_loss=0.009211, over 3041335.76 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:53:17,898 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311500 2023-11-22 19:53:38,663 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.21 vs. limit=15.0 2023-11-22 19:53:46,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2076760.0, ans=0.0 2023-11-22 19:54:08,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2076893.3333333333, ans=0.125 2023-11-22 19:54:09,627 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.142e+01 8.836e+01 9.377e+01 1.085e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 19:54:16,919 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 10950, loss[loss=0.09378, simple_loss=0.1288, pruned_loss=0.02119, audio_tagging_loss=0.008183, over 15484.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09236, pruned_loss=0.01461, audio_tagging_loss=0.009236, over 3042935.95 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:54:21,918 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311550 2023-11-22 19:54:31,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2077026.6666666667, ans=0.0 2023-11-22 19:55:02,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2077160.0, ans=0.0 2023-11-22 19:55:04,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.17 vs. limit=10.0 2023-11-22 19:55:21,193 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11000, loss[loss=0.06828, simple_loss=0.09243, pruned_loss=0.01394, audio_tagging_loss=0.008126, over 15945.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09242, pruned_loss=0.01453, audio_tagging_loss=0.009337, over 3041588.59 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:55:26,154 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311600 2023-11-22 19:55:26,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2077293.3333333333, ans=0.1 2023-11-22 19:55:30,054 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 19:55:37,128 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.61 vs. limit=15.0 2023-11-22 19:55:43,855 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0 2023-11-22 19:55:54,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2077426.6666666667, ans=0.0 2023-11-22 19:56:04,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2077493.3333333333, ans=0.125 2023-11-22 19:56:05,481 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0 2023-11-22 19:56:06,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.31 vs. limit=22.5 2023-11-22 19:56:17,629 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.294e+01 8.863e+01 9.568e+01 1.311e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-22 19:56:17,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2077560.0, ans=0.1 2023-11-22 19:56:26,014 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11050, loss[loss=0.0795, simple_loss=0.1146, pruned_loss=0.01529, audio_tagging_loss=0.006921, over 15177.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09318, pruned_loss=0.01459, audio_tagging_loss=0.00928, over 3041116.72 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:56:31,582 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311650 2023-11-22 19:56:40,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2077693.3333333333, ans=0.0 2023-11-22 19:56:41,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.48 vs. limit=15.0 2023-11-22 19:57:18,354 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.35 vs. limit=10.0 2023-11-22 19:57:29,637 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11100, loss[loss=0.1007, simple_loss=0.1363, pruned_loss=0.0263, audio_tagging_loss=0.006302, over 15938.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09339, pruned_loss=0.01474, audio_tagging_loss=0.009375, over 3038044.70 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:57:33,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2077960.0, ans=0.0 2023-11-22 19:57:33,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.16 vs. limit=22.5 2023-11-22 19:57:34,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311700 2023-11-22 19:57:39,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2077960.0, ans=0.0 2023-11-22 19:57:45,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2078026.6666666667, ans=0.09899494936611666 2023-11-22 19:57:54,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.63 vs. limit=15.0 2023-11-22 19:57:59,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.83 vs. limit=15.0 2023-11-22 19:58:06,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2078093.3333333333, ans=0.125 2023-11-22 19:58:25,321 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.159e+01 8.862e+01 9.664e+01 1.542e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-22 19:58:32,635 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11150, loss[loss=0.06767, simple_loss=0.094, pruned_loss=0.0114, audio_tagging_loss=0.009272, over 15546.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09286, pruned_loss=0.01476, audio_tagging_loss=0.009524, over 3048330.47 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 19:58:37,514 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311750 2023-11-22 19:58:38,311 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.02 vs. limit=15.0 2023-11-22 19:58:43,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2078293.3333333333, ans=0.1 2023-11-22 19:58:49,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2078360.0, ans=0.125 2023-11-22 19:59:10,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2078493.3333333333, ans=0.125 2023-11-22 19:59:11,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2078493.3333333333, ans=0.125 2023-11-22 19:59:14,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2078493.3333333333, ans=0.125 2023-11-22 19:59:20,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.85 vs. limit=10.0 2023-11-22 19:59:29,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2078560.0, ans=0.2 2023-11-22 19:59:37,910 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11200, loss[loss=0.07649, simple_loss=0.0966, pruned_loss=0.01556, audio_tagging_loss=0.01263, over 16172.00 frames. ], tot_loss[loss=0.071, simple_loss=0.0932, pruned_loss=0.01482, audio_tagging_loss=0.009576, over 3051837.49 frames. ], batch size: 63, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 19:59:43,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311800 2023-11-22 20:00:06,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2078760.0, ans=0.2 2023-11-22 20:00:09,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2078760.0, ans=0.0 2023-11-22 20:00:18,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.64 vs. limit=12.0 2023-11-22 20:00:21,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2078826.6666666667, ans=0.09899494936611666 2023-11-22 20:00:34,756 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 7.967e+01 8.796e+01 9.529e+01 1.148e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 20:00:36,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2078893.3333333333, ans=0.0 2023-11-22 20:00:42,243 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11250, loss[loss=0.06667, simple_loss=0.07972, pruned_loss=0.01556, audio_tagging_loss=0.01125, over 14685.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09236, pruned_loss=0.01476, audio_tagging_loss=0.009529, over 3049111.77 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 20:00:47,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311850 2023-11-22 20:00:51,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2078960.0, ans=0.1 2023-11-22 20:01:09,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2079093.3333333333, ans=0.0 2023-11-22 20:01:22,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2079160.0, ans=0.1 2023-11-22 20:01:25,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.80 vs. limit=15.0 2023-11-22 20:01:27,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2079160.0, ans=0.125 2023-11-22 20:01:45,812 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11300, loss[loss=0.03384, simple_loss=0.04063, pruned_loss=0.006187, audio_tagging_loss=0.007333, over 15748.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09214, pruned_loss=0.01476, audio_tagging_loss=0.009416, over 3048350.79 frames. ], batch size: 65, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 20:01:50,719 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311900 2023-11-22 20:02:01,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2079360.0, ans=0.125 2023-11-22 20:02:06,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.94 vs. limit=15.0 2023-11-22 20:02:22,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2079493.3333333333, ans=0.2 2023-11-22 20:02:25,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=22.5 2023-11-22 20:02:26,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2079493.3333333333, ans=0.0 2023-11-22 20:02:41,667 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.136e+01 8.665e+01 9.544e+01 1.497e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-22 20:02:47,793 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11350, loss[loss=0.05993, simple_loss=0.0831, pruned_loss=0.008097, audio_tagging_loss=0.01028, over 16006.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09273, pruned_loss=0.01474, audio_tagging_loss=0.009225, over 3049997.50 frames. ], batch size: 62, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:02:53,873 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 311950 2023-11-22 20:03:20,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2079760.0, ans=0.1 2023-11-22 20:03:21,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2079760.0, ans=0.0 2023-11-22 20:03:28,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2079826.6666666667, ans=0.125 2023-11-22 20:03:28,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2079826.6666666667, ans=0.0 2023-11-22 20:03:35,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2079826.6666666667, ans=0.1 2023-11-22 20:03:48,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2079893.3333333333, ans=0.1 2023-11-22 20:03:51,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2079960.0, ans=0.09899494936611666 2023-11-22 20:03:52,226 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11400, loss[loss=0.0485, simple_loss=0.05737, pruned_loss=0.008591, audio_tagging_loss=0.01122, over 15764.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.0933, pruned_loss=0.01487, audio_tagging_loss=0.009064, over 3050952.26 frames. ], batch size: 62, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:03:57,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312000 2023-11-22 20:04:07,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0 2023-11-22 20:04:52,985 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.235e+01 8.932e+01 9.432e+01 1.169e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 20:04:59,057 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11450, loss[loss=0.06939, simple_loss=0.1012, pruned_loss=0.01393, audio_tagging_loss=0.00485, over 15413.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09312, pruned_loss=0.01477, audio_tagging_loss=0.009129, over 3046971.95 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:04:59,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2080293.3333333333, ans=0.125 2023-11-22 20:05:04,005 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312050 2023-11-22 20:05:11,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2080360.0, ans=0.0 2023-11-22 20:05:12,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2080360.0, ans=0.0 2023-11-22 20:05:15,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2080360.0, ans=0.1 2023-11-22 20:05:41,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2080493.3333333333, ans=0.0 2023-11-22 20:05:45,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2080493.3333333333, ans=0.125 2023-11-22 20:05:45,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2080493.3333333333, ans=0.0 2023-11-22 20:06:01,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2080626.6666666667, ans=0.0 2023-11-22 20:06:02,421 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11500, loss[loss=0.0941, simple_loss=0.1284, pruned_loss=0.02198, audio_tagging_loss=0.007943, over 16241.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09353, pruned_loss=0.01496, audio_tagging_loss=0.009159, over 3047011.72 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:06:07,734 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312100 2023-11-22 20:06:25,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.88 vs. limit=10.0 2023-11-22 20:06:46,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.69 vs. limit=8.0 2023-11-22 20:06:50,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2080826.6666666667, ans=0.125 2023-11-22 20:06:52,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2080893.3333333333, ans=0.1 2023-11-22 20:06:53,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2080893.3333333333, ans=0.0 2023-11-22 20:07:00,681 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.062e+01 8.847e+01 9.578e+01 1.218e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-22 20:07:06,748 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11550, loss[loss=0.05657, simple_loss=0.08005, pruned_loss=0.007205, audio_tagging_loss=0.009345, over 14624.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09283, pruned_loss=0.01466, audio_tagging_loss=0.00918, over 3049224.57 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:07:11,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312150 2023-11-22 20:07:22,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2081026.6666666667, ans=0.0 2023-11-22 20:07:43,833 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 20:08:09,797 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11600, loss[loss=0.06564, simple_loss=0.09019, pruned_loss=0.01123, audio_tagging_loss=0.009314, over 15243.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09279, pruned_loss=0.01456, audio_tagging_loss=0.009172, over 3050848.65 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 20:08:14,769 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312200 2023-11-22 20:08:27,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2081360.0, ans=0.125 2023-11-22 20:08:46,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2081426.6666666667, ans=0.125 2023-11-22 20:09:05,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2081560.0, ans=0.0 2023-11-22 20:09:08,466 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.935e+01 8.330e+01 9.074e+01 9.765e+01 1.554e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-22 20:09:13,395 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11650, loss[loss=0.07402, simple_loss=0.1022, pruned_loss=0.01582, audio_tagging_loss=0.007097, over 15351.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09364, pruned_loss=0.01477, audio_tagging_loss=0.009175, over 3050417.07 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:09:18,982 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312250 2023-11-22 20:09:19,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2081626.6666666667, ans=0.2 2023-11-22 20:09:23,627 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.94 vs. limit=15.0 2023-11-22 20:09:28,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2081693.3333333333, ans=0.125 2023-11-22 20:09:30,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2081693.3333333333, ans=0.1 2023-11-22 20:09:52,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2081826.6666666667, ans=0.125 2023-11-22 20:10:13,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2081893.3333333333, ans=0.1 2023-11-22 20:10:13,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2081893.3333333333, ans=0.0 2023-11-22 20:10:18,464 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11700, loss[loss=0.07387, simple_loss=0.09765, pruned_loss=0.01649, audio_tagging_loss=0.008557, over 15026.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09215, pruned_loss=0.01438, audio_tagging_loss=0.00927, over 3046778.13 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:10:18,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2081960.0, ans=0.125 2023-11-22 20:10:23,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2081960.0, ans=0.0 2023-11-22 20:10:23,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312300 2023-11-22 20:10:24,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2081960.0, ans=0.0 2023-11-22 20:10:31,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2082026.6666666667, ans=0.125 2023-11-22 20:10:35,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2082026.6666666667, ans=0.0 2023-11-22 20:10:36,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2082026.6666666667, ans=0.125 2023-11-22 20:10:45,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2082093.3333333333, ans=0.125 2023-11-22 20:11:02,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2082160.0, ans=0.0 2023-11-22 20:11:15,465 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0 2023-11-22 20:11:17,083 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.195e+01 8.722e+01 9.532e+01 1.280e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-22 20:11:20,072 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 20:11:22,129 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11750, loss[loss=0.05418, simple_loss=0.07897, pruned_loss=0.006947, audio_tagging_loss=0.007742, over 13028.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09236, pruned_loss=0.01449, audio_tagging_loss=0.009278, over 3043952.97 frames. ], batch size: 52, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:11:27,111 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312350 2023-11-22 20:11:28,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2082293.3333333333, ans=0.1 2023-11-22 20:11:57,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.53 vs. limit=15.0 2023-11-22 20:12:01,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2082493.3333333333, ans=0.0 2023-11-22 20:12:10,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2082493.3333333333, ans=0.025 2023-11-22 20:12:12,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2082560.0, ans=0.125 2023-11-22 20:12:25,270 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11800, loss[loss=0.08036, simple_loss=0.1071, pruned_loss=0.01715, audio_tagging_loss=0.009661, over 16532.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09413, pruned_loss=0.0149, audio_tagging_loss=0.009255, over 3044874.54 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:12:29,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2082626.6666666667, ans=0.1 2023-11-22 20:12:30,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312400 2023-11-22 20:12:30,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.54 vs. limit=15.0 2023-11-22 20:12:40,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2082693.3333333333, ans=0.125 2023-11-22 20:13:05,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2082826.6666666667, ans=0.025 2023-11-22 20:13:24,247 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.495e+01 8.351e+01 8.869e+01 9.580e+01 1.621e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-22 20:13:30,291 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11850, loss[loss=0.06056, simple_loss=0.06951, pruned_loss=0.01214, audio_tagging_loss=0.01366, over 15220.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09396, pruned_loss=0.01503, audio_tagging_loss=0.009377, over 3039229.77 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:13:35,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312450 2023-11-22 20:14:15,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2083160.0, ans=0.0 2023-11-22 20:14:24,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2083226.6666666667, ans=0.125 2023-11-22 20:14:27,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2083226.6666666667, ans=0.0 2023-11-22 20:14:34,590 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11900, loss[loss=0.07273, simple_loss=0.1023, pruned_loss=0.01239, audio_tagging_loss=0.00918, over 13885.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09366, pruned_loss=0.01482, audio_tagging_loss=0.009538, over 3038720.58 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:14:37,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.52 vs. limit=22.5 2023-11-22 20:14:39,596 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312500 2023-11-22 20:15:00,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2083426.6666666667, ans=0.125 2023-11-22 20:15:06,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2083426.6666666667, ans=0.125 2023-11-22 20:15:07,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2083426.6666666667, ans=0.125 2023-11-22 20:15:12,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2083493.3333333333, ans=0.2 2023-11-22 20:15:14,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2083493.3333333333, ans=0.0 2023-11-22 20:15:20,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2083493.3333333333, ans=0.0 2023-11-22 20:15:24,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2083560.0, ans=0.2 2023-11-22 20:15:32,658 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.017e+01 8.415e+01 9.018e+01 9.656e+01 1.551e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-22 20:15:34,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2083560.0, ans=0.125 2023-11-22 20:15:37,473 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 11950, loss[loss=0.06017, simple_loss=0.07678, pruned_loss=0.01261, audio_tagging_loss=0.009175, over 15508.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09332, pruned_loss=0.01469, audio_tagging_loss=0.009504, over 3043187.19 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 16.0 2023-11-22 20:15:42,407 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312550 2023-11-22 20:15:43,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.22 vs. limit=15.0 2023-11-22 20:15:56,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2083693.3333333333, ans=0.125 2023-11-22 20:16:02,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2083760.0, ans=0.0 2023-11-22 20:16:05,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2083760.0, ans=0.125 2023-11-22 20:16:11,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2083760.0, ans=0.125 2023-11-22 20:16:38,924 INFO [train_asr.py:1221] (3/4) Epoch 26, batch 12000, loss[loss=0.05488, simple_loss=0.0713, pruned_loss=0.009462, audio_tagging_loss=0.009764, over 16011.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09325, pruned_loss=0.01472, audio_tagging_loss=0.009461, over 3047307.74 frames. ], batch size: 63, lr: 2.60e-03, grad_scale: 32.0 2023-11-22 20:16:38,925 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 20:16:58,423 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.7688, 4.4734, 4.1459, 4.2393], device='cuda:3') 2023-11-22 20:17:17,314 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.4567, 3.8498, 4.3706, 3.3960], device='cuda:3') 2023-11-22 20:17:19,053 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.4795, 1.6836, 2.4000, 2.4070, 2.1825, 2.3630, 1.7477, 2.2886], device='cuda:3') 2023-11-22 20:17:20,954 INFO [train_asr.py:1253] (3/4) Epoch 26, validation: loss=0.05885, simple_loss=0.05139, pruned_loss=0.00512, audio_tagging_loss=0.02803, over 4681554.00 frames. 2023-11-22 20:17:20,955 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 20:17:21,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2083960.0, ans=0.0 2023-11-22 20:17:25,607 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312600 2023-11-22 20:18:24,316 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 0, loss[loss=0.07064, simple_loss=0.08682, pruned_loss=0.007885, audio_tagging_loss=0.01934, over 15300.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.08682, pruned_loss=0.007885, audio_tagging_loss=0.01934, over 15300.00 frames. ], batch size: 57, lr: 2.55e-03, grad_scale: 32.0 2023-11-22 20:18:24,316 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 20:18:43,906 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([3.5521, 3.2641, 2.1031, 3.1276], device='cuda:3') 2023-11-22 20:19:01,954 INFO [train_asr.py:1253] (3/4) Epoch 27, validation: loss=0.05818, simple_loss=0.05133, pruned_loss=0.005046, audio_tagging_loss=0.02747, over 4681554.00 frames. 2023-11-22 20:19:01,955 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 20:19:22,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2084180.0, ans=0.125 2023-11-22 20:19:24,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2084180.0, ans=0.125 2023-11-22 20:19:29,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2084246.6666666667, ans=0.1 2023-11-22 20:19:32,309 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.281e+01 9.255e+01 1.009e+02 1.305e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-22 20:19:33,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2084246.6666666667, ans=0.125 2023-11-22 20:19:43,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312650 2023-11-22 20:19:59,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2084380.0, ans=0.125 2023-11-22 20:20:01,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2084380.0, ans=0.015 2023-11-22 20:20:06,135 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 50, loss[loss=0.06547, simple_loss=0.07671, pruned_loss=0.01235, audio_tagging_loss=0.01477, over 14916.00 frames. ], tot_loss[loss=0.08058, simple_loss=0.09532, pruned_loss=0.01524, audio_tagging_loss=0.01768, over 685876.98 frames. ], batch size: 56, lr: 2.55e-03, grad_scale: 16.0 2023-11-22 20:20:12,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2084446.6666666667, ans=0.0 2023-11-22 20:20:13,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2084446.6666666667, ans=0.125 2023-11-22 20:20:33,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2084580.0, ans=0.0 2023-11-22 20:20:47,031 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312700 2023-11-22 20:21:01,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2084713.3333333333, ans=0.125 2023-11-22 20:21:12,352 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 100, loss[loss=0.06784, simple_loss=0.08284, pruned_loss=0.0125, audio_tagging_loss=0.01391, over 13971.00 frames. ], tot_loss[loss=0.0784, simple_loss=0.09331, pruned_loss=0.0148, audio_tagging_loss=0.01694, over 1208234.46 frames. ], batch size: 54, lr: 2.55e-03, grad_scale: 16.0 2023-11-22 20:21:13,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2084780.0, ans=0.125 2023-11-22 20:21:17,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2084780.0, ans=0.2 2023-11-22 20:21:19,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2084780.0, ans=0.1 2023-11-22 20:21:34,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2084846.6666666667, ans=0.1 2023-11-22 20:21:42,711 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.400e+01 8.923e+01 9.571e+01 1.006e+02 1.303e+02, threshold=1.914e+02, percent-clipped=0.0 2023-11-22 20:21:51,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312750 2023-11-22 20:21:59,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2084980.0, ans=0.125 2023-11-22 20:22:00,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2084980.0, ans=0.125 2023-11-22 20:22:05,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.24 vs. limit=15.0 2023-11-22 20:22:17,547 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 150, loss[loss=0.08864, simple_loss=0.1122, pruned_loss=0.02091, audio_tagging_loss=0.01164, over 15610.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.09413, pruned_loss=0.01491, audio_tagging_loss=0.01501, over 1609238.51 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:22:26,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.22 vs. limit=10.0 2023-11-22 20:22:38,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2085180.0, ans=0.0 2023-11-22 20:22:51,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2085246.6666666667, ans=0.125 2023-11-22 20:22:53,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2085246.6666666667, ans=0.1 2023-11-22 20:22:57,510 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312800 2023-11-22 20:23:06,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2085313.3333333333, ans=0.125 2023-11-22 20:23:16,369 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.87 vs. limit=10.0 2023-11-22 20:23:22,027 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 200, loss[loss=0.08804, simple_loss=0.1163, pruned_loss=0.01666, audio_tagging_loss=0.01321, over 15247.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09496, pruned_loss=0.01469, audio_tagging_loss=0.01327, over 1920334.31 frames. ], batch size: 53, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:23:26,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2085446.6666666667, ans=0.0 2023-11-22 20:23:30,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2085446.6666666667, ans=0.07 2023-11-22 20:23:50,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2085580.0, ans=10.0 2023-11-22 20:23:53,565 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.118e+01 8.443e+01 8.985e+01 9.584e+01 1.364e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 20:23:59,951 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.16 vs. limit=15.0 2023-11-22 20:24:03,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312850 2023-11-22 20:24:03,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.50 vs. limit=10.0 2023-11-22 20:24:05,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2085646.6666666667, ans=0.125 2023-11-22 20:24:23,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2085713.3333333333, ans=0.1 2023-11-22 20:24:28,006 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 250, loss[loss=0.07098, simple_loss=0.09743, pruned_loss=0.01372, audio_tagging_loss=0.008544, over 14694.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.094, pruned_loss=0.01464, audio_tagging_loss=0.01207, over 2166395.50 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:24:29,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2085780.0, ans=0.1 2023-11-22 20:24:51,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2085846.6666666667, ans=0.0 2023-11-22 20:24:59,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2085913.3333333333, ans=0.125 2023-11-22 20:25:03,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2085913.3333333333, ans=0.125 2023-11-22 20:25:07,745 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312900 2023-11-22 20:25:22,883 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0 2023-11-22 20:25:31,867 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 300, loss[loss=0.05227, simple_loss=0.06824, pruned_loss=0.01043, audio_tagging_loss=0.007718, over 14628.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09512, pruned_loss=0.01478, audio_tagging_loss=0.01115, over 2364426.02 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:25:34,774 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0 2023-11-22 20:25:38,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2086113.3333333333, ans=0.1 2023-11-22 20:25:46,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2086180.0, ans=0.125 2023-11-22 20:25:51,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2086180.0, ans=0.0 2023-11-22 20:26:03,559 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.485e+01 8.248e+01 9.049e+01 9.988e+01 1.263e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-22 20:26:07,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2086246.6666666667, ans=0.1 2023-11-22 20:26:12,557 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 312950 2023-11-22 20:26:17,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2086313.3333333333, ans=0.0 2023-11-22 20:26:19,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2086313.3333333333, ans=0.0 2023-11-22 20:26:21,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2086313.3333333333, ans=0.1 2023-11-22 20:26:34,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2086380.0, ans=0.125 2023-11-22 20:26:36,496 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 350, loss[loss=0.06425, simple_loss=0.08086, pruned_loss=0.01188, audio_tagging_loss=0.01194, over 13876.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09428, pruned_loss=0.01464, audio_tagging_loss=0.01065, over 2513881.37 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:26:39,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2086446.6666666667, ans=0.0 2023-11-22 20:26:43,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2086446.6666666667, ans=0.125 2023-11-22 20:26:45,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2086446.6666666667, ans=0.0 2023-11-22 20:26:53,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2086513.3333333333, ans=0.0 2023-11-22 20:27:06,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2086580.0, ans=0.1 2023-11-22 20:27:09,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.02 vs. limit=12.0 2023-11-22 20:27:12,337 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.53 vs. limit=22.5 2023-11-22 20:27:16,430 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313000 2023-11-22 20:27:23,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2086646.6666666667, ans=0.1 2023-11-22 20:27:33,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=15.0 2023-11-22 20:27:33,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2086713.3333333333, ans=0.125 2023-11-22 20:27:42,146 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 400, loss[loss=0.07367, simple_loss=0.1069, pruned_loss=0.01337, audio_tagging_loss=0.006845, over 16171.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09449, pruned_loss=0.01457, audio_tagging_loss=0.01029, over 2629093.16 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:27:45,126 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.72 vs. limit=10.0 2023-11-22 20:27:46,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.14 vs. limit=15.0 2023-11-22 20:27:56,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2086846.6666666667, ans=0.0 2023-11-22 20:27:59,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2086846.6666666667, ans=0.125 2023-11-22 20:28:01,367 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.43 vs. limit=15.0 2023-11-22 20:28:10,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2086913.3333333333, ans=0.0 2023-11-22 20:28:12,329 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.142e+01 8.725e+01 9.349e+01 1.086e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-22 20:28:21,044 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313050 2023-11-22 20:28:29,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.63 vs. limit=15.0 2023-11-22 20:28:31,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2086980.0, ans=0.0 2023-11-22 20:28:38,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.34 vs. limit=12.0 2023-11-22 20:28:45,808 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 450, loss[loss=0.0522, simple_loss=0.05919, pruned_loss=0.007085, audio_tagging_loss=0.01552, over 14924.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09436, pruned_loss=0.01469, audio_tagging_loss=0.009996, over 2720270.64 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:29:19,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2087246.6666666667, ans=0.0 2023-11-22 20:29:22,316 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.37 vs. limit=10.0 2023-11-22 20:29:25,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313100 2023-11-22 20:29:26,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2087313.3333333333, ans=0.125 2023-11-22 20:29:34,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2087313.3333333333, ans=0.125 2023-11-22 20:29:49,308 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 500, loss[loss=0.07161, simple_loss=0.09202, pruned_loss=0.01747, audio_tagging_loss=0.008133, over 15544.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09343, pruned_loss=0.01462, audio_tagging_loss=0.00983, over 2794307.96 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:29:52,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2087446.6666666667, ans=0.125 2023-11-22 20:29:56,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2087446.6666666667, ans=0.0 2023-11-22 20:29:58,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2087446.6666666667, ans=0.125 2023-11-22 20:30:02,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2087513.3333333333, ans=0.05 2023-11-22 20:30:08,170 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 20:30:14,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2087580.0, ans=0.0 2023-11-22 20:30:19,917 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.170e+01 8.718e+01 9.505e+01 1.331e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-22 20:30:20,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2087580.0, ans=0.125 2023-11-22 20:30:21,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2087580.0, ans=0.125 2023-11-22 20:30:28,590 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313150 2023-11-22 20:30:49,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2087713.3333333333, ans=0.1 2023-11-22 20:30:54,211 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 550, loss[loss=0.05676, simple_loss=0.06964, pruned_loss=0.01223, audio_tagging_loss=0.00971, over 14784.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09402, pruned_loss=0.01468, audio_tagging_loss=0.009648, over 2852984.53 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:31:05,822 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.22 vs. limit=15.0 2023-11-22 20:31:09,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.86 vs. limit=22.5 2023-11-22 20:31:32,785 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313200 2023-11-22 20:31:43,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2087980.0, ans=0.07 2023-11-22 20:31:57,883 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 600, loss[loss=0.06937, simple_loss=0.09347, pruned_loss=0.01578, audio_tagging_loss=0.006848, over 16874.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09452, pruned_loss=0.01475, audio_tagging_loss=0.009465, over 2900569.43 frames. ], batch size: 64, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:32:00,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.04 vs. limit=15.0 2023-11-22 20:32:01,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2088113.3333333333, ans=10.0 2023-11-22 20:32:28,506 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.589e+01 9.222e+01 1.004e+02 1.377e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-22 20:32:37,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313250 2023-11-22 20:32:42,205 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.01 vs. limit=15.0 2023-11-22 20:32:46,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2088313.3333333333, ans=0.125 2023-11-22 20:33:01,082 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 650, loss[loss=0.09361, simple_loss=0.1216, pruned_loss=0.02141, audio_tagging_loss=0.01139, over 15024.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09575, pruned_loss=0.01512, audio_tagging_loss=0.009316, over 2929436.73 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:33:01,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.12 vs. limit=6.0 2023-11-22 20:33:07,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2088446.6666666667, ans=0.125 2023-11-22 20:33:13,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2088513.3333333333, ans=0.125 2023-11-22 20:33:16,720 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.51 vs. limit=15.0 2023-11-22 20:33:18,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2088513.3333333333, ans=0.125 2023-11-22 20:33:28,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2088580.0, ans=0.125 2023-11-22 20:33:42,025 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313300 2023-11-22 20:34:03,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2088713.3333333333, ans=10.0 2023-11-22 20:34:03,764 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0 2023-11-22 20:34:06,063 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 700, loss[loss=0.0674, simple_loss=0.09534, pruned_loss=0.01107, audio_tagging_loss=0.008667, over 15604.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09549, pruned_loss=0.01503, audio_tagging_loss=0.009367, over 2956421.33 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:34:34,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2088913.3333333333, ans=0.125 2023-11-22 20:34:35,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2088913.3333333333, ans=0.125 2023-11-22 20:34:36,744 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.373e+01 8.817e+01 9.744e+01 1.427e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-22 20:34:45,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313350 2023-11-22 20:35:06,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.87 vs. limit=15.0 2023-11-22 20:35:11,623 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 750, loss[loss=0.06527, simple_loss=0.08123, pruned_loss=0.01331, audio_tagging_loss=0.01134, over 14957.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09537, pruned_loss=0.01494, audio_tagging_loss=0.009444, over 2978623.71 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:35:17,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2089113.3333333333, ans=0.125 2023-11-22 20:35:28,335 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.36 vs. limit=15.0 2023-11-22 20:35:37,938 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.18 vs. limit=22.5 2023-11-22 20:35:52,100 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313400 2023-11-22 20:35:53,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2089313.3333333333, ans=0.04949747468305833 2023-11-22 20:36:02,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2089380.0, ans=0.1 2023-11-22 20:36:09,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2089380.0, ans=0.1 2023-11-22 20:36:15,363 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 800, loss[loss=0.08263, simple_loss=0.1229, pruned_loss=0.01463, audio_tagging_loss=0.006526, over 15810.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09536, pruned_loss=0.01483, audio_tagging_loss=0.009425, over 2994632.54 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:36:20,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2089446.6666666667, ans=0.1 2023-11-22 20:36:24,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2089446.6666666667, ans=0.0 2023-11-22 20:36:29,734 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.99 vs. limit=22.5 2023-11-22 20:36:47,050 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.197e+01 8.696e+01 9.566e+01 1.219e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-22 20:36:55,804 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313450 2023-11-22 20:36:56,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=2089646.6666666667, ans=0.2 2023-11-22 20:36:58,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2089646.6666666667, ans=0.09899494936611666 2023-11-22 20:37:08,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2089713.3333333333, ans=0.0 2023-11-22 20:37:18,935 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 850, loss[loss=0.06743, simple_loss=0.08727, pruned_loss=0.01461, audio_tagging_loss=0.009191, over 15919.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09457, pruned_loss=0.01457, audio_tagging_loss=0.009493, over 3008752.24 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:37:21,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2089780.0, ans=0.0 2023-11-22 20:37:28,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2089780.0, ans=0.125 2023-11-22 20:37:34,502 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.43 vs. limit=10.0 2023-11-22 20:37:59,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313500 2023-11-22 20:38:07,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2089980.0, ans=0.1 2023-11-22 20:38:18,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=15.0 2023-11-22 20:38:24,650 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 900, loss[loss=0.06744, simple_loss=0.08863, pruned_loss=0.01595, audio_tagging_loss=0.007173, over 15912.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09412, pruned_loss=0.01463, audio_tagging_loss=0.009519, over 3023582.99 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:38:24,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2090113.3333333333, ans=0.1 2023-11-22 20:38:33,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2090113.3333333333, ans=0.125 2023-11-22 20:38:33,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2090113.3333333333, ans=0.1 2023-11-22 20:38:34,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2090113.3333333333, ans=0.125 2023-11-22 20:38:37,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2090180.0, ans=0.125 2023-11-22 20:38:40,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2090180.0, ans=0.125 2023-11-22 20:38:53,873 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.151e+01 8.989e+01 9.712e+01 1.407e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-22 20:38:58,293 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0 2023-11-22 20:39:04,394 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313550 2023-11-22 20:39:18,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2090380.0, ans=0.0 2023-11-22 20:39:27,636 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 950, loss[loss=0.08162, simple_loss=0.1076, pruned_loss=0.02009, audio_tagging_loss=0.007723, over 15014.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09473, pruned_loss=0.01463, audio_tagging_loss=0.009364, over 3029822.78 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:39:31,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2090446.6666666667, ans=0.125 2023-11-22 20:39:51,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.95 vs. limit=22.5 2023-11-22 20:39:55,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2090580.0, ans=0.0 2023-11-22 20:39:58,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.55 vs. limit=15.0 2023-11-22 20:40:07,736 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313600 2023-11-22 20:40:10,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2090646.6666666667, ans=0.05 2023-11-22 20:40:31,674 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1000, loss[loss=0.08148, simple_loss=0.1152, pruned_loss=0.01496, audio_tagging_loss=0.008911, over 15823.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09465, pruned_loss=0.01476, audio_tagging_loss=0.009298, over 3025727.72 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:40:45,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.11 vs. limit=22.5 2023-11-22 20:40:51,486 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.67 vs. limit=10.0 2023-11-22 20:40:57,968 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 20:41:03,830 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.703e+01 8.088e+01 8.617e+01 9.308e+01 1.248e+02, threshold=1.723e+02, percent-clipped=0.0 2023-11-22 20:41:11,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313650 2023-11-22 20:41:11,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2090980.0, ans=0.125 2023-11-22 20:41:36,664 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1050, loss[loss=0.06823, simple_loss=0.08941, pruned_loss=0.01444, audio_tagging_loss=0.009084, over 14988.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09305, pruned_loss=0.01436, audio_tagging_loss=0.009274, over 3029155.48 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:41:47,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2091180.0, ans=0.125 2023-11-22 20:41:55,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2091180.0, ans=0.0 2023-11-22 20:41:56,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2091180.0, ans=0.125 2023-11-22 20:42:14,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2091313.3333333333, ans=0.125 2023-11-22 20:42:15,409 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313700 2023-11-22 20:42:37,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2091380.0, ans=0.04949747468305833 2023-11-22 20:42:39,784 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1100, loss[loss=0.06988, simple_loss=0.09245, pruned_loss=0.01617, audio_tagging_loss=0.007487, over 15380.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09373, pruned_loss=0.01453, audio_tagging_loss=0.009182, over 3036301.04 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:42:42,278 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 20:42:45,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2091446.6666666667, ans=0.0 2023-11-22 20:43:12,478 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 8.123e+01 8.810e+01 9.567e+01 1.299e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-22 20:43:20,172 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313750 2023-11-22 20:43:34,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2091713.3333333333, ans=10.0 2023-11-22 20:43:43,608 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1150, loss[loss=0.06784, simple_loss=0.09534, pruned_loss=0.01035, audio_tagging_loss=0.009823, over 14871.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09336, pruned_loss=0.01458, audio_tagging_loss=0.00917, over 3033144.59 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:43:47,831 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 20:44:23,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313800 2023-11-22 20:44:30,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2091980.0, ans=0.125 2023-11-22 20:44:49,084 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1200, loss[loss=0.0822, simple_loss=0.1002, pruned_loss=0.01872, audio_tagging_loss=0.01338, over 15266.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09349, pruned_loss=0.01461, audio_tagging_loss=0.009166, over 3030157.41 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:44:56,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2092113.3333333333, ans=0.1 2023-11-22 20:45:14,799 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.44 vs. limit=12.0 2023-11-22 20:45:18,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2092246.6666666667, ans=0.125 2023-11-22 20:45:19,786 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.331e+01 9.096e+01 9.827e+01 1.345e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-22 20:45:20,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2092246.6666666667, ans=0.125 2023-11-22 20:45:27,200 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313850 2023-11-22 20:45:34,097 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 20:45:45,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2092380.0, ans=0.125 2023-11-22 20:45:52,590 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1250, loss[loss=0.07326, simple_loss=0.102, pruned_loss=0.01663, audio_tagging_loss=0.005644, over 14470.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09235, pruned_loss=0.01452, audio_tagging_loss=0.009119, over 3030899.48 frames. ], batch size: 53, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:45:58,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2092446.6666666667, ans=0.125 2023-11-22 20:46:03,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.30 vs. limit=10.0 2023-11-22 20:46:30,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.25 vs. limit=15.0 2023-11-22 20:46:32,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313900 2023-11-22 20:46:36,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2092646.6666666667, ans=0.0 2023-11-22 20:46:45,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2092713.3333333333, ans=0.125 2023-11-22 20:46:50,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2092713.3333333333, ans=0.025 2023-11-22 20:46:56,826 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1300, loss[loss=0.06308, simple_loss=0.08274, pruned_loss=0.01309, audio_tagging_loss=0.008619, over 15031.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09235, pruned_loss=0.01444, audio_tagging_loss=0.009076, over 3027779.52 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:47:14,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2092846.6666666667, ans=0.1 2023-11-22 20:47:24,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2092913.3333333333, ans=0.0 2023-11-22 20:47:29,475 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.264e+01 8.633e+01 9.270e+01 1.307e+02, threshold=1.727e+02, percent-clipped=0.0 2023-11-22 20:47:36,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 313950 2023-11-22 20:47:49,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2093046.6666666667, ans=0.0 2023-11-22 20:48:01,445 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1350, loss[loss=0.07638, simple_loss=0.1034, pruned_loss=0.01688, audio_tagging_loss=0.007807, over 15267.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09214, pruned_loss=0.0145, audio_tagging_loss=0.009113, over 3039497.65 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:48:03,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2093113.3333333333, ans=0.1 2023-11-22 20:48:40,486 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314000 2023-11-22 20:48:40,893 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.70 vs. limit=15.0 2023-11-22 20:48:47,381 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 20:48:52,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2093380.0, ans=0.125 2023-11-22 20:49:05,255 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1400, loss[loss=0.06549, simple_loss=0.08806, pruned_loss=0.01191, audio_tagging_loss=0.009559, over 15390.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09255, pruned_loss=0.01461, audio_tagging_loss=0.00915, over 3041332.03 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:49:17,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.39 vs. limit=15.0 2023-11-22 20:49:19,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2093513.3333333333, ans=0.125 2023-11-22 20:49:38,505 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.151e+01 8.829e+01 9.680e+01 1.142e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 20:49:44,801 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314050 2023-11-22 20:50:01,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2093713.3333333333, ans=0.125 2023-11-22 20:50:08,771 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1450, loss[loss=0.07518, simple_loss=0.1085, pruned_loss=0.01504, audio_tagging_loss=0.005899, over 15231.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09289, pruned_loss=0.01467, audio_tagging_loss=0.009195, over 3044305.44 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:50:21,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2093846.6666666667, ans=0.2 2023-11-22 20:50:39,414 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0 2023-11-22 20:50:49,256 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314100 2023-11-22 20:51:12,968 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1500, loss[loss=0.06358, simple_loss=0.07601, pruned_loss=0.01443, audio_tagging_loss=0.01115, over 14680.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09211, pruned_loss=0.01449, audio_tagging_loss=0.009371, over 3053897.57 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:51:20,592 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.24 vs. limit=15.0 2023-11-22 20:51:25,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2094180.0, ans=0.0 2023-11-22 20:51:28,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2094180.0, ans=0.1 2023-11-22 20:51:33,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2094180.0, ans=0.125 2023-11-22 20:51:46,471 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.220e+01 8.857e+01 9.629e+01 1.228e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-22 20:51:52,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314150 2023-11-22 20:51:53,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2094313.3333333333, ans=0.125 2023-11-22 20:51:57,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2094313.3333333333, ans=0.0 2023-11-22 20:52:01,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2094313.3333333333, ans=0.2 2023-11-22 20:52:17,417 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1550, loss[loss=0.06248, simple_loss=0.07783, pruned_loss=0.01102, audio_tagging_loss=0.01255, over 14960.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09243, pruned_loss=0.01453, audio_tagging_loss=0.0095, over 3050921.88 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 20:52:22,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2094446.6666666667, ans=0.2 2023-11-22 20:52:40,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2094513.3333333333, ans=0.125 2023-11-22 20:52:49,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2094580.0, ans=0.1 2023-11-22 20:52:57,416 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314200 2023-11-22 20:53:02,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2094646.6666666667, ans=0.04949747468305833 2023-11-22 20:53:02,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2094646.6666666667, ans=0.95 2023-11-22 20:53:21,765 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1600, loss[loss=0.09304, simple_loss=0.137, pruned_loss=0.01962, audio_tagging_loss=0.00493, over 15754.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09199, pruned_loss=0.01456, audio_tagging_loss=0.009535, over 3053232.90 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:53:28,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.21 vs. limit=8.0 2023-11-22 20:53:43,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2094846.6666666667, ans=0.125 2023-11-22 20:53:54,981 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.130e+01 8.806e+01 9.580e+01 1.162e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-22 20:53:56,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2094913.3333333333, ans=0.125 2023-11-22 20:54:01,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314250 2023-11-22 20:54:08,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2094980.0, ans=0.125 2023-11-22 20:54:08,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2094980.0, ans=0.2 2023-11-22 20:54:15,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2095046.6666666667, ans=0.0 2023-11-22 20:54:21,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2095046.6666666667, ans=0.2 2023-11-22 20:54:25,234 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1650, loss[loss=0.06538, simple_loss=0.08938, pruned_loss=0.01185, audio_tagging_loss=0.008839, over 16026.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09302, pruned_loss=0.01479, audio_tagging_loss=0.009457, over 3051423.89 frames. ], batch size: 62, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:54:27,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2095113.3333333333, ans=0.125 2023-11-22 20:54:48,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2095180.0, ans=0.1 2023-11-22 20:54:51,092 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.40 vs. limit=15.0 2023-11-22 20:55:05,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314300 2023-11-22 20:55:08,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2095313.3333333333, ans=0.05 2023-11-22 20:55:21,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2095380.0, ans=0.2 2023-11-22 20:55:25,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2095380.0, ans=0.125 2023-11-22 20:55:30,022 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1700, loss[loss=0.07049, simple_loss=0.08964, pruned_loss=0.01471, audio_tagging_loss=0.01096, over 14873.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09302, pruned_loss=0.01474, audio_tagging_loss=0.009452, over 3050711.52 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:55:34,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2095446.6666666667, ans=0.0 2023-11-22 20:55:35,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2095446.6666666667, ans=0.2 2023-11-22 20:55:49,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2095513.3333333333, ans=0.125 2023-11-22 20:55:52,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2095513.3333333333, ans=0.125 2023-11-22 20:56:02,461 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.152e+01 8.274e+01 8.793e+01 9.384e+01 1.139e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-22 20:56:09,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2095646.6666666667, ans=15.0 2023-11-22 20:56:10,147 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314350 2023-11-22 20:56:16,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2095646.6666666667, ans=0.125 2023-11-22 20:56:33,708 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1750, loss[loss=0.04955, simple_loss=0.05437, pruned_loss=0.00928, audio_tagging_loss=0.01309, over 15235.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09276, pruned_loss=0.01477, audio_tagging_loss=0.009361, over 3056294.89 frames. ], batch size: 60, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:56:42,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2095780.0, ans=10.0 2023-11-22 20:56:44,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2095780.0, ans=0.0 2023-11-22 20:56:48,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2095846.6666666667, ans=10.0 2023-11-22 20:56:48,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2095846.6666666667, ans=0.0 2023-11-22 20:56:53,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2095846.6666666667, ans=0.0 2023-11-22 20:56:54,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2095846.6666666667, ans=0.015 2023-11-22 20:57:11,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.97 vs. limit=12.0 2023-11-22 20:57:13,184 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.68 vs. limit=12.0 2023-11-22 20:57:14,011 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314400 2023-11-22 20:57:16,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2095980.0, ans=0.0 2023-11-22 20:57:32,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2096046.6666666667, ans=0.125 2023-11-22 20:57:38,165 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1800, loss[loss=0.05673, simple_loss=0.07752, pruned_loss=0.01093, audio_tagging_loss=0.007036, over 15161.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.0931, pruned_loss=0.01462, audio_tagging_loss=0.009278, over 3056737.54 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:58:09,792 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.09 vs. limit=12.0 2023-11-22 20:58:11,548 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 8.025e+01 8.684e+01 9.480e+01 1.547e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-22 20:58:14,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2096246.6666666667, ans=0.1 2023-11-22 20:58:17,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314450 2023-11-22 20:58:42,932 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1850, loss[loss=0.07006, simple_loss=0.102, pruned_loss=0.0127, audio_tagging_loss=0.006389, over 16152.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09293, pruned_loss=0.01455, audio_tagging_loss=0.009286, over 3050789.07 frames. ], batch size: 62, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:59:13,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2096580.0, ans=0.0 2023-11-22 20:59:22,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314500 2023-11-22 20:59:35,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2096713.3333333333, ans=0.125 2023-11-22 20:59:43,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.88 vs. limit=10.0 2023-11-22 20:59:46,114 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1900, loss[loss=0.05957, simple_loss=0.0815, pruned_loss=0.01018, audio_tagging_loss=0.008641, over 14370.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09225, pruned_loss=0.01443, audio_tagging_loss=0.009263, over 3051460.49 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 20:59:52,620 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:00:07,836 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:00:19,874 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 7.994e+01 8.463e+01 9.067e+01 1.382e+02, threshold=1.693e+02, percent-clipped=0.0 2023-11-22 21:00:21,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2096913.3333333333, ans=0.125 2023-11-22 21:00:26,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314550 2023-11-22 21:00:38,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2097046.6666666667, ans=10.0 2023-11-22 21:00:49,516 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 1950, loss[loss=0.07605, simple_loss=0.1075, pruned_loss=0.01475, audio_tagging_loss=0.007555, over 16075.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09229, pruned_loss=0.01442, audio_tagging_loss=0.009173, over 3046746.78 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:00:54,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2097113.3333333333, ans=0.125 2023-11-22 21:00:57,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2097113.3333333333, ans=0.0 2023-11-22 21:01:18,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2097246.6666666665, ans=0.2 2023-11-22 21:01:20,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2097246.6666666665, ans=0.125 2023-11-22 21:01:26,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2097313.3333333335, ans=0.1 2023-11-22 21:01:27,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2097313.3333333335, ans=0.2 2023-11-22 21:01:28,631 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314600 2023-11-22 21:01:43,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2097380.0, ans=0.1 2023-11-22 21:01:54,206 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2000, loss[loss=0.0984, simple_loss=0.1332, pruned_loss=0.02043, audio_tagging_loss=0.01136, over 15701.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.0927, pruned_loss=0.01459, audio_tagging_loss=0.009262, over 3052022.21 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:01:55,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2097446.6666666665, ans=10.0 2023-11-22 21:01:59,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2097446.6666666665, ans=0.0 2023-11-22 21:02:01,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2097446.6666666665, ans=0.0 2023-11-22 21:02:05,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2097513.3333333335, ans=0.125 2023-11-22 21:02:16,926 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.82 vs. limit=15.0 2023-11-22 21:02:26,122 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.310e+01 8.293e+01 8.864e+01 9.588e+01 1.359e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-22 21:02:29,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2097580.0, ans=0.125 2023-11-22 21:02:32,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314650 2023-11-22 21:02:34,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0 2023-11-22 21:02:40,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2097646.6666666665, ans=0.2 2023-11-22 21:02:45,080 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:02:51,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2097713.3333333335, ans=0.0 2023-11-22 21:02:51,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2097713.3333333335, ans=0.125 2023-11-22 21:02:57,057 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2050, loss[loss=0.06435, simple_loss=0.08894, pruned_loss=0.01233, audio_tagging_loss=0.007551, over 14590.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09348, pruned_loss=0.01483, audio_tagging_loss=0.009191, over 3053935.45 frames. ], batch size: 53, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:03:08,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2097846.6666666665, ans=0.1 2023-11-22 21:03:26,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2097913.3333333335, ans=0.125 2023-11-22 21:03:37,206 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314700 2023-11-22 21:03:47,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2098046.6666666665, ans=0.025 2023-11-22 21:03:49,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2098046.6666666665, ans=0.125 2023-11-22 21:04:00,566 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2100, loss[loss=0.0845, simple_loss=0.09766, pruned_loss=0.02447, audio_tagging_loss=0.0112, over 16367.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09358, pruned_loss=0.01485, audio_tagging_loss=0.009085, over 3046765.94 frames. ], batch size: 63, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:04:34,481 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.684e+01 9.132e+01 9.760e+01 1.245e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-22 21:04:37,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2098246.6666666665, ans=0.0 2023-11-22 21:04:39,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2098313.3333333335, ans=0.125 2023-11-22 21:04:40,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314750 2023-11-22 21:04:51,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2098380.0, ans=0.0 2023-11-22 21:04:54,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.whiten.whitening_limit, batch_count=2098380.0, ans=12.0 2023-11-22 21:05:05,810 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2150, loss[loss=0.06615, simple_loss=0.0871, pruned_loss=0.01277, audio_tagging_loss=0.009825, over 15206.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09364, pruned_loss=0.01484, audio_tagging_loss=0.009032, over 3047522.88 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:05:42,526 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:05:43,780 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314800 2023-11-22 21:05:43,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2098646.6666666665, ans=0.1 2023-11-22 21:06:08,670 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2200, loss[loss=0.06804, simple_loss=0.08739, pruned_loss=0.01641, audio_tagging_loss=0.007937, over 15718.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09404, pruned_loss=0.01487, audio_tagging_loss=0.009054, over 3046818.18 frames. ], batch size: 60, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:06:08,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2098780.0, ans=0.0 2023-11-22 21:06:20,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2098846.6666666665, ans=0.2 2023-11-22 21:06:28,542 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:06:42,190 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.590e+01 8.202e+01 8.834e+01 9.532e+01 1.382e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-22 21:06:48,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314850 2023-11-22 21:06:52,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2098980.0, ans=0.125 2023-11-22 21:06:53,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.92 vs. limit=15.0 2023-11-22 21:06:55,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2098980.0, ans=0.125 2023-11-22 21:07:00,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2099046.6666666665, ans=0.05 2023-11-22 21:07:06,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2099046.6666666665, ans=15.0 2023-11-22 21:07:11,834 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2250, loss[loss=0.07, simple_loss=0.08945, pruned_loss=0.0171, audio_tagging_loss=0.008178, over 14632.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09433, pruned_loss=0.01481, audio_tagging_loss=0.008997, over 3042223.50 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:07:29,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2099180.0, ans=0.025 2023-11-22 21:07:42,458 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.00 vs. limit=15.0 2023-11-22 21:07:51,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314900 2023-11-22 21:08:07,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2099380.0, ans=0.0 2023-11-22 21:08:15,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2099446.6666666665, ans=0.0 2023-11-22 21:08:15,201 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:08:15,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.29 vs. limit=15.0 2023-11-22 21:08:16,725 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2300, loss[loss=0.06181, simple_loss=0.08087, pruned_loss=0.01328, audio_tagging_loss=0.008097, over 15084.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09424, pruned_loss=0.01474, audio_tagging_loss=0.009159, over 3037641.89 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 21:08:25,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2099446.6666666665, ans=0.1 2023-11-22 21:08:35,506 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.86 vs. limit=15.0 2023-11-22 21:08:39,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2099513.3333333335, ans=0.125 2023-11-22 21:08:44,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2099580.0, ans=0.05 2023-11-22 21:08:48,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.84 vs. limit=10.0 2023-11-22 21:08:50,535 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.386e+01 8.086e+01 8.698e+01 9.161e+01 1.218e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-22 21:08:52,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2099580.0, ans=0.125 2023-11-22 21:08:55,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 314950 2023-11-22 21:09:08,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2099713.3333333335, ans=0.0 2023-11-22 21:09:10,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2099713.3333333335, ans=0.0 2023-11-22 21:09:10,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2099713.3333333335, ans=0.2 2023-11-22 21:09:13,534 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:09:20,826 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2350, loss[loss=0.0677, simple_loss=0.0941, pruned_loss=0.01234, audio_tagging_loss=0.008316, over 14446.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09373, pruned_loss=0.01476, audio_tagging_loss=0.009258, over 3037032.83 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 21:09:36,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.09 vs. limit=10.0 2023-11-22 21:09:45,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2099913.3333333335, ans=0.0 2023-11-22 21:10:00,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315000 2023-11-22 21:10:24,669 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2400, loss[loss=0.07621, simple_loss=0.09309, pruned_loss=0.02079, audio_tagging_loss=0.008881, over 15359.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09382, pruned_loss=0.0148, audio_tagging_loss=0.009407, over 3043447.12 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0 2023-11-22 21:10:27,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.01 vs. limit=15.0 2023-11-22 21:10:40,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2100180.0, ans=0.125 2023-11-22 21:10:57,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2100246.6666666665, ans=0.09899494936611666 2023-11-22 21:11:00,962 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.260e+01 8.417e+01 9.057e+01 9.650e+01 1.239e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-22 21:11:04,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315050 2023-11-22 21:11:26,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2100380.0, ans=0.0 2023-11-22 21:11:28,999 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2450, loss[loss=0.06838, simple_loss=0.0837, pruned_loss=0.0141, audio_tagging_loss=0.01243, over 15377.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09489, pruned_loss=0.01489, audio_tagging_loss=0.009453, over 3051094.61 frames. ], batch size: 61, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 21:11:30,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2100446.6666666665, ans=0.125 2023-11-22 21:11:36,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2100446.6666666665, ans=0.0 2023-11-22 21:11:41,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2100513.3333333335, ans=0.125 2023-11-22 21:11:56,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2100580.0, ans=0.125 2023-11-22 21:12:08,055 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315100 2023-11-22 21:12:09,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2100646.6666666665, ans=0.1 2023-11-22 21:12:33,349 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2500, loss[loss=0.06556, simple_loss=0.09215, pruned_loss=0.01045, audio_tagging_loss=0.009044, over 15475.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09423, pruned_loss=0.01483, audio_tagging_loss=0.009566, over 3044889.35 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 21:13:02,224 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.25 vs. limit=15.0 2023-11-22 21:13:08,896 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.214e+01 8.930e+01 9.863e+01 1.228e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 21:13:13,298 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315150 2023-11-22 21:13:20,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2100980.0, ans=0.0 2023-11-22 21:13:28,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2101046.6666666665, ans=0.125 2023-11-22 21:13:36,981 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2550, loss[loss=0.07735, simple_loss=0.1017, pruned_loss=0.01682, audio_tagging_loss=0.009676, over 14771.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09321, pruned_loss=0.01464, audio_tagging_loss=0.009543, over 3035872.52 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0 2023-11-22 21:13:39,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2101113.3333333335, ans=0.125 2023-11-22 21:13:41,981 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:13:52,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2101180.0, ans=0.125 2023-11-22 21:14:07,492 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.81 vs. limit=15.0 2023-11-22 21:14:17,287 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315200 2023-11-22 21:14:40,764 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2600, loss[loss=0.08919, simple_loss=0.1171, pruned_loss=0.02155, audio_tagging_loss=0.009115, over 15648.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09249, pruned_loss=0.01456, audio_tagging_loss=0.009365, over 3036292.29 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:15:00,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2101513.3333333335, ans=0.125 2023-11-22 21:15:13,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2101580.0, ans=0.0 2023-11-22 21:15:15,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2101580.0, ans=0.0 2023-11-22 21:15:17,713 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 5.989e+01 8.300e+01 8.753e+01 9.403e+01 1.364e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-22 21:15:21,572 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315250 2023-11-22 21:15:21,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2101646.6666666665, ans=0.125 2023-11-22 21:15:24,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.07 vs. limit=10.0 2023-11-22 21:15:27,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2101646.6666666665, ans=0.125 2023-11-22 21:15:30,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2101646.6666666665, ans=0.125 2023-11-22 21:15:43,392 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0 2023-11-22 21:15:46,431 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2650, loss[loss=0.07435, simple_loss=0.0971, pruned_loss=0.01649, audio_tagging_loss=0.00931, over 15539.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09269, pruned_loss=0.01449, audio_tagging_loss=0.00929, over 3037862.85 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:15:51,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2101780.0, ans=0.2 2023-11-22 21:16:09,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.54 vs. limit=15.0 2023-11-22 21:16:21,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2101913.3333333335, ans=0.125 2023-11-22 21:16:26,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315300 2023-11-22 21:16:30,921 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.59 vs. limit=15.0 2023-11-22 21:16:35,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2101980.0, ans=0.2 2023-11-22 21:16:42,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2102046.6666666665, ans=0.0 2023-11-22 21:16:50,641 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2700, loss[loss=0.07153, simple_loss=0.09843, pruned_loss=0.01195, audio_tagging_loss=0.01036, over 15108.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.0934, pruned_loss=0.01454, audio_tagging_loss=0.009225, over 3035332.80 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:16:59,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2102113.3333333335, ans=0.2 2023-11-22 21:17:26,858 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.486e+01 8.121e+01 8.686e+01 9.401e+01 1.913e+02, threshold=1.737e+02, percent-clipped=1.0 2023-11-22 21:17:30,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315350 2023-11-22 21:17:37,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2102313.3333333335, ans=0.125 2023-11-22 21:17:41,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2102380.0, ans=0.125 2023-11-22 21:17:54,227 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2750, loss[loss=0.08685, simple_loss=0.1138, pruned_loss=0.02164, audio_tagging_loss=0.008316, over 15287.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.0928, pruned_loss=0.01438, audio_tagging_loss=0.009251, over 3039973.42 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:17:57,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2102446.6666666665, ans=0.125 2023-11-22 21:17:59,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2102446.6666666665, ans=0.2 2023-11-22 21:18:16,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2102513.3333333335, ans=0.1 2023-11-22 21:18:34,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315400 2023-11-22 21:18:39,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=2102646.6666666665, ans=22.5 2023-11-22 21:18:50,958 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:19:00,008 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2800, loss[loss=0.08086, simple_loss=0.108, pruned_loss=0.01771, audio_tagging_loss=0.009171, over 15984.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.0927, pruned_loss=0.01435, audio_tagging_loss=0.009187, over 3034526.09 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:19:16,947 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:19:33,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2102913.3333333335, ans=0.2 2023-11-22 21:19:36,992 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.680e+01 8.174e+01 8.854e+01 9.674e+01 2.365e+02, threshold=1.771e+02, percent-clipped=1.0 2023-11-22 21:19:40,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315450 2023-11-22 21:19:45,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2102980.0, ans=0.0 2023-11-22 21:20:03,930 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2850, loss[loss=0.08622, simple_loss=0.1207, pruned_loss=0.01702, audio_tagging_loss=0.008831, over 15430.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09263, pruned_loss=0.01455, audio_tagging_loss=0.009164, over 3034473.40 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:20:08,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2103113.3333333335, ans=0.025 2023-11-22 21:20:09,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2103113.3333333335, ans=0.0 2023-11-22 21:20:28,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2103180.0, ans=0.125 2023-11-22 21:20:32,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2103246.6666666665, ans=0.125 2023-11-22 21:20:44,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315500 2023-11-22 21:20:54,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.22 vs. limit=15.0 2023-11-22 21:20:55,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2103380.0, ans=0.2 2023-11-22 21:21:08,287 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2900, loss[loss=0.06086, simple_loss=0.08026, pruned_loss=0.01138, audio_tagging_loss=0.009341, over 15062.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09344, pruned_loss=0.01447, audio_tagging_loss=0.009052, over 3032624.32 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:21:33,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2103580.0, ans=0.0 2023-11-22 21:21:39,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2103580.0, ans=0.1 2023-11-22 21:21:45,629 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.319e+01 9.006e+01 9.775e+01 1.156e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-22 21:21:48,830 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315550 2023-11-22 21:21:48,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2103646.6666666665, ans=0.2 2023-11-22 21:21:54,168 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.77 vs. limit=15.0 2023-11-22 21:21:56,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2103646.6666666665, ans=0.125 2023-11-22 21:22:01,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2103713.3333333335, ans=0.125 2023-11-22 21:22:13,425 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 2950, loss[loss=0.05572, simple_loss=0.07157, pruned_loss=0.01038, audio_tagging_loss=0.009558, over 13231.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09396, pruned_loss=0.01474, audio_tagging_loss=0.009062, over 3034249.31 frames. ], batch size: 53, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:22:23,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2103780.0, ans=0.125 2023-11-22 21:22:48,811 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.03 vs. limit=12.0 2023-11-22 21:22:52,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315600 2023-11-22 21:23:03,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.36 vs. limit=6.0 2023-11-22 21:23:11,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2104046.6666666665, ans=0.0 2023-11-22 21:23:17,727 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3000, loss[loss=0.07722, simple_loss=0.1003, pruned_loss=0.01652, audio_tagging_loss=0.01057, over 13972.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09386, pruned_loss=0.01465, audio_tagging_loss=0.009146, over 3041807.08 frames. ], batch size: 52, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:23:17,728 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 21:23:59,568 INFO [train_asr.py:1253] (3/4) Epoch 27, validation: loss=0.058, simple_loss=0.05133, pruned_loss=0.005079, audio_tagging_loss=0.02726, over 4681554.00 frames. 2023-11-22 21:23:59,569 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 21:24:28,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2104246.6666666665, ans=0.125 2023-11-22 21:24:36,799 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.281e+01 8.900e+01 9.558e+01 2.196e+02, threshold=1.780e+02, percent-clipped=1.0 2023-11-22 21:24:39,339 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315650 2023-11-22 21:25:04,722 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3050, loss[loss=0.06422, simple_loss=0.07677, pruned_loss=0.01282, audio_tagging_loss=0.01302, over 15079.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09321, pruned_loss=0.01468, audio_tagging_loss=0.00929, over 3038453.56 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:25:11,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2104446.6666666665, ans=0.125 2023-11-22 21:25:29,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.96 vs. limit=15.0 2023-11-22 21:25:41,309 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:25:43,906 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315700 2023-11-22 21:25:44,524 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.60 vs. limit=15.0 2023-11-22 21:25:58,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.whiten.whitening_limit, batch_count=2104713.3333333335, ans=15.0 2023-11-22 21:26:07,723 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3100, loss[loss=0.0746, simple_loss=0.1013, pruned_loss=0.01437, audio_tagging_loss=0.009582, over 14774.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09457, pruned_loss=0.01486, audio_tagging_loss=0.009321, over 3041052.76 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:26:11,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2104780.0, ans=0.1 2023-11-22 21:26:26,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2104846.6666666665, ans=0.125 2023-11-22 21:26:37,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2104913.3333333335, ans=0.2 2023-11-22 21:26:44,228 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.175e+01 8.886e+01 9.764e+01 1.279e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 21:26:44,582 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:26:46,788 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315750 2023-11-22 21:26:56,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2105046.6666666665, ans=0.125 2023-11-22 21:27:09,963 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3150, loss[loss=0.0824, simple_loss=0.122, pruned_loss=0.01403, audio_tagging_loss=0.007388, over 15590.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09417, pruned_loss=0.0148, audio_tagging_loss=0.009316, over 3037877.53 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:27:39,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2105246.6666666665, ans=0.125 2023-11-22 21:27:47,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2105313.3333333335, ans=0.125 2023-11-22 21:27:50,033 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315800 2023-11-22 21:28:06,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2105380.0, ans=0.5 2023-11-22 21:28:09,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2105380.0, ans=0.125 2023-11-22 21:28:12,242 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.04 vs. limit=15.0 2023-11-22 21:28:16,021 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3200, loss[loss=0.0874, simple_loss=0.117, pruned_loss=0.01827, audio_tagging_loss=0.01062, over 15855.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09371, pruned_loss=0.01465, audio_tagging_loss=0.009386, over 3032302.45 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:28:18,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2105446.6666666665, ans=0.125 2023-11-22 21:28:24,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2105446.6666666665, ans=0.125 2023-11-22 21:28:36,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2105513.3333333335, ans=0.125 2023-11-22 21:28:46,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2105580.0, ans=0.125 2023-11-22 21:28:53,854 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.364e+01 8.971e+01 9.873e+01 1.649e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-22 21:28:54,536 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.01 vs. limit=22.5 2023-11-22 21:28:55,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315850 2023-11-22 21:29:01,954 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.34 vs. limit=22.5 2023-11-22 21:29:02,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2105646.6666666665, ans=0.0 2023-11-22 21:29:04,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2105646.6666666665, ans=0.125 2023-11-22 21:29:20,011 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3250, loss[loss=0.0729, simple_loss=0.09599, pruned_loss=0.01548, audio_tagging_loss=0.009423, over 15388.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09355, pruned_loss=0.0147, audio_tagging_loss=0.009447, over 3036945.00 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:29:20,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2105780.0, ans=0.125 2023-11-22 21:29:24,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2105780.0, ans=0.0 2023-11-22 21:29:38,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2105846.6666666665, ans=0.125 2023-11-22 21:29:54,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2105913.3333333335, ans=0.1 2023-11-22 21:29:58,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.92 vs. limit=15.0 2023-11-22 21:30:00,782 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315900 2023-11-22 21:30:05,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2105980.0, ans=0.2 2023-11-22 21:30:09,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.30 vs. limit=15.0 2023-11-22 21:30:13,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2106046.6666666665, ans=0.2 2023-11-22 21:30:24,053 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3300, loss[loss=0.06976, simple_loss=0.09703, pruned_loss=0.01291, audio_tagging_loss=0.008336, over 16230.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09455, pruned_loss=0.01486, audio_tagging_loss=0.009581, over 3046943.26 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:30:40,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2106180.0, ans=0.125 2023-11-22 21:30:59,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2106246.6666666665, ans=0.0 2023-11-22 21:31:02,499 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.403e+01 9.072e+01 9.867e+01 1.352e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-22 21:31:03,948 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 315950 2023-11-22 21:31:27,785 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3350, loss[loss=0.06967, simple_loss=0.09035, pruned_loss=0.01485, audio_tagging_loss=0.009645, over 14233.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09386, pruned_loss=0.01464, audio_tagging_loss=0.00956, over 3046331.47 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:31:48,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2106513.3333333335, ans=0.125 2023-11-22 21:31:50,564 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.57 vs. limit=15.0 2023-11-22 21:32:01,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2106580.0, ans=0.125 2023-11-22 21:32:07,812 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316000 2023-11-22 21:32:29,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.08 vs. limit=15.0 2023-11-22 21:32:30,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2106713.3333333335, ans=0.0 2023-11-22 21:32:35,462 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3400, loss[loss=0.06949, simple_loss=0.1006, pruned_loss=0.01221, audio_tagging_loss=0.006988, over 15205.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09468, pruned_loss=0.0147, audio_tagging_loss=0.009298, over 3052470.10 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:32:51,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2106846.6666666665, ans=0.125 2023-11-22 21:32:54,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2106846.6666666665, ans=0.0 2023-11-22 21:33:03,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2106913.3333333335, ans=0.125 2023-11-22 21:33:14,296 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.192e+01 8.214e+01 8.729e+01 9.303e+01 1.198e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-22 21:33:15,700 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316050 2023-11-22 21:33:34,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2107046.6666666665, ans=10.0 2023-11-22 21:33:39,068 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3450, loss[loss=0.0609, simple_loss=0.06759, pruned_loss=0.01715, audio_tagging_loss=0.009957, over 16231.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09337, pruned_loss=0.0146, audio_tagging_loss=0.00915, over 3050930.61 frames. ], batch size: 63, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:33:39,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.81 vs. limit=15.0 2023-11-22 21:34:14,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2107246.6666666665, ans=0.125 2023-11-22 21:34:15,655 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.39 vs. limit=15.0 2023-11-22 21:34:18,740 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316100 2023-11-22 21:34:43,097 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3500, loss[loss=0.05603, simple_loss=0.0757, pruned_loss=0.008478, audio_tagging_loss=0.009701, over 14796.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09284, pruned_loss=0.01455, audio_tagging_loss=0.00918, over 3050486.15 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:34:58,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2107513.3333333335, ans=0.125 2023-11-22 21:35:16,221 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:35:20,934 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 8.106e+01 8.631e+01 9.458e+01 1.678e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-22 21:35:22,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316150 2023-11-22 21:35:31,283 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.58 vs. limit=22.5 2023-11-22 21:35:38,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.25 vs. limit=15.0 2023-11-22 21:35:47,507 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3550, loss[loss=0.08721, simple_loss=0.1136, pruned_loss=0.02129, audio_tagging_loss=0.009129, over 14653.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09218, pruned_loss=0.0144, audio_tagging_loss=0.009238, over 3049337.98 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:35:50,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2107780.0, ans=0.125 2023-11-22 21:35:53,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2107780.0, ans=0.125 2023-11-22 21:35:58,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2107846.6666666665, ans=0.125 2023-11-22 21:36:16,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff2.min_abs, batch_count=2107913.3333333335, ans=0.1 2023-11-22 21:36:27,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316200 2023-11-22 21:36:40,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2108046.6666666665, ans=0.125 2023-11-22 21:36:42,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2108046.6666666665, ans=0.125 2023-11-22 21:36:45,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2108046.6666666665, ans=0.125 2023-11-22 21:36:49,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2108046.6666666665, ans=0.2 2023-11-22 21:36:51,770 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3600, loss[loss=0.09067, simple_loss=0.129, pruned_loss=0.01932, audio_tagging_loss=0.006848, over 16256.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09307, pruned_loss=0.01452, audio_tagging_loss=0.009089, over 3054442.15 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:36:58,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2108113.3333333335, ans=0.1 2023-11-22 21:37:02,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2108113.3333333335, ans=0.125 2023-11-22 21:37:12,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2108180.0, ans=0.0 2023-11-22 21:37:27,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2108246.6666666665, ans=0.125 2023-11-22 21:37:30,455 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.329e+01 8.258e+01 8.715e+01 9.542e+01 1.505e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-22 21:37:31,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316250 2023-11-22 21:37:42,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2108380.0, ans=0.125 2023-11-22 21:37:43,130 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:37:44,658 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.03 vs. limit=22.5 2023-11-22 21:37:47,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2108380.0, ans=0.0 2023-11-22 21:37:56,466 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3650, loss[loss=0.06894, simple_loss=0.09842, pruned_loss=0.013, audio_tagging_loss=0.006726, over 15183.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09286, pruned_loss=0.01446, audio_tagging_loss=0.009086, over 3045708.84 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:38:10,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2108513.3333333335, ans=0.125 2023-11-22 21:38:22,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2108580.0, ans=0.125 2023-11-22 21:38:35,926 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316300 2023-11-22 21:38:45,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2108646.6666666665, ans=0.125 2023-11-22 21:38:52,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2108713.3333333335, ans=0.1 2023-11-22 21:39:00,755 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3700, loss[loss=0.05738, simple_loss=0.0761, pruned_loss=0.01021, audio_tagging_loss=0.009118, over 15423.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09324, pruned_loss=0.01465, audio_tagging_loss=0.009048, over 3053126.84 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:39:09,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2108780.0, ans=0.125 2023-11-22 21:39:24,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2108913.3333333335, ans=0.125 2023-11-22 21:39:26,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2108913.3333333335, ans=0.125 2023-11-22 21:39:40,494 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.949e+01 8.416e+01 9.143e+01 1.024e+02 1.510e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-22 21:39:40,636 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316350 2023-11-22 21:39:59,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2109046.6666666665, ans=0.2 2023-11-22 21:40:05,171 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3750, loss[loss=0.07938, simple_loss=0.1056, pruned_loss=0.01818, audio_tagging_loss=0.008418, over 14780.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09323, pruned_loss=0.01465, audio_tagging_loss=0.009109, over 3052306.50 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:40:12,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2109113.3333333335, ans=0.1 2023-11-22 21:40:14,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2109113.3333333335, ans=0.1 2023-11-22 21:40:17,340 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:40:18,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2109180.0, ans=0.125 2023-11-22 21:40:27,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2109180.0, ans=0.2 2023-11-22 21:40:37,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2109246.6666666665, ans=0.2 2023-11-22 21:40:45,169 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316400 2023-11-22 21:40:50,969 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:40:58,477 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 21:41:03,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2109380.0, ans=0.0 2023-11-22 21:41:10,556 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3800, loss[loss=0.08486, simple_loss=0.1159, pruned_loss=0.02151, audio_tagging_loss=0.005421, over 14934.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09408, pruned_loss=0.01483, audio_tagging_loss=0.009087, over 3055039.82 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:41:29,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2109513.3333333335, ans=0.07 2023-11-22 21:41:32,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.79 vs. limit=15.0 2023-11-22 21:41:36,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2109580.0, ans=0.0 2023-11-22 21:41:40,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2109580.0, ans=0.125 2023-11-22 21:41:50,083 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.525e+01 8.963e+01 9.873e+01 1.311e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-22 21:41:50,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316450 2023-11-22 21:41:52,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2109646.6666666665, ans=0.1 2023-11-22 21:42:03,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2109713.3333333335, ans=0.0 2023-11-22 21:42:14,513 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3850, loss[loss=0.06431, simple_loss=0.08577, pruned_loss=0.01079, audio_tagging_loss=0.01064, over 15255.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09422, pruned_loss=0.01472, audio_tagging_loss=0.009118, over 3050197.35 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:42:24,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2109780.0, ans=0.125 2023-11-22 21:42:31,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2109846.6666666665, ans=0.07 2023-11-22 21:42:32,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.32 vs. limit=15.0 2023-11-22 21:42:41,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.59 vs. limit=6.0 2023-11-22 21:42:44,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2109913.3333333335, ans=0.0 2023-11-22 21:42:44,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2109913.3333333335, ans=0.025 2023-11-22 21:42:49,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2109913.3333333335, ans=0.0 2023-11-22 21:42:53,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316500 2023-11-22 21:42:59,859 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.45 vs. limit=15.0 2023-11-22 21:43:04,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2110046.6666666665, ans=0.0 2023-11-22 21:43:05,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2110046.6666666665, ans=0.0 2023-11-22 21:43:17,560 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3900, loss[loss=0.05852, simple_loss=0.06105, pruned_loss=0.01254, audio_tagging_loss=0.01545, over 14670.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09353, pruned_loss=0.01461, audio_tagging_loss=0.009264, over 3042976.95 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:43:21,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2110113.3333333335, ans=0.5 2023-11-22 21:43:30,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2110180.0, ans=0.125 2023-11-22 21:43:39,127 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.48 vs. limit=22.5 2023-11-22 21:43:49,305 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.89 vs. limit=15.0 2023-11-22 21:43:50,587 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.32 vs. limit=15.0 2023-11-22 21:43:57,193 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.364e+01 8.359e+01 8.893e+01 9.456e+01 1.423e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-22 21:43:57,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316550 2023-11-22 21:44:13,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2110380.0, ans=0.1 2023-11-22 21:44:17,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2110380.0, ans=0.125 2023-11-22 21:44:18,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2110380.0, ans=0.125 2023-11-22 21:44:21,478 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 3950, loss[loss=0.06988, simple_loss=0.1045, pruned_loss=0.01039, audio_tagging_loss=0.007258, over 15414.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09262, pruned_loss=0.01449, audio_tagging_loss=0.009344, over 3042357.41 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:44:31,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2110446.6666666665, ans=0.125 2023-11-22 21:44:31,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2110446.6666666665, ans=0.125 2023-11-22 21:44:35,095 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.30 vs. limit=22.5 2023-11-22 21:44:35,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2110513.3333333335, ans=0.1 2023-11-22 21:44:39,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2110513.3333333335, ans=0.125 2023-11-22 21:44:48,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2110580.0, ans=0.2 2023-11-22 21:45:00,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316600 2023-11-22 21:45:24,992 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.84 vs. limit=10.0 2023-11-22 21:45:25,353 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4000, loss[loss=0.08059, simple_loss=0.1058, pruned_loss=0.01897, audio_tagging_loss=0.008703, over 14713.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09335, pruned_loss=0.01464, audio_tagging_loss=0.009366, over 3047817.47 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:45:35,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2110780.0, ans=0.125 2023-11-22 21:45:43,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2110846.6666666665, ans=0.1 2023-11-22 21:46:04,310 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.309e+01 8.921e+01 9.654e+01 1.157e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-22 21:46:04,452 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316650 2023-11-22 21:46:20,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2111046.6666666665, ans=0.125 2023-11-22 21:46:22,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2111046.6666666665, ans=0.125 2023-11-22 21:46:25,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2111046.6666666665, ans=0.125 2023-11-22 21:46:28,460 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4050, loss[loss=0.09057, simple_loss=0.1282, pruned_loss=0.02026, audio_tagging_loss=0.006201, over 15567.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09413, pruned_loss=0.01478, audio_tagging_loss=0.009388, over 3049003.38 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:46:32,056 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:47:07,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316700 2023-11-22 21:47:09,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2111313.3333333335, ans=0.125 2023-11-22 21:47:25,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2111380.0, ans=0.125 2023-11-22 21:47:25,658 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.26 vs. limit=15.0 2023-11-22 21:47:31,745 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4100, loss[loss=0.0769, simple_loss=0.1068, pruned_loss=0.01494, audio_tagging_loss=0.008555, over 15832.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.0942, pruned_loss=0.01474, audio_tagging_loss=0.009406, over 3049613.13 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:47:45,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.76 vs. limit=6.0 2023-11-22 21:47:52,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2111513.3333333335, ans=0.2 2023-11-22 21:48:10,811 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.312e+01 9.101e+01 1.011e+02 1.374e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-22 21:48:10,990 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316750 2023-11-22 21:48:12,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2111646.6666666665, ans=0.0 2023-11-22 21:48:27,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2111713.3333333335, ans=0.0 2023-11-22 21:48:36,907 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4150, loss[loss=0.07963, simple_loss=0.1136, pruned_loss=0.01738, audio_tagging_loss=0.005428, over 15643.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09332, pruned_loss=0.01463, audio_tagging_loss=0.009336, over 3041844.08 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:48:51,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2111846.6666666665, ans=0.0 2023-11-22 21:48:52,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.96 vs. limit=10.0 2023-11-22 21:48:55,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2111846.6666666665, ans=0.125 2023-11-22 21:49:10,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2111913.3333333335, ans=0.125 2023-11-22 21:49:14,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2111980.0, ans=0.125 2023-11-22 21:49:16,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316800 2023-11-22 21:49:17,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.96 vs. limit=10.0 2023-11-22 21:49:23,124 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:49:31,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2112046.6666666665, ans=0.125 2023-11-22 21:49:32,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2112046.6666666665, ans=0.0 2023-11-22 21:49:40,876 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4200, loss[loss=0.0757, simple_loss=0.1048, pruned_loss=0.01638, audio_tagging_loss=0.006921, over 15479.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09333, pruned_loss=0.0145, audio_tagging_loss=0.009174, over 3045210.96 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:50:06,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2112246.6666666665, ans=0.2 2023-11-22 21:50:09,504 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.04 vs. limit=15.0 2023-11-22 21:50:10,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2112246.6666666665, ans=0.0 2023-11-22 21:50:17,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2112246.6666666665, ans=0.1 2023-11-22 21:50:20,404 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.143e+01 8.631e+01 9.392e+01 1.233e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-22 21:50:20,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316850 2023-11-22 21:50:24,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0 2023-11-22 21:50:41,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2112380.0, ans=0.125 2023-11-22 21:50:43,581 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4250, loss[loss=0.07038, simple_loss=0.09618, pruned_loss=0.01478, audio_tagging_loss=0.007501, over 15693.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09407, pruned_loss=0.01469, audio_tagging_loss=0.009137, over 3048507.63 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:50:45,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2112446.6666666665, ans=0.1 2023-11-22 21:51:05,436 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.80 vs. limit=15.0 2023-11-22 21:51:19,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.25 vs. limit=12.0 2023-11-22 21:51:22,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2112646.6666666665, ans=0.1 2023-11-22 21:51:23,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316900 2023-11-22 21:51:29,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2112646.6666666665, ans=0.125 2023-11-22 21:51:41,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2112713.3333333335, ans=0.0 2023-11-22 21:51:43,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2112713.3333333335, ans=0.125 2023-11-22 21:51:47,699 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4300, loss[loss=0.06378, simple_loss=0.07915, pruned_loss=0.01423, audio_tagging_loss=0.009983, over 14122.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09472, pruned_loss=0.01487, audio_tagging_loss=0.009108, over 3048696.69 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:52:25,866 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.51 vs. limit=15.0 2023-11-22 21:52:26,448 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.241e+01 8.941e+01 9.538e+01 1.182e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-22 21:52:26,608 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 316950 2023-11-22 21:52:46,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2113046.6666666665, ans=0.125 2023-11-22 21:52:46,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2113046.6666666665, ans=0.1 2023-11-22 21:52:48,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2113046.6666666665, ans=0.2 2023-11-22 21:52:51,092 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4350, loss[loss=0.0629, simple_loss=0.07616, pruned_loss=0.01359, audio_tagging_loss=0.01124, over 15218.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09463, pruned_loss=0.01508, audio_tagging_loss=0.009077, over 3045209.00 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:52:52,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2113113.3333333335, ans=0.125 2023-11-22 21:52:56,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2113113.3333333335, ans=0.05 2023-11-22 21:52:58,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2113113.3333333335, ans=0.1 2023-11-22 21:53:11,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.12 vs. limit=12.0 2023-11-22 21:53:28,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2113246.6666666665, ans=0.125 2023-11-22 21:53:31,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317000 2023-11-22 21:53:46,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2113380.0, ans=0.125 2023-11-22 21:53:55,194 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4400, loss[loss=0.07922, simple_loss=0.1096, pruned_loss=0.01698, audio_tagging_loss=0.007427, over 15728.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09464, pruned_loss=0.01496, audio_tagging_loss=0.009039, over 3040438.52 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:53:56,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2113446.6666666665, ans=0.0 2023-11-22 21:54:05,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2113446.6666666665, ans=0.0 2023-11-22 21:54:18,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.10 vs. limit=15.0 2023-11-22 21:54:34,935 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.698e+01 8.507e+01 9.216e+01 9.885e+01 1.286e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-22 21:54:35,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317050 2023-11-22 21:54:50,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2113713.3333333335, ans=0.0 2023-11-22 21:54:59,461 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4450, loss[loss=0.04514, simple_loss=0.05297, pruned_loss=0.00724, audio_tagging_loss=0.01141, over 14732.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09305, pruned_loss=0.01479, audio_tagging_loss=0.009122, over 3042813.74 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:55:08,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2113780.0, ans=0.09899494936611666 2023-11-22 21:55:17,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2113846.6666666665, ans=0.125 2023-11-22 21:55:39,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317100 2023-11-22 21:55:53,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.80 vs. limit=15.0 2023-11-22 21:55:56,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.65 vs. limit=15.0 2023-11-22 21:55:57,420 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.26 vs. limit=22.5 2023-11-22 21:56:04,073 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4500, loss[loss=0.05826, simple_loss=0.07176, pruned_loss=0.01189, audio_tagging_loss=0.01049, over 16201.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09279, pruned_loss=0.01462, audio_tagging_loss=0.009145, over 3047126.27 frames. ], batch size: 66, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:56:06,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2114113.3333333335, ans=0.125 2023-11-22 21:56:08,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2114113.3333333335, ans=0.125 2023-11-22 21:56:44,905 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.228e+01 8.995e+01 9.708e+01 1.219e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 21:56:45,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317150 2023-11-22 21:56:47,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2114313.3333333335, ans=0.0 2023-11-22 21:56:50,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0 2023-11-22 21:56:53,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.16 vs. limit=15.0 2023-11-22 21:57:08,294 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4550, loss[loss=0.04849, simple_loss=0.07004, pruned_loss=0.006907, audio_tagging_loss=0.006561, over 15299.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09155, pruned_loss=0.01438, audio_tagging_loss=0.009238, over 3044324.68 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 21:57:12,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.21 vs. limit=10.0 2023-11-22 21:57:28,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.42 vs. limit=15.0 2023-11-22 21:57:46,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2114646.6666666665, ans=0.125 2023-11-22 21:57:48,874 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317200 2023-11-22 21:57:50,674 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.42 vs. limit=15.0 2023-11-22 21:57:57,918 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 21:58:13,827 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4600, loss[loss=0.07627, simple_loss=0.1075, pruned_loss=0.01373, audio_tagging_loss=0.00881, over 15665.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09179, pruned_loss=0.01448, audio_tagging_loss=0.009325, over 3049020.80 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:58:15,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2114780.0, ans=0.0 2023-11-22 21:58:29,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2114846.6666666665, ans=0.0 2023-11-22 21:58:53,204 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317250 2023-11-22 21:58:54,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.032e+01 8.605e+01 9.322e+01 1.270e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-22 21:59:18,568 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4650, loss[loss=0.05109, simple_loss=0.05535, pruned_loss=0.01209, audio_tagging_loss=0.01134, over 15104.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.0921, pruned_loss=0.01451, audio_tagging_loss=0.009354, over 3050114.39 frames. ], batch size: 61, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 21:59:39,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.71 vs. limit=15.0 2023-11-22 21:59:55,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2115313.3333333335, ans=0.1 2023-11-22 21:59:57,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317300 2023-11-22 22:00:00,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2115313.3333333335, ans=0.125 2023-11-22 22:00:06,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2115313.3333333335, ans=0.0 2023-11-22 22:00:21,182 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4700, loss[loss=0.07314, simple_loss=0.09803, pruned_loss=0.01706, audio_tagging_loss=0.007062, over 14866.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09299, pruned_loss=0.01456, audio_tagging_loss=0.009356, over 3051844.24 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 22:00:42,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2115513.3333333335, ans=0.0 2023-11-22 22:00:45,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2115580.0, ans=0.125 2023-11-22 22:00:48,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2115580.0, ans=0.125 2023-11-22 22:00:51,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.04 vs. limit=15.0 2023-11-22 22:00:59,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2115646.6666666665, ans=0.125 2023-11-22 22:01:00,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317350 2023-11-22 22:01:01,992 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.187e+01 8.736e+01 9.527e+01 1.154e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 22:01:04,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2115646.6666666665, ans=0.2 2023-11-22 22:01:22,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2115713.3333333335, ans=0.2 2023-11-22 22:01:25,034 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4750, loss[loss=0.06089, simple_loss=0.08401, pruned_loss=0.01076, audio_tagging_loss=0.008119, over 15530.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09301, pruned_loss=0.01448, audio_tagging_loss=0.00942, over 3055334.29 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 22:01:42,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2115846.6666666665, ans=0.2 2023-11-22 22:02:03,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317400 2023-11-22 22:02:25,210 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.12 vs. limit=6.0 2023-11-22 22:02:29,390 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4800, loss[loss=0.06057, simple_loss=0.07718, pruned_loss=0.01167, audio_tagging_loss=0.0103, over 14893.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09301, pruned_loss=0.01442, audio_tagging_loss=0.009433, over 3052612.67 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 22:02:30,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2116113.3333333335, ans=0.07 2023-11-22 22:02:54,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2116246.6666666665, ans=0.125 2023-11-22 22:03:00,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2116246.6666666665, ans=0.1 2023-11-22 22:03:08,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.23 vs. limit=15.0 2023-11-22 22:03:09,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317450 2023-11-22 22:03:10,367 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.616e+01 8.423e+01 9.043e+01 9.752e+01 1.234e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-22 22:03:21,581 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.01 vs. limit=22.5 2023-11-22 22:03:22,496 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.70 vs. limit=15.0 2023-11-22 22:03:33,803 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4850, loss[loss=0.08342, simple_loss=0.1241, pruned_loss=0.01509, audio_tagging_loss=0.006287, over 16075.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09376, pruned_loss=0.01447, audio_tagging_loss=0.009471, over 3048399.60 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 32.0 2023-11-22 22:03:49,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2116513.3333333335, ans=0.015 2023-11-22 22:03:51,492 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.80 vs. limit=15.0 2023-11-22 22:04:05,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2116580.0, ans=0.2 2023-11-22 22:04:09,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.07 vs. limit=22.5 2023-11-22 22:04:12,050 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.58 vs. limit=15.0 2023-11-22 22:04:13,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317500 2023-11-22 22:04:15,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2116646.6666666665, ans=0.125 2023-11-22 22:04:26,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2116713.3333333335, ans=0.2 2023-11-22 22:04:26,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2116713.3333333335, ans=0.0 2023-11-22 22:04:38,103 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4900, loss[loss=0.05922, simple_loss=0.07866, pruned_loss=0.009961, audio_tagging_loss=0.00993, over 15353.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09399, pruned_loss=0.01457, audio_tagging_loss=0.009384, over 3044400.61 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 22:04:45,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2116780.0, ans=0.95 2023-11-22 22:04:50,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2116846.6666666665, ans=0.125 2023-11-22 22:05:14,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2116913.3333333335, ans=0.0 2023-11-22 22:05:18,294 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317550 2023-11-22 22:05:18,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2116980.0, ans=0.025 2023-11-22 22:05:20,646 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.184e+01 8.772e+01 9.580e+01 1.151e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-22 22:05:31,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2117046.6666666665, ans=0.1 2023-11-22 22:05:41,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.25 vs. limit=15.0 2023-11-22 22:05:42,998 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 4950, loss[loss=0.07048, simple_loss=0.08814, pruned_loss=0.01553, audio_tagging_loss=0.01089, over 14158.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09345, pruned_loss=0.01436, audio_tagging_loss=0.009205, over 3045181.15 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 22:05:51,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2117113.3333333335, ans=0.1 2023-11-22 22:05:57,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2117180.0, ans=0.0 2023-11-22 22:05:58,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2117180.0, ans=0.1 2023-11-22 22:06:22,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317600 2023-11-22 22:06:32,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2117313.3333333335, ans=0.0 2023-11-22 22:06:37,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.22 vs. limit=10.0 2023-11-22 22:06:46,920 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5000, loss[loss=0.06339, simple_loss=0.08972, pruned_loss=0.01169, audio_tagging_loss=0.006833, over 15505.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09274, pruned_loss=0.0142, audio_tagging_loss=0.00914, over 3041499.82 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 22:06:47,718 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.31 vs. limit=15.0 2023-11-22 22:07:13,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2117580.0, ans=0.125 2023-11-22 22:07:21,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.36 vs. limit=15.0 2023-11-22 22:07:26,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2117646.6666666665, ans=0.125 2023-11-22 22:07:27,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317650 2023-11-22 22:07:29,598 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.065e+01 8.747e+01 9.520e+01 1.397e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 22:07:29,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2117646.6666666665, ans=0.125 2023-11-22 22:07:32,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2117646.6666666665, ans=0.125 2023-11-22 22:07:38,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.81 vs. limit=15.0 2023-11-22 22:07:50,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.08 vs. limit=15.0 2023-11-22 22:07:51,020 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5050, loss[loss=0.04975, simple_loss=0.06705, pruned_loss=0.006508, audio_tagging_loss=0.009717, over 15692.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09378, pruned_loss=0.01446, audio_tagging_loss=0.009016, over 3044623.80 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 16.0 2023-11-22 22:08:24,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2117913.3333333335, ans=0.1 2023-11-22 22:08:31,694 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317700 2023-11-22 22:08:35,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2117980.0, ans=0.125 2023-11-22 22:08:44,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2118046.6666666665, ans=0.125 2023-11-22 22:08:51,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2118046.6666666665, ans=0.125 2023-11-22 22:08:53,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2118046.6666666665, ans=0.1 2023-11-22 22:08:56,521 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5100, loss[loss=0.0802, simple_loss=0.1047, pruned_loss=0.01864, audio_tagging_loss=0.009237, over 15308.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09392, pruned_loss=0.01456, audio_tagging_loss=0.00896, over 3044444.24 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:09:08,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.05 vs. limit=15.0 2023-11-22 22:09:11,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2118180.0, ans=0.0 2023-11-22 22:09:19,456 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0 2023-11-22 22:09:23,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2118246.6666666665, ans=0.125 2023-11-22 22:09:24,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2118246.6666666665, ans=0.0 2023-11-22 22:09:28,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2118246.6666666665, ans=0.1 2023-11-22 22:09:36,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317750 2023-11-22 22:09:36,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2118313.3333333335, ans=0.0 2023-11-22 22:09:36,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2118313.3333333335, ans=0.125 2023-11-22 22:09:38,769 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.041e+01 8.209e+01 8.781e+01 9.360e+01 1.201e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 22:09:45,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2118313.3333333335, ans=0.125 2023-11-22 22:09:54,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2118380.0, ans=0.95 2023-11-22 22:10:00,178 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5150, loss[loss=0.07791, simple_loss=0.09795, pruned_loss=0.01945, audio_tagging_loss=0.009495, over 14851.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09424, pruned_loss=0.01466, audio_tagging_loss=0.008975, over 3045741.68 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:10:00,996 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.41 vs. limit=10.0 2023-11-22 22:10:40,788 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317800 2023-11-22 22:10:44,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2118646.6666666665, ans=0.1 2023-11-22 22:10:57,417 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:11:04,852 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5200, loss[loss=0.07855, simple_loss=0.1016, pruned_loss=0.01875, audio_tagging_loss=0.009021, over 15364.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09385, pruned_loss=0.01477, audio_tagging_loss=0.008998, over 3036968.33 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:11:06,604 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0 2023-11-22 22:11:11,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2118780.0, ans=0.125 2023-11-22 22:11:26,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2118846.6666666665, ans=0.125 2023-11-22 22:11:28,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2118846.6666666665, ans=0.125 2023-11-22 22:11:44,370 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317850 2023-11-22 22:11:46,663 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.652e+01 9.262e+01 9.962e+01 1.336e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-22 22:12:03,933 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.66 vs. limit=22.5 2023-11-22 22:12:06,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2119046.6666666665, ans=0.125 2023-11-22 22:12:09,251 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5250, loss[loss=0.06975, simple_loss=0.09208, pruned_loss=0.01321, audio_tagging_loss=0.0105, over 15238.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09429, pruned_loss=0.01484, audio_tagging_loss=0.00895, over 3044854.73 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:12:17,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2119113.3333333335, ans=0.2 2023-11-22 22:12:18,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2119113.3333333335, ans=0.2 2023-11-22 22:12:36,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2119246.6666666665, ans=0.125 2023-11-22 22:12:48,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317900 2023-11-22 22:12:57,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2119313.3333333335, ans=0.0 2023-11-22 22:13:12,503 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5300, loss[loss=0.06866, simple_loss=0.09347, pruned_loss=0.01375, audio_tagging_loss=0.00818, over 15976.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09521, pruned_loss=0.01478, audio_tagging_loss=0.008922, over 3044643.10 frames. ], batch size: 61, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:13:25,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2119513.3333333335, ans=0.2 2023-11-22 22:13:53,075 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 317950 2023-11-22 22:13:56,549 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.353e+01 8.983e+01 9.482e+01 1.198e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 22:14:03,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2119713.3333333335, ans=0.1 2023-11-22 22:14:15,903 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.96 vs. limit=15.0 2023-11-22 22:14:16,245 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5350, loss[loss=0.07295, simple_loss=0.09871, pruned_loss=0.01501, audio_tagging_loss=0.008595, over 14954.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09468, pruned_loss=0.01483, audio_tagging_loss=0.008981, over 3035690.20 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:14:20,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2119780.0, ans=0.125 2023-11-22 22:14:40,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2119846.6666666665, ans=0.125 2023-11-22 22:14:56,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318000 2023-11-22 22:15:05,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2119980.0, ans=0.125 2023-11-22 22:15:10,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2120046.6666666665, ans=0.125 2023-11-22 22:15:20,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2120113.3333333335, ans=0.0 2023-11-22 22:15:21,774 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5400, loss[loss=0.05561, simple_loss=0.07675, pruned_loss=0.008284, audio_tagging_loss=0.008946, over 14506.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09365, pruned_loss=0.01449, audio_tagging_loss=0.009102, over 3038291.97 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:15:24,942 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0 2023-11-22 22:15:44,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.76 vs. limit=15.0 2023-11-22 22:15:48,435 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:15:51,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2120246.6666666665, ans=10.0 2023-11-22 22:16:00,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318050 2023-11-22 22:16:04,447 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.392e+01 8.997e+01 9.617e+01 1.276e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 22:16:25,801 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5450, loss[loss=0.09033, simple_loss=0.1142, pruned_loss=0.02264, audio_tagging_loss=0.0106, over 15942.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09388, pruned_loss=0.0146, audio_tagging_loss=0.0091, over 3039161.35 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:17:06,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318100 2023-11-22 22:17:20,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2120713.3333333335, ans=0.0 2023-11-22 22:17:29,796 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5500, loss[loss=0.07698, simple_loss=0.09845, pruned_loss=0.01546, audio_tagging_loss=0.01229, over 15837.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09391, pruned_loss=0.01476, audio_tagging_loss=0.009114, over 3041499.60 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:17:36,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2120780.0, ans=0.0 2023-11-22 22:17:49,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2120846.6666666665, ans=0.0 2023-11-22 22:18:00,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2120913.3333333335, ans=0.1 2023-11-22 22:18:03,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2120913.3333333335, ans=0.1 2023-11-22 22:18:09,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318150 2023-11-22 22:18:13,411 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.175e+01 8.992e+01 9.514e+01 1.142e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-22 22:18:20,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2121046.6666666665, ans=0.125 2023-11-22 22:18:33,982 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5550, loss[loss=0.06669, simple_loss=0.09388, pruned_loss=0.01077, audio_tagging_loss=0.008982, over 15167.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09381, pruned_loss=0.01482, audio_tagging_loss=0.009296, over 3043197.51 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:18:42,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2121113.3333333335, ans=0.125 2023-11-22 22:18:49,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2121180.0, ans=0.125 2023-11-22 22:18:50,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2121180.0, ans=0.125 2023-11-22 22:19:13,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318200 2023-11-22 22:19:39,035 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5600, loss[loss=0.05732, simple_loss=0.07948, pruned_loss=0.00894, audio_tagging_loss=0.008644, over 15628.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09351, pruned_loss=0.01466, audio_tagging_loss=0.009389, over 3048121.24 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:19:48,264 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.82 vs. limit=10.0 2023-11-22 22:20:18,204 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318250 2023-11-22 22:20:22,781 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.364e+01 9.057e+01 9.975e+01 1.357e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-22 22:20:25,267 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 22:20:31,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2121713.3333333335, ans=0.125 2023-11-22 22:20:39,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2121713.3333333335, ans=0.1 2023-11-22 22:20:42,427 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5650, loss[loss=0.07775, simple_loss=0.1024, pruned_loss=0.01729, audio_tagging_loss=0.009248, over 14149.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09293, pruned_loss=0.01461, audio_tagging_loss=0.00953, over 3043540.46 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:20:47,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2121780.0, ans=0.1 2023-11-22 22:20:47,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2121780.0, ans=0.125 2023-11-22 22:20:52,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2121780.0, ans=0.125 2023-11-22 22:21:10,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2121913.3333333335, ans=0.2 2023-11-22 22:21:11,801 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.30 vs. limit=22.5 2023-11-22 22:21:22,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318300 2023-11-22 22:21:25,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2121980.0, ans=0.125 2023-11-22 22:21:26,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2121980.0, ans=0.125 2023-11-22 22:21:29,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2121980.0, ans=0.125 2023-11-22 22:21:46,056 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5700, loss[loss=0.06878, simple_loss=0.08363, pruned_loss=0.01685, audio_tagging_loss=0.01011, over 14930.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.0919, pruned_loss=0.01451, audio_tagging_loss=0.009556, over 3037707.08 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:21:49,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2122113.3333333335, ans=0.125 2023-11-22 22:22:10,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2122180.0, ans=0.0 2023-11-22 22:22:25,779 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318350 2023-11-22 22:22:27,389 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.12 vs. limit=15.0 2023-11-22 22:22:29,263 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.246e+01 8.801e+01 9.549e+01 1.194e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-22 22:22:34,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2122313.3333333335, ans=0.125 2023-11-22 22:22:37,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2122380.0, ans=0.2 2023-11-22 22:22:48,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2122380.0, ans=0.125 2023-11-22 22:22:50,406 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5750, loss[loss=0.06835, simple_loss=0.08346, pruned_loss=0.01305, audio_tagging_loss=0.01356, over 15768.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09264, pruned_loss=0.01476, audio_tagging_loss=0.009416, over 3041196.85 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:23:03,455 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:23:15,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2122580.0, ans=0.0 2023-11-22 22:23:29,911 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318400 2023-11-22 22:23:30,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2122646.6666666665, ans=0.0 2023-11-22 22:23:40,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2122713.3333333335, ans=10.0 2023-11-22 22:23:52,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2122713.3333333335, ans=0.2 2023-11-22 22:23:54,257 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5800, loss[loss=0.0823, simple_loss=0.1104, pruned_loss=0.01758, audio_tagging_loss=0.009536, over 16102.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09261, pruned_loss=0.01479, audio_tagging_loss=0.009345, over 3040513.18 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:23:59,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2122780.0, ans=0.2 2023-11-22 22:24:04,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2122780.0, ans=0.0 2023-11-22 22:24:06,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2122846.6666666665, ans=0.125 2023-11-22 22:24:22,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2122913.3333333335, ans=0.0 2023-11-22 22:24:28,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2122913.3333333335, ans=0.05 2023-11-22 22:24:34,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318450 2023-11-22 22:24:35,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2122980.0, ans=0.125 2023-11-22 22:24:38,377 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.593e+01 8.198e+01 8.735e+01 9.439e+01 1.113e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-22 22:24:41,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2122980.0, ans=0.125 2023-11-22 22:24:58,481 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5850, loss[loss=0.07941, simple_loss=0.1091, pruned_loss=0.01782, audio_tagging_loss=0.007057, over 14790.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09251, pruned_loss=0.01459, audio_tagging_loss=0.009197, over 3035594.06 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:25:15,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2123180.0, ans=0.1 2023-11-22 22:25:38,287 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318500 2023-11-22 22:25:42,374 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.92 vs. limit=6.0 2023-11-22 22:25:43,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2123313.3333333335, ans=0.0 2023-11-22 22:25:54,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2123380.0, ans=0.2 2023-11-22 22:26:03,117 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5900, loss[loss=0.0608, simple_loss=0.07285, pruned_loss=0.01358, audio_tagging_loss=0.0108, over 14140.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09291, pruned_loss=0.01467, audio_tagging_loss=0.009166, over 3030119.75 frames. ], batch size: 52, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 22:26:29,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2123580.0, ans=0.125 2023-11-22 22:26:29,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2123580.0, ans=0.125 2023-11-22 22:26:41,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2123646.6666666665, ans=0.125 2023-11-22 22:26:41,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2123646.6666666665, ans=0.125 2023-11-22 22:26:42,763 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318550 2023-11-22 22:26:48,714 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.253e+01 8.842e+01 9.499e+01 1.797e+02, threshold=1.768e+02, percent-clipped=1.0 2023-11-22 22:27:00,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2123713.3333333335, ans=0.125 2023-11-22 22:27:06,800 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 5950, loss[loss=0.0555, simple_loss=0.07234, pruned_loss=0.00778, audio_tagging_loss=0.01155, over 15179.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09345, pruned_loss=0.01476, audio_tagging_loss=0.009081, over 3044404.59 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 22:27:20,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2123846.6666666665, ans=0.125 2023-11-22 22:27:46,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318600 2023-11-22 22:28:10,925 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6000, loss[loss=0.06378, simple_loss=0.08621, pruned_loss=0.01176, audio_tagging_loss=0.008918, over 14542.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09301, pruned_loss=0.01468, audio_tagging_loss=0.009142, over 3042449.74 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:28:10,926 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 22:28:54,207 INFO [train_asr.py:1253] (3/4) Epoch 27, validation: loss=0.05853, simple_loss=0.05134, pruned_loss=0.005103, audio_tagging_loss=0.02775, over 4681554.00 frames. 2023-11-22 22:28:54,208 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 22:29:23,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2124246.6666666665, ans=0.125 2023-11-22 22:29:34,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318650 2023-11-22 22:29:40,609 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.043e+01 8.303e+01 8.890e+01 9.466e+01 1.522e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-22 22:29:41,913 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 22:29:52,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2124380.0, ans=0.5 2023-11-22 22:29:57,684 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6050, loss[loss=0.05938, simple_loss=0.07319, pruned_loss=0.01419, audio_tagging_loss=0.008599, over 14231.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09267, pruned_loss=0.01459, audio_tagging_loss=0.009128, over 3041378.42 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:30:38,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318700 2023-11-22 22:31:02,037 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6100, loss[loss=0.0819, simple_loss=0.1127, pruned_loss=0.01843, audio_tagging_loss=0.007099, over 14461.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09277, pruned_loss=0.01435, audio_tagging_loss=0.00912, over 3038593.03 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:31:18,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2124846.6666666665, ans=0.125 2023-11-22 22:31:22,449 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.22 vs. limit=15.0 2023-11-22 22:31:23,250 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:31:29,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2124913.3333333335, ans=0.0 2023-11-22 22:31:41,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318750 2023-11-22 22:31:48,028 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.696e+01 8.282e+01 8.867e+01 9.593e+01 1.255e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-22 22:31:58,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2125046.6666666665, ans=0.125 2023-11-22 22:32:06,867 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6150, loss[loss=0.07892, simple_loss=0.1037, pruned_loss=0.01777, audio_tagging_loss=0.009306, over 14915.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09278, pruned_loss=0.0143, audio_tagging_loss=0.009163, over 3045437.14 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:32:07,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2125113.3333333335, ans=0.125 2023-11-22 22:32:30,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2125246.6666666665, ans=0.1 2023-11-22 22:32:47,402 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318800 2023-11-22 22:33:11,343 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6200, loss[loss=0.08541, simple_loss=0.1177, pruned_loss=0.01906, audio_tagging_loss=0.00749, over 14531.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09261, pruned_loss=0.01437, audio_tagging_loss=0.009247, over 3039908.91 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:33:14,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2125446.6666666665, ans=0.07 2023-11-22 22:33:40,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2125580.0, ans=0.1 2023-11-22 22:33:50,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2125646.6666666665, ans=0.0 2023-11-22 22:33:52,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318850 2023-11-22 22:33:57,957 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.282e+01 8.300e+01 8.948e+01 9.929e+01 3.016e+02, threshold=1.790e+02, percent-clipped=1.0 2023-11-22 22:34:16,003 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6250, loss[loss=0.08799, simple_loss=0.1221, pruned_loss=0.0203, audio_tagging_loss=0.006665, over 16078.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.0925, pruned_loss=0.01441, audio_tagging_loss=0.009331, over 3044510.77 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:34:28,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2125846.6666666665, ans=0.125 2023-11-22 22:34:43,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2125913.3333333335, ans=0.0 2023-11-22 22:34:54,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.99 vs. limit=15.0 2023-11-22 22:34:55,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318900 2023-11-22 22:35:06,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2126046.6666666665, ans=0.95 2023-11-22 22:35:20,688 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6300, loss[loss=0.07809, simple_loss=0.1075, pruned_loss=0.01679, audio_tagging_loss=0.007556, over 16908.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09326, pruned_loss=0.01465, audio_tagging_loss=0.009287, over 3051032.07 frames. ], batch size: 62, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:35:26,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2126113.3333333335, ans=0.09899494936611666 2023-11-22 22:35:36,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2126180.0, ans=0.125 2023-11-22 22:35:45,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2126246.6666666665, ans=0.2 2023-11-22 22:35:55,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2126246.6666666665, ans=0.0 2023-11-22 22:36:01,579 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 318950 2023-11-22 22:36:07,419 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.041e+01 8.820e+01 9.703e+01 1.205e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 22:36:10,092 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:36:13,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2126380.0, ans=0.125 2023-11-22 22:36:17,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.25 vs. limit=15.0 2023-11-22 22:36:25,210 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6350, loss[loss=0.07454, simple_loss=0.09646, pruned_loss=0.01577, audio_tagging_loss=0.01053, over 15981.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09261, pruned_loss=0.01457, audio_tagging_loss=0.009323, over 3049398.07 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:37:05,510 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319000 2023-11-22 22:37:29,132 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6400, loss[loss=0.0759, simple_loss=0.09991, pruned_loss=0.0157, audio_tagging_loss=0.01025, over 14630.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09257, pruned_loss=0.01458, audio_tagging_loss=0.009429, over 3045148.28 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:37:40,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2126780.0, ans=0.0 2023-11-22 22:37:52,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2126846.6666666665, ans=0.125 2023-11-22 22:38:08,541 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319050 2023-11-22 22:38:15,682 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.479e+01 8.884e+01 9.505e+01 1.366e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 22:38:32,765 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6450, loss[loss=0.06194, simple_loss=0.07574, pruned_loss=0.01444, audio_tagging_loss=0.009623, over 15225.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09105, pruned_loss=0.01437, audio_tagging_loss=0.009647, over 3043956.38 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:38:32,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2127113.3333333335, ans=0.125 2023-11-22 22:38:35,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2127113.3333333335, ans=0.125 2023-11-22 22:38:38,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2127113.3333333335, ans=0.0 2023-11-22 22:38:40,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2127113.3333333335, ans=0.125 2023-11-22 22:38:57,032 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.48 vs. limit=22.5 2023-11-22 22:39:03,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2127246.6666666665, ans=0.125 2023-11-22 22:39:11,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319100 2023-11-22 22:39:26,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2127380.0, ans=10.0 2023-11-22 22:39:30,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2127380.0, ans=0.125 2023-11-22 22:39:36,758 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6500, loss[loss=0.07802, simple_loss=0.1062, pruned_loss=0.01687, audio_tagging_loss=0.008045, over 15192.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09131, pruned_loss=0.0144, audio_tagging_loss=0.009622, over 3044956.48 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:39:59,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2127513.3333333335, ans=0.125 2023-11-22 22:40:08,112 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:40:17,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319150 2023-11-22 22:40:24,329 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.580e+01 8.470e+01 8.995e+01 9.586e+01 1.282e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-22 22:40:26,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2127713.3333333335, ans=0.2 2023-11-22 22:40:40,333 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6550, loss[loss=0.07006, simple_loss=0.0918, pruned_loss=0.01331, audio_tagging_loss=0.01084, over 16168.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09218, pruned_loss=0.0146, audio_tagging_loss=0.009484, over 3043564.53 frames. ], batch size: 61, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:40:40,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2127780.0, ans=0.1 2023-11-22 22:40:55,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2127846.6666666665, ans=0.2 2023-11-22 22:40:56,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2127846.6666666665, ans=0.1 2023-11-22 22:41:10,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2127913.3333333335, ans=0.0 2023-11-22 22:41:11,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0 2023-11-22 22:41:12,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2127913.3333333335, ans=0.0 2023-11-22 22:41:18,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2127980.0, ans=0.125 2023-11-22 22:41:21,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319200 2023-11-22 22:41:43,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2128046.6666666665, ans=0.025 2023-11-22 22:41:45,853 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6600, loss[loss=0.08699, simple_loss=0.1278, pruned_loss=0.01728, audio_tagging_loss=0.005813, over 15767.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09266, pruned_loss=0.01459, audio_tagging_loss=0.009329, over 3045048.24 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:41:46,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2128113.3333333335, ans=0.0 2023-11-22 22:41:59,853 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:42:13,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2128246.6666666665, ans=0.2 2023-11-22 22:42:16,223 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.47 vs. limit=15.0 2023-11-22 22:42:25,551 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319250 2023-11-22 22:42:27,396 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0 2023-11-22 22:42:33,337 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.441e+01 8.485e+01 9.023e+01 9.901e+01 1.531e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-22 22:42:41,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2128380.0, ans=0.0 2023-11-22 22:42:44,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2128380.0, ans=0.0 2023-11-22 22:42:49,978 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6650, loss[loss=0.0626, simple_loss=0.08024, pruned_loss=0.0118, audio_tagging_loss=0.01068, over 16018.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.0933, pruned_loss=0.01464, audio_tagging_loss=0.009221, over 3039512.83 frames. ], batch size: 62, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:43:19,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2128580.0, ans=0.125 2023-11-22 22:43:20,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2128580.0, ans=0.125 2023-11-22 22:43:21,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2128580.0, ans=0.1 2023-11-22 22:43:26,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2128646.6666666665, ans=0.2 2023-11-22 22:43:29,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319300 2023-11-22 22:43:38,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2128646.6666666665, ans=0.125 2023-11-22 22:43:53,683 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6700, loss[loss=0.06736, simple_loss=0.0844, pruned_loss=0.01276, audio_tagging_loss=0.01239, over 15353.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09363, pruned_loss=0.01455, audio_tagging_loss=0.009153, over 3043230.23 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:44:01,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.90 vs. limit=22.5 2023-11-22 22:44:20,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0 2023-11-22 22:44:34,183 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319350 2023-11-22 22:44:38,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0 2023-11-22 22:44:41,538 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.124e+01 8.630e+01 9.282e+01 1.213e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-22 22:44:50,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2129046.6666666665, ans=0.1 2023-11-22 22:44:58,815 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6750, loss[loss=0.06931, simple_loss=0.08566, pruned_loss=0.01412, audio_tagging_loss=0.01236, over 15191.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09332, pruned_loss=0.01457, audio_tagging_loss=0.009234, over 3048402.02 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 22:45:08,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2129113.3333333335, ans=0.125 2023-11-22 22:45:12,123 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0 2023-11-22 22:45:14,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2129180.0, ans=0.125 2023-11-22 22:45:15,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2129180.0, ans=0.0 2023-11-22 22:45:24,921 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0 2023-11-22 22:45:25,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2129246.6666666665, ans=0.2 2023-11-22 22:45:32,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2129246.6666666665, ans=0.125 2023-11-22 22:45:37,696 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319400 2023-11-22 22:45:59,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2129380.0, ans=0.05 2023-11-22 22:46:03,694 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6800, loss[loss=0.07574, simple_loss=0.09297, pruned_loss=0.017, audio_tagging_loss=0.01225, over 15347.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09296, pruned_loss=0.01461, audio_tagging_loss=0.009171, over 3050315.92 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:46:13,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2129446.6666666665, ans=0.0 2023-11-22 22:46:37,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2129580.0, ans=0.0 2023-11-22 22:46:43,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319450 2023-11-22 22:46:50,909 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.253e+01 9.028e+01 9.739e+01 1.351e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-22 22:47:07,612 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6850, loss[loss=0.06725, simple_loss=0.09016, pruned_loss=0.0147, audio_tagging_loss=0.007475, over 16486.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09277, pruned_loss=0.01451, audio_tagging_loss=0.009143, over 3041606.13 frames. ], batch size: 63, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:47:21,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2129846.6666666665, ans=0.125 2023-11-22 22:47:38,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2129913.3333333335, ans=0.0 2023-11-22 22:47:41,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2129913.3333333335, ans=0.0 2023-11-22 22:47:47,742 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319500 2023-11-22 22:48:11,924 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6900, loss[loss=0.09018, simple_loss=0.1271, pruned_loss=0.01738, audio_tagging_loss=0.00926, over 14800.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09232, pruned_loss=0.01433, audio_tagging_loss=0.009194, over 3043573.36 frames. ], batch size: 53, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:48:19,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2130113.3333333335, ans=0.125 2023-11-22 22:48:22,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2130113.3333333335, ans=0.125 2023-11-22 22:48:25,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2130180.0, ans=0.1 2023-11-22 22:48:48,858 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.75 vs. limit=12.0 2023-11-22 22:48:51,945 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319550 2023-11-22 22:48:59,090 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.191e+01 8.934e+01 9.580e+01 1.102e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-22 22:49:02,188 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 22:49:15,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.55 vs. limit=22.5 2023-11-22 22:49:15,981 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 6950, loss[loss=0.07378, simple_loss=0.1038, pruned_loss=0.01332, audio_tagging_loss=0.008542, over 15913.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09293, pruned_loss=0.01442, audio_tagging_loss=0.009215, over 3047909.33 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:49:36,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2130513.3333333335, ans=0.125 2023-11-22 22:49:55,211 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319600 2023-11-22 22:50:18,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2130780.0, ans=0.1 2023-11-22 22:50:19,649 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7000, loss[loss=0.06208, simple_loss=0.08704, pruned_loss=0.009229, audio_tagging_loss=0.009331, over 16518.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09221, pruned_loss=0.01427, audio_tagging_loss=0.009313, over 3043479.69 frames. ], batch size: 61, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:50:22,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2130780.0, ans=0.125 2023-11-22 22:50:26,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.93 vs. limit=10.0 2023-11-22 22:50:36,481 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.36 vs. limit=15.0 2023-11-22 22:50:39,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2130846.6666666665, ans=0.0 2023-11-22 22:50:58,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2130980.0, ans=0.125 2023-11-22 22:50:59,634 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319650 2023-11-22 22:51:07,652 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.087e+01 8.954e+01 9.728e+01 1.359e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-22 22:51:23,707 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7050, loss[loss=0.0855, simple_loss=0.1186, pruned_loss=0.01917, audio_tagging_loss=0.007019, over 16107.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09174, pruned_loss=0.01434, audio_tagging_loss=0.009301, over 3040323.08 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:51:28,685 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.57 vs. limit=15.0 2023-11-22 22:52:00,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2131246.6666666665, ans=0.2 2023-11-22 22:52:01,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2131313.3333333335, ans=0.0 2023-11-22 22:52:03,890 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319700 2023-11-22 22:52:17,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2131380.0, ans=0.1 2023-11-22 22:52:23,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2131380.0, ans=0.0 2023-11-22 22:52:28,766 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7100, loss[loss=0.06576, simple_loss=0.08414, pruned_loss=0.01409, audio_tagging_loss=0.009594, over 16294.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09238, pruned_loss=0.01427, audio_tagging_loss=0.009352, over 3043280.86 frames. ], batch size: 62, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:52:29,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2131446.6666666665, ans=0.0 2023-11-22 22:52:30,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2131446.6666666665, ans=0.2 2023-11-22 22:52:39,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.87 vs. limit=15.0 2023-11-22 22:52:47,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2131513.3333333335, ans=0.125 2023-11-22 22:53:07,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319750 2023-11-22 22:53:15,649 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 7.978e+01 8.785e+01 9.427e+01 1.365e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-22 22:53:24,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2131713.3333333335, ans=0.0 2023-11-22 22:53:28,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2131713.3333333335, ans=0.2 2023-11-22 22:53:31,766 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7150, loss[loss=0.07426, simple_loss=0.1001, pruned_loss=0.01726, audio_tagging_loss=0.006926, over 16750.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09333, pruned_loss=0.01426, audio_tagging_loss=0.009373, over 3045626.12 frames. ], batch size: 61, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:53:36,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.86 vs. limit=12.0 2023-11-22 22:53:44,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2131846.6666666665, ans=0.0 2023-11-22 22:53:56,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2131913.3333333335, ans=0.0 2023-11-22 22:54:02,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2131913.3333333335, ans=0.125 2023-11-22 22:54:09,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2131980.0, ans=0.1 2023-11-22 22:54:09,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2131980.0, ans=0.0 2023-11-22 22:54:11,786 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319800 2023-11-22 22:54:35,961 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7200, loss[loss=0.05884, simple_loss=0.06916, pruned_loss=0.0136, audio_tagging_loss=0.01065, over 13647.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09277, pruned_loss=0.01438, audio_tagging_loss=0.009432, over 3046453.45 frames. ], batch size: 52, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:54:42,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2132113.3333333335, ans=0.125 2023-11-22 22:55:13,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2132313.3333333335, ans=0.125 2023-11-22 22:55:15,835 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319850 2023-11-22 22:55:23,094 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.344e+01 8.207e+01 8.744e+01 9.394e+01 1.258e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-22 22:55:23,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2132313.3333333335, ans=0.0 2023-11-22 22:55:35,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2132380.0, ans=0.125 2023-11-22 22:55:39,841 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7250, loss[loss=0.07957, simple_loss=0.1039, pruned_loss=0.0154, audio_tagging_loss=0.01221, over 15333.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.0928, pruned_loss=0.0143, audio_tagging_loss=0.009461, over 3044767.75 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 32.0 2023-11-22 22:56:08,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2132580.0, ans=0.2 2023-11-22 22:56:18,900 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319900 2023-11-22 22:56:22,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2132646.6666666665, ans=0.0 2023-11-22 22:56:22,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2132646.6666666665, ans=0.125 2023-11-22 22:56:37,274 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-22 22:56:43,078 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7300, loss[loss=0.0657, simple_loss=0.08201, pruned_loss=0.01361, audio_tagging_loss=0.01108, over 15188.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09389, pruned_loss=0.01457, audio_tagging_loss=0.009355, over 3044542.72 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 22:56:44,968 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.63 vs. limit=15.0 2023-11-22 22:56:59,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.69 vs. limit=15.0 2023-11-22 22:57:04,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2132846.6666666665, ans=0.0 2023-11-22 22:57:11,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=15.0 2023-11-22 22:57:22,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 319950 2023-11-22 22:57:31,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2132980.0, ans=0.1 2023-11-22 22:57:32,779 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.257e+01 8.750e+01 9.620e+01 1.168e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-22 22:57:42,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2133046.6666666665, ans=0.125 2023-11-22 22:57:44,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2133046.6666666665, ans=0.125 2023-11-22 22:57:44,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2133046.6666666665, ans=0.125 2023-11-22 22:57:46,968 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7350, loss[loss=0.07404, simple_loss=0.1018, pruned_loss=0.0174, audio_tagging_loss=0.005733, over 15724.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09419, pruned_loss=0.01457, audio_tagging_loss=0.009031, over 3047127.79 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 22:57:58,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.87 vs. limit=22.5 2023-11-22 22:58:01,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2133180.0, ans=0.0 2023-11-22 22:58:02,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2133180.0, ans=0.0 2023-11-22 22:58:26,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320000 2023-11-22 22:58:34,805 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.07 vs. limit=22.5 2023-11-22 22:58:53,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2133446.6666666665, ans=0.125 2023-11-22 22:58:53,957 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7400, loss[loss=0.05071, simple_loss=0.06571, pruned_loss=0.007, audio_tagging_loss=0.01085, over 14856.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09394, pruned_loss=0.01448, audio_tagging_loss=0.008996, over 3045518.55 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 22:59:03,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2133446.6666666665, ans=0.04949747468305833 2023-11-22 22:59:08,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2133513.3333333335, ans=0.125 2023-11-22 22:59:14,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2133513.3333333335, ans=0.125 2023-11-22 22:59:33,171 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320050 2023-11-22 22:59:43,443 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.110e+01 8.793e+01 9.338e+01 3.637e+02, threshold=1.759e+02, percent-clipped=1.0 2023-11-22 22:59:54,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2133713.3333333335, ans=0.1 2023-11-22 22:59:57,621 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7450, loss[loss=0.05409, simple_loss=0.06035, pruned_loss=0.0108, audio_tagging_loss=0.01312, over 14189.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09344, pruned_loss=0.01446, audio_tagging_loss=0.009007, over 3045877.28 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 22:59:58,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.43 vs. limit=12.0 2023-11-22 23:00:20,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2133846.6666666665, ans=0.0 2023-11-22 23:00:38,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320100 2023-11-22 23:00:40,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2133980.0, ans=0.125 2023-11-22 23:00:47,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2134046.6666666665, ans=0.1 2023-11-22 23:00:56,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.94 vs. limit=12.0 2023-11-22 23:00:58,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2134046.6666666665, ans=0.125 2023-11-22 23:01:01,247 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7500, loss[loss=0.06741, simple_loss=0.09042, pruned_loss=0.01247, audio_tagging_loss=0.009731, over 14899.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.0943, pruned_loss=0.01466, audio_tagging_loss=0.008906, over 3046264.33 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 23:01:11,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.04 vs. limit=15.0 2023-11-22 23:01:22,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2134180.0, ans=0.09899494936611666 2023-11-22 23:01:41,520 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320150 2023-11-22 23:01:51,165 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.201e+01 8.778e+01 9.335e+01 1.224e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 23:02:05,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2134446.6666666665, ans=0.0 2023-11-22 23:02:05,948 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7550, loss[loss=0.07083, simple_loss=0.09508, pruned_loss=0.01393, audio_tagging_loss=0.009356, over 16607.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09447, pruned_loss=0.01471, audio_tagging_loss=0.008926, over 3051404.47 frames. ], batch size: 63, lr: 2.52e-03, grad_scale: 8.0 2023-11-22 23:02:14,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2134446.6666666665, ans=0.0 2023-11-22 23:02:16,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2134446.6666666665, ans=0.0 2023-11-22 23:02:28,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2134513.3333333335, ans=0.125 2023-11-22 23:02:45,360 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320200 2023-11-22 23:03:00,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2134713.3333333335, ans=0.0 2023-11-22 23:03:10,694 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7600, loss[loss=0.06409, simple_loss=0.08731, pruned_loss=0.0124, audio_tagging_loss=0.008035, over 15242.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09348, pruned_loss=0.0145, audio_tagging_loss=0.008985, over 3051417.60 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 16.0 2023-11-22 23:03:46,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2134913.3333333335, ans=0.0 2023-11-22 23:03:50,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320250 2023-11-22 23:03:57,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2134980.0, ans=0.1 2023-11-22 23:04:00,560 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.533e+01 8.089e+01 8.781e+01 9.511e+01 1.232e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-22 23:04:06,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2135046.6666666665, ans=0.125 2023-11-22 23:04:07,025 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:04:07,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2135046.6666666665, ans=0.0 2023-11-22 23:04:13,897 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7650, loss[loss=0.05968, simple_loss=0.07185, pruned_loss=0.0123, audio_tagging_loss=0.01145, over 13906.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09286, pruned_loss=0.0144, audio_tagging_loss=0.009078, over 3044573.85 frames. ], batch size: 54, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:04:22,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2135113.3333333335, ans=0.125 2023-11-22 23:04:26,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2135180.0, ans=0.0 2023-11-22 23:04:37,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2135180.0, ans=0.0 2023-11-22 23:04:41,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2135246.6666666665, ans=0.0 2023-11-22 23:04:53,714 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320300 2023-11-22 23:05:04,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2135380.0, ans=10.0 2023-11-22 23:05:04,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2135380.0, ans=0.125 2023-11-22 23:05:08,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2135380.0, ans=0.0 2023-11-22 23:05:18,699 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7700, loss[loss=0.07428, simple_loss=0.09539, pruned_loss=0.01524, audio_tagging_loss=0.01135, over 15124.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09208, pruned_loss=0.01418, audio_tagging_loss=0.009136, over 3038353.77 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:05:23,275 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=12.0 2023-11-22 23:05:35,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2135513.3333333335, ans=0.1 2023-11-22 23:05:37,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2135513.3333333335, ans=0.125 2023-11-22 23:05:50,186 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.32 vs. limit=15.0 2023-11-22 23:05:59,439 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320350 2023-11-22 23:06:02,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2135646.6666666665, ans=0.125 2023-11-22 23:06:10,432 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.326e+01 9.020e+01 9.879e+01 1.267e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-22 23:06:10,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2135713.3333333335, ans=0.2 2023-11-22 23:06:15,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2135713.3333333335, ans=0.125 2023-11-22 23:06:25,284 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7750, loss[loss=0.05269, simple_loss=0.06424, pruned_loss=0.007838, audio_tagging_loss=0.01274, over 14387.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09163, pruned_loss=0.01412, audio_tagging_loss=0.00931, over 3040067.96 frames. ], batch size: 54, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:06:29,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2135780.0, ans=0.0 2023-11-22 23:06:34,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2135780.0, ans=0.125 2023-11-22 23:06:37,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2135846.6666666665, ans=0.0 2023-11-22 23:06:43,459 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.98 vs. limit=6.0 2023-11-22 23:07:05,785 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320400 2023-11-22 23:07:07,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2135980.0, ans=0.125 2023-11-22 23:07:10,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=2135980.0, ans=0.5 2023-11-22 23:07:29,753 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7800, loss[loss=0.04577, simple_loss=0.05715, pruned_loss=0.007146, audio_tagging_loss=0.01005, over 15394.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09245, pruned_loss=0.0143, audio_tagging_loss=0.009251, over 3040915.66 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:07:30,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2136113.3333333335, ans=0.1 2023-11-22 23:07:33,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2136113.3333333335, ans=0.125 2023-11-22 23:07:56,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2136246.6666666665, ans=0.0 2023-11-22 23:08:01,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2136246.6666666665, ans=0.125 2023-11-22 23:08:02,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2136246.6666666665, ans=0.0 2023-11-22 23:08:05,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2136246.6666666665, ans=0.0 2023-11-22 23:08:10,037 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320450 2023-11-22 23:08:10,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2136313.3333333335, ans=0.07 2023-11-22 23:08:16,657 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.01 vs. limit=12.0 2023-11-22 23:08:19,798 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.253e+01 8.929e+01 9.795e+01 1.214e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 23:08:33,308 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7850, loss[loss=0.09453, simple_loss=0.1266, pruned_loss=0.02322, audio_tagging_loss=0.007987, over 15485.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.093, pruned_loss=0.01442, audio_tagging_loss=0.009295, over 3046789.73 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:08:38,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2136446.6666666665, ans=0.125 2023-11-22 23:08:58,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.63 vs. limit=15.0 2023-11-22 23:09:02,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2136580.0, ans=0.0 2023-11-22 23:09:14,185 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320500 2023-11-22 23:09:34,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2136713.3333333335, ans=0.1 2023-11-22 23:09:34,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2136713.3333333335, ans=0.125 2023-11-22 23:09:39,736 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7900, loss[loss=0.06425, simple_loss=0.09019, pruned_loss=0.009481, audio_tagging_loss=0.009674, over 15625.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09335, pruned_loss=0.01462, audio_tagging_loss=0.009371, over 3048658.81 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:10:19,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320550 2023-11-22 23:10:19,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2136980.0, ans=0.125 2023-11-22 23:10:20,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2136980.0, ans=0.2 2023-11-22 23:10:29,991 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.096e+01 8.900e+01 9.590e+01 1.237e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-22 23:10:32,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2137046.6666666665, ans=0.0 2023-11-22 23:10:43,638 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 7950, loss[loss=0.07993, simple_loss=0.1119, pruned_loss=0.01522, audio_tagging_loss=0.008736, over 16390.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09241, pruned_loss=0.01453, audio_tagging_loss=0.009447, over 3048482.79 frames. ], batch size: 60, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:10:47,662 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.20 vs. limit=22.5 2023-11-22 23:10:53,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.02 vs. limit=22.5 2023-11-22 23:10:58,097 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 23:11:17,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2137246.6666666665, ans=0.125 2023-11-22 23:11:18,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.12 vs. limit=15.0 2023-11-22 23:11:24,021 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320600 2023-11-22 23:11:32,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.87 vs. limit=15.0 2023-11-22 23:11:47,660 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8000, loss[loss=0.0601, simple_loss=0.07175, pruned_loss=0.01059, audio_tagging_loss=0.01364, over 15822.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09182, pruned_loss=0.01427, audio_tagging_loss=0.009547, over 3054525.72 frames. ], batch size: 64, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:11:50,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2137446.6666666665, ans=0.1 2023-11-22 23:11:50,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2137446.6666666665, ans=0.2 2023-11-22 23:11:53,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2137446.6666666665, ans=0.125 2023-11-22 23:12:00,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.98 vs. limit=12.0 2023-11-22 23:12:07,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2137513.3333333335, ans=0.125 2023-11-22 23:12:16,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2137580.0, ans=0.0 2023-11-22 23:12:27,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320650 2023-11-22 23:12:37,620 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.271e+01 8.710e+01 9.655e+01 1.155e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-22 23:12:52,949 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8050, loss[loss=0.0767, simple_loss=0.1006, pruned_loss=0.01725, audio_tagging_loss=0.009159, over 15149.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09202, pruned_loss=0.01436, audio_tagging_loss=0.009575, over 3048992.86 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:13:08,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2137846.6666666665, ans=0.125 2023-11-22 23:13:13,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2137846.6666666665, ans=0.125 2023-11-22 23:13:25,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2137913.3333333335, ans=0.1 2023-11-22 23:13:32,223 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320700 2023-11-22 23:13:37,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2137980.0, ans=0.125 2023-11-22 23:13:51,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2138046.6666666665, ans=0.0 2023-11-22 23:13:57,271 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8100, loss[loss=0.06162, simple_loss=0.08633, pruned_loss=0.01166, audio_tagging_loss=0.0068, over 14953.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09096, pruned_loss=0.01412, audio_tagging_loss=0.009541, over 3046619.21 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:14:25,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2138246.6666666665, ans=0.125 2023-11-22 23:14:37,241 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320750 2023-11-22 23:14:47,691 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.347e+01 8.289e+01 8.980e+01 9.464e+01 1.714e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-22 23:14:50,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2138380.0, ans=0.0 2023-11-22 23:14:54,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2138380.0, ans=0.125 2023-11-22 23:15:01,307 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8150, loss[loss=0.07057, simple_loss=0.09142, pruned_loss=0.01622, audio_tagging_loss=0.008647, over 15441.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09279, pruned_loss=0.01462, audio_tagging_loss=0.009296, over 3049992.54 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:15:05,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2138446.6666666665, ans=0.125 2023-11-22 23:15:22,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2138513.3333333335, ans=0.1 2023-11-22 23:15:41,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320800 2023-11-22 23:15:44,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2138646.6666666665, ans=0.0 2023-11-22 23:15:55,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2138713.3333333335, ans=0.0 2023-11-22 23:16:03,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2138713.3333333335, ans=0.125 2023-11-22 23:16:05,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2138780.0, ans=0.125 2023-11-22 23:16:06,078 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8200, loss[loss=0.0711, simple_loss=0.09645, pruned_loss=0.01216, audio_tagging_loss=0.01072, over 15580.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09268, pruned_loss=0.01454, audio_tagging_loss=0.009226, over 3057289.76 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:16:06,135 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 23:16:20,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2138846.6666666665, ans=0.2 2023-11-22 23:16:22,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2138846.6666666665, ans=15.0 2023-11-22 23:16:26,606 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.34 vs. limit=8.0 2023-11-22 23:16:31,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.86 vs. limit=15.0 2023-11-22 23:16:34,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2138913.3333333335, ans=0.1 2023-11-22 23:16:46,243 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320850 2023-11-22 23:16:56,805 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.281e+01 9.046e+01 9.469e+01 1.170e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-22 23:17:11,161 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8250, loss[loss=0.06856, simple_loss=0.08774, pruned_loss=0.01303, audio_tagging_loss=0.01166, over 15881.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09266, pruned_loss=0.01463, audio_tagging_loss=0.009243, over 3059106.79 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:17:11,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2139113.3333333335, ans=0.0 2023-11-22 23:17:15,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2139113.3333333335, ans=0.125 2023-11-22 23:17:29,450 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.55 vs. limit=22.5 2023-11-22 23:17:35,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2139246.6666666665, ans=0.125 2023-11-22 23:17:49,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2139313.3333333335, ans=0.125 2023-11-22 23:17:51,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320900 2023-11-22 23:17:54,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2139313.3333333335, ans=0.1 2023-11-22 23:17:58,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2139313.3333333335, ans=0.125 2023-11-22 23:18:05,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2139380.0, ans=0.0 2023-11-22 23:18:06,622 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.22 vs. limit=15.0 2023-11-22 23:18:15,851 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8300, loss[loss=0.06607, simple_loss=0.09223, pruned_loss=0.01281, audio_tagging_loss=0.007135, over 15628.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09367, pruned_loss=0.0148, audio_tagging_loss=0.009225, over 3061393.59 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:18:26,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2139446.6666666665, ans=0.125 2023-11-22 23:18:39,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2139513.3333333335, ans=0.0 2023-11-22 23:18:56,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 320950 2023-11-22 23:18:57,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2139646.6666666665, ans=0.09899494936611666 2023-11-22 23:19:05,949 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.400e+01 9.007e+01 9.767e+01 1.180e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-22 23:19:14,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2139713.3333333335, ans=0.1 2023-11-22 23:19:17,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.75 vs. limit=15.0 2023-11-22 23:19:20,889 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8350, loss[loss=0.05575, simple_loss=0.06598, pruned_loss=0.01074, audio_tagging_loss=0.01202, over 15514.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09256, pruned_loss=0.01448, audio_tagging_loss=0.009246, over 3060314.67 frames. ], batch size: 60, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:19:43,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2139846.6666666665, ans=0.0 2023-11-22 23:19:59,971 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321000 2023-11-22 23:20:03,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2139980.0, ans=0.125 2023-11-22 23:20:20,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2140046.6666666665, ans=0.125 2023-11-22 23:20:24,977 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8400, loss[loss=0.0481, simple_loss=0.0598, pruned_loss=0.00921, audio_tagging_loss=0.008988, over 14884.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09243, pruned_loss=0.01445, audio_tagging_loss=0.009243, over 3054236.51 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:20:26,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.31 vs. limit=15.0 2023-11-22 23:20:47,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2140180.0, ans=0.2 2023-11-22 23:20:57,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2140246.6666666665, ans=0.0 2023-11-22 23:21:06,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321050 2023-11-22 23:21:12,233 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.78 vs. limit=22.5 2023-11-22 23:21:17,702 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.024e+01 8.884e+01 9.515e+01 1.445e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-22 23:21:18,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.94 vs. limit=15.0 2023-11-22 23:21:30,762 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8450, loss[loss=0.1039, simple_loss=0.1444, pruned_loss=0.02457, audio_tagging_loss=0.007091, over 15658.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09331, pruned_loss=0.01449, audio_tagging_loss=0.009263, over 3058639.56 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:21:33,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2140446.6666666665, ans=0.04949747468305833 2023-11-22 23:21:49,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2140513.3333333335, ans=0.0 2023-11-22 23:21:50,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2140513.3333333335, ans=0.125 2023-11-22 23:22:00,533 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.78 vs. limit=15.0 2023-11-22 23:22:11,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321100 2023-11-22 23:22:29,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2140713.3333333335, ans=0.0 2023-11-22 23:22:32,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.16 vs. limit=15.0 2023-11-22 23:22:36,197 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8500, loss[loss=0.06777, simple_loss=0.08885, pruned_loss=0.01211, audio_tagging_loss=0.01123, over 16989.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09485, pruned_loss=0.01483, audio_tagging_loss=0.009176, over 3063912.93 frames. ], batch size: 63, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:22:41,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.59 vs. limit=6.0 2023-11-22 23:22:55,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2140846.6666666665, ans=0.125 2023-11-22 23:23:06,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.37 vs. limit=15.0 2023-11-22 23:23:07,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2140913.3333333335, ans=0.125 2023-11-22 23:23:13,429 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.87 vs. limit=22.5 2023-11-22 23:23:15,186 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321150 2023-11-22 23:23:27,518 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.243e+01 8.956e+01 9.576e+01 1.335e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-22 23:23:40,535 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8550, loss[loss=0.08796, simple_loss=0.1187, pruned_loss=0.0208, audio_tagging_loss=0.007795, over 14359.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09452, pruned_loss=0.01469, audio_tagging_loss=0.009208, over 3061079.73 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:23:58,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2141180.0, ans=0.1 2023-11-22 23:24:05,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2141246.6666666665, ans=0.1 2023-11-22 23:24:11,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2141246.6666666665, ans=0.2 2023-11-22 23:24:20,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321200 2023-11-22 23:24:25,566 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.22 vs. limit=10.0 2023-11-22 23:24:44,466 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8600, loss[loss=0.08241, simple_loss=0.1045, pruned_loss=0.01831, audio_tagging_loss=0.01184, over 15210.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09543, pruned_loss=0.01491, audio_tagging_loss=0.009247, over 3063349.26 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:25:06,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2141513.3333333335, ans=0.2 2023-11-22 23:25:15,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2141580.0, ans=0.0 2023-11-22 23:25:23,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2141646.6666666665, ans=0.125 2023-11-22 23:25:25,101 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321250 2023-11-22 23:25:25,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.88 vs. limit=15.0 2023-11-22 23:25:26,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2141646.6666666665, ans=0.0 2023-11-22 23:25:33,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2141646.6666666665, ans=0.0 2023-11-22 23:25:36,062 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.429e+01 8.395e+01 8.985e+01 9.646e+01 1.201e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-22 23:25:43,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2141713.3333333335, ans=0.2 2023-11-22 23:25:46,961 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.48 vs. limit=15.0 2023-11-22 23:25:49,422 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8650, loss[loss=0.08707, simple_loss=0.1204, pruned_loss=0.01852, audio_tagging_loss=0.00835, over 15604.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09529, pruned_loss=0.0149, audio_tagging_loss=0.009223, over 3060643.67 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:26:28,859 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321300 2023-11-22 23:26:50,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2142046.6666666665, ans=0.0 2023-11-22 23:26:50,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2142046.6666666665, ans=0.125 2023-11-22 23:26:54,815 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8700, loss[loss=0.0707, simple_loss=0.09159, pruned_loss=0.01503, audio_tagging_loss=0.009882, over 15530.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09456, pruned_loss=0.01492, audio_tagging_loss=0.009296, over 3060002.30 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:26:57,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2142113.3333333335, ans=0.125 2023-11-22 23:27:02,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2142113.3333333335, ans=0.0 2023-11-22 23:27:09,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2142180.0, ans=0.125 2023-11-22 23:27:34,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321350 2023-11-22 23:27:38,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2142313.3333333335, ans=0.025 2023-11-22 23:27:45,845 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 8.323e+01 8.903e+01 9.633e+01 3.250e+02, threshold=1.781e+02, percent-clipped=1.0 2023-11-22 23:27:58,129 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8750, loss[loss=0.08502, simple_loss=0.1107, pruned_loss=0.01964, audio_tagging_loss=0.01003, over 14870.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09439, pruned_loss=0.01486, audio_tagging_loss=0.00942, over 3052779.35 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:28:22,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2142513.3333333335, ans=0.2 2023-11-22 23:28:35,894 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.17 vs. limit=10.0 2023-11-22 23:28:39,042 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321400 2023-11-22 23:28:54,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2142713.3333333335, ans=0.04949747468305833 2023-11-22 23:29:03,169 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8800, loss[loss=0.05273, simple_loss=0.06783, pruned_loss=0.008408, audio_tagging_loss=0.0104, over 14799.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09451, pruned_loss=0.01482, audio_tagging_loss=0.00951, over 3056889.43 frames. ], batch size: 54, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:29:03,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2142780.0, ans=0.1 2023-11-22 23:29:42,055 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:29:44,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321450 2023-11-22 23:29:48,685 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0 2023-11-22 23:29:55,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2143046.6666666665, ans=0.0 2023-11-22 23:29:55,936 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.814e+01 8.244e+01 8.962e+01 9.489e+01 1.229e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-22 23:29:56,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2143046.6666666665, ans=0.2 2023-11-22 23:30:03,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2143046.6666666665, ans=0.0 2023-11-22 23:30:10,451 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8850, loss[loss=0.07362, simple_loss=0.1037, pruned_loss=0.01625, audio_tagging_loss=0.005522, over 14731.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09421, pruned_loss=0.01483, audio_tagging_loss=0.009505, over 3051182.16 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:30:12,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2143113.3333333335, ans=0.1 2023-11-22 23:30:21,759 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 23:30:25,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.06 vs. limit=15.0 2023-11-22 23:30:50,364 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321500 2023-11-22 23:30:59,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2143313.3333333335, ans=0.125 2023-11-22 23:31:14,523 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8900, loss[loss=0.06168, simple_loss=0.08741, pruned_loss=0.008751, audio_tagging_loss=0.009221, over 15069.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09434, pruned_loss=0.01469, audio_tagging_loss=0.009375, over 3053217.73 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:31:46,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2143580.0, ans=0.0 2023-11-22 23:31:55,139 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321550 2023-11-22 23:32:00,598 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.03 vs. limit=12.0 2023-11-22 23:32:05,883 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 8.181e+01 8.758e+01 9.474e+01 1.266e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-22 23:32:10,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0 2023-11-22 23:32:13,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2143713.3333333335, ans=0.1 2023-11-22 23:32:18,178 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 8950, loss[loss=0.06161, simple_loss=0.08498, pruned_loss=0.01268, audio_tagging_loss=0.006444, over 14112.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09416, pruned_loss=0.01475, audio_tagging_loss=0.009272, over 3048009.79 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:32:20,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2143780.0, ans=0.0 2023-11-22 23:32:25,858 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.07 vs. limit=15.0 2023-11-22 23:32:38,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2143846.6666666665, ans=0.125 2023-11-22 23:32:46,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2143913.3333333335, ans=0.0 2023-11-22 23:32:54,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2143913.3333333335, ans=0.125 2023-11-22 23:32:55,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2143913.3333333335, ans=0.125 2023-11-22 23:33:00,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321600 2023-11-22 23:33:22,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2144046.6666666665, ans=0.125 2023-11-22 23:33:26,993 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9000, loss[loss=0.07099, simple_loss=0.08758, pruned_loss=0.01708, audio_tagging_loss=0.01012, over 15158.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09335, pruned_loss=0.01456, audio_tagging_loss=0.00921, over 3051382.67 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:33:26,994 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-22 23:34:01,571 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4882, 3.2497, 3.7192, 3.4898], device='cuda:3') 2023-11-22 23:34:08,297 INFO [train_asr.py:1253] (3/4) Epoch 27, validation: loss=0.05906, simple_loss=0.05129, pruned_loss=0.005052, audio_tagging_loss=0.02836, over 4681554.00 frames. 2023-11-22 23:34:08,298 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-22 23:34:08,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.87 vs. limit=15.0 2023-11-22 23:34:12,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2144113.3333333335, ans=0.1 2023-11-22 23:34:17,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2144113.3333333335, ans=0.1 2023-11-22 23:34:27,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.47 vs. limit=10.0 2023-11-22 23:34:49,949 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321650 2023-11-22 23:34:50,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.48 vs. limit=5.0 2023-11-22 23:34:52,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2144313.3333333335, ans=0.0 2023-11-22 23:34:56,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2144313.3333333335, ans=0.0 2023-11-22 23:35:00,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2144380.0, ans=0.0 2023-11-22 23:35:01,328 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.041e+01 8.419e+01 9.018e+01 9.927e+01 1.306e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-22 23:35:04,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2144380.0, ans=0.125 2023-11-22 23:35:04,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2144380.0, ans=0.05 2023-11-22 23:35:10,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2144380.0, ans=0.125 2023-11-22 23:35:13,815 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9050, loss[loss=0.07004, simple_loss=0.08888, pruned_loss=0.01594, audio_tagging_loss=0.009663, over 15451.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09448, pruned_loss=0.01486, audio_tagging_loss=0.009122, over 3052269.29 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:35:18,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2144446.6666666665, ans=0.125 2023-11-22 23:35:23,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.63 vs. limit=10.0 2023-11-22 23:35:36,598 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=22.5 2023-11-22 23:35:46,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2144580.0, ans=0.125 2023-11-22 23:35:54,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321700 2023-11-22 23:35:56,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2144646.6666666665, ans=0.125 2023-11-22 23:35:58,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2144646.6666666665, ans=0.5 2023-11-22 23:36:01,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2144646.6666666665, ans=0.0 2023-11-22 23:36:19,468 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9100, loss[loss=0.05632, simple_loss=0.07242, pruned_loss=0.01053, audio_tagging_loss=0.009589, over 15557.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09429, pruned_loss=0.01471, audio_tagging_loss=0.009029, over 3053145.45 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:36:54,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2144913.3333333335, ans=0.0 2023-11-22 23:36:58,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321750 2023-11-22 23:37:05,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2144980.0, ans=0.1 2023-11-22 23:37:11,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.202e+01 8.707e+01 9.347e+01 1.081e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-22 23:37:22,636 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9150, loss[loss=0.06442, simple_loss=0.07972, pruned_loss=0.01498, audio_tagging_loss=0.00958, over 14448.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09336, pruned_loss=0.01466, audio_tagging_loss=0.009055, over 3047616.65 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:37:34,160 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.10 vs. limit=22.5 2023-11-22 23:37:36,372 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:38:02,850 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321800 2023-11-22 23:38:06,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2145313.3333333335, ans=0.125 2023-11-22 23:38:20,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2145380.0, ans=0.125 2023-11-22 23:38:26,075 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9200, loss[loss=0.09185, simple_loss=0.1304, pruned_loss=0.01886, audio_tagging_loss=0.007779, over 14909.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09381, pruned_loss=0.01474, audio_tagging_loss=0.009091, over 3051369.48 frames. ], batch size: 54, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:38:33,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2145446.6666666665, ans=0.125 2023-11-22 23:38:37,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2145446.6666666665, ans=0.0 2023-11-22 23:39:05,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2145646.6666666665, ans=0.125 2023-11-22 23:39:06,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321850 2023-11-22 23:39:07,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2145646.6666666665, ans=0.0 2023-11-22 23:39:18,717 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.155e+01 8.815e+01 9.566e+01 1.382e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-22 23:39:25,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2145713.3333333335, ans=0.1 2023-11-22 23:39:28,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2145713.3333333335, ans=0.0 2023-11-22 23:39:30,946 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9250, loss[loss=0.07753, simple_loss=0.1042, pruned_loss=0.01672, audio_tagging_loss=0.008701, over 14649.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09326, pruned_loss=0.01454, audio_tagging_loss=0.009123, over 3059267.14 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:39:33,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2145780.0, ans=0.125 2023-11-22 23:39:45,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2145846.6666666665, ans=0.2 2023-11-22 23:40:09,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2145980.0, ans=0.125 2023-11-22 23:40:10,310 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.75 vs. limit=6.0 2023-11-22 23:40:10,930 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321900 2023-11-22 23:40:16,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2145980.0, ans=0.125 2023-11-22 23:40:22,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2146046.6666666665, ans=0.0 2023-11-22 23:40:29,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2146046.6666666665, ans=0.125 2023-11-22 23:40:35,069 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9300, loss[loss=0.0696, simple_loss=0.08982, pruned_loss=0.01792, audio_tagging_loss=0.006763, over 14666.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09306, pruned_loss=0.01441, audio_tagging_loss=0.009107, over 3054075.98 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:40:40,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2146113.3333333335, ans=0.05 2023-11-22 23:40:41,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2146113.3333333335, ans=0.125 2023-11-22 23:40:43,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2146113.3333333335, ans=0.125 2023-11-22 23:40:59,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2146246.6666666665, ans=0.0 2023-11-22 23:41:05,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2146246.6666666665, ans=0.1 2023-11-22 23:41:14,786 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 321950 2023-11-22 23:41:24,032 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0 2023-11-22 23:41:26,761 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.225e+01 8.828e+01 9.380e+01 1.148e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-22 23:41:29,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2146380.0, ans=0.2 2023-11-22 23:41:37,954 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9350, loss[loss=0.06472, simple_loss=0.07716, pruned_loss=0.01426, audio_tagging_loss=0.01187, over 14995.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09276, pruned_loss=0.01444, audio_tagging_loss=0.009188, over 3051374.44 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:41:42,269 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.50 vs. limit=22.5 2023-11-22 23:41:48,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2146446.6666666665, ans=0.125 2023-11-22 23:42:17,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2146646.6666666665, ans=0.125 2023-11-22 23:42:17,909 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322000 2023-11-22 23:42:36,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2146713.3333333335, ans=0.1 2023-11-22 23:42:43,231 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9400, loss[loss=0.05695, simple_loss=0.07628, pruned_loss=0.01047, audio_tagging_loss=0.00834, over 15597.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09334, pruned_loss=0.01458, audio_tagging_loss=0.009193, over 3051833.97 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:42:44,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2146780.0, ans=0.2 2023-11-22 23:42:45,142 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.34 vs. limit=6.0 2023-11-22 23:42:47,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2146780.0, ans=0.0 2023-11-22 23:42:49,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2146780.0, ans=0.0 2023-11-22 23:43:01,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2146846.6666666665, ans=0.125 2023-11-22 23:43:08,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2146913.3333333335, ans=0.125 2023-11-22 23:43:18,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2146913.3333333335, ans=0.0 2023-11-22 23:43:20,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2146980.0, ans=0.0 2023-11-22 23:43:21,465 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.72 vs. limit=15.0 2023-11-22 23:43:23,250 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322050 2023-11-22 23:43:36,903 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.391e+01 8.949e+01 9.601e+01 1.196e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-22 23:43:43,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.38 vs. limit=15.0 2023-11-22 23:43:44,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2147046.6666666665, ans=0.0 2023-11-22 23:43:44,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2147046.6666666665, ans=0.1 2023-11-22 23:43:46,457 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 23:43:49,035 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9450, loss[loss=0.05436, simple_loss=0.06554, pruned_loss=0.01164, audio_tagging_loss=0.009943, over 14275.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.093, pruned_loss=0.01457, audio_tagging_loss=0.009264, over 3049625.11 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:43:54,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2147113.3333333335, ans=0.125 2023-11-22 23:44:11,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2147180.0, ans=0.2 2023-11-22 23:44:29,029 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322100 2023-11-22 23:44:40,109 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.41 vs. limit=22.5 2023-11-22 23:44:53,041 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9500, loss[loss=0.07472, simple_loss=0.1034, pruned_loss=0.01405, audio_tagging_loss=0.008956, over 15302.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09283, pruned_loss=0.01467, audio_tagging_loss=0.009309, over 3044480.57 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:44:55,232 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.59 vs. limit=15.0 2023-11-22 23:45:13,440 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:45:19,967 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0 2023-11-22 23:45:32,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2147646.6666666665, ans=0.125 2023-11-22 23:45:33,421 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322150 2023-11-22 23:45:35,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2147646.6666666665, ans=0.125 2023-11-22 23:45:45,471 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.349e+01 8.991e+01 9.634e+01 1.651e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-22 23:45:55,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2147713.3333333335, ans=0.1 2023-11-22 23:45:57,760 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9550, loss[loss=0.07002, simple_loss=0.09733, pruned_loss=0.01229, audio_tagging_loss=0.009055, over 15510.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09258, pruned_loss=0.01441, audio_tagging_loss=0.009425, over 3040104.06 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:46:05,354 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.14 vs. limit=15.0 2023-11-22 23:46:26,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2147913.3333333335, ans=0.125 2023-11-22 23:46:27,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2147913.3333333335, ans=0.0 2023-11-22 23:46:37,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322200 2023-11-22 23:46:39,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.26 vs. limit=15.0 2023-11-22 23:46:40,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2147980.0, ans=0.0 2023-11-22 23:46:51,879 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.23 vs. limit=22.5 2023-11-22 23:47:01,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2148046.6666666665, ans=0.2 2023-11-22 23:47:03,433 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9600, loss[loss=0.07877, simple_loss=0.11, pruned_loss=0.01778, audio_tagging_loss=0.005974, over 15046.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.0937, pruned_loss=0.01452, audio_tagging_loss=0.009476, over 3044056.66 frames. ], batch size: 54, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:47:07,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2148113.3333333335, ans=0.125 2023-11-22 23:47:25,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2148180.0, ans=0.0 2023-11-22 23:47:26,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2148180.0, ans=0.0 2023-11-22 23:47:39,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.63 vs. limit=15.0 2023-11-22 23:47:43,911 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322250 2023-11-22 23:47:57,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.150e+01 8.727e+01 9.513e+01 1.269e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-22 23:48:06,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2148446.6666666665, ans=0.125 2023-11-22 23:48:07,664 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9650, loss[loss=0.04664, simple_loss=0.05462, pruned_loss=0.008992, audio_tagging_loss=0.01034, over 14574.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09316, pruned_loss=0.01445, audio_tagging_loss=0.009447, over 3048466.06 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:48:16,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2148446.6666666665, ans=0.1 2023-11-22 23:48:43,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.61 vs. limit=15.0 2023-11-22 23:48:44,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2148580.0, ans=0.1 2023-11-22 23:48:46,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2148646.6666666665, ans=0.1 2023-11-22 23:48:47,530 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322300 2023-11-22 23:49:11,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2148780.0, ans=0.125 2023-11-22 23:49:12,276 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9700, loss[loss=0.07769, simple_loss=0.1034, pruned_loss=0.01711, audio_tagging_loss=0.008866, over 15144.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09337, pruned_loss=0.01435, audio_tagging_loss=0.009235, over 3049157.00 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:49:52,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322350 2023-11-22 23:49:53,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2148980.0, ans=0.2 2023-11-22 23:49:57,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2148980.0, ans=0.0 2023-11-22 23:50:01,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2148980.0, ans=0.125 2023-11-22 23:50:07,998 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.299e+01 9.052e+01 9.780e+01 1.137e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-22 23:50:16,519 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9750, loss[loss=0.0625, simple_loss=0.07773, pruned_loss=0.013, audio_tagging_loss=0.01063, over 15480.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09391, pruned_loss=0.01432, audio_tagging_loss=0.009083, over 3049242.62 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:50:25,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2149113.3333333335, ans=0.0 2023-11-22 23:50:27,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2149180.0, ans=0.0 2023-11-22 23:50:36,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2149180.0, ans=0.2 2023-11-22 23:50:55,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2149313.3333333335, ans=0.0 2023-11-22 23:50:56,599 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322400 2023-11-22 23:50:59,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2149313.3333333335, ans=0.0 2023-11-22 23:51:01,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.38 vs. limit=15.0 2023-11-22 23:51:06,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.25 vs. limit=10.0 2023-11-22 23:51:20,624 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9800, loss[loss=0.07005, simple_loss=0.09411, pruned_loss=0.01215, audio_tagging_loss=0.01085, over 15226.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09404, pruned_loss=0.01435, audio_tagging_loss=0.009038, over 3055421.96 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:51:39,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2149513.3333333335, ans=0.07 2023-11-22 23:52:01,159 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322450 2023-11-22 23:52:01,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2149646.6666666665, ans=0.125 2023-11-22 23:52:14,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.66 vs. limit=22.5 2023-11-22 23:52:14,123 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.41 vs. limit=10.0 2023-11-22 23:52:16,547 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.583e+01 9.205e+01 9.862e+01 1.172e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-22 23:52:17,947 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 23:52:20,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2149713.3333333335, ans=0.2 2023-11-22 23:52:25,291 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9850, loss[loss=0.0594, simple_loss=0.07177, pruned_loss=0.0126, audio_tagging_loss=0.01092, over 15845.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09409, pruned_loss=0.01447, audio_tagging_loss=0.009025, over 3052091.92 frames. ], batch size: 61, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:52:25,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2149780.0, ans=0.125 2023-11-22 23:52:42,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2149846.6666666665, ans=0.125 2023-11-22 23:53:06,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322500 2023-11-22 23:53:06,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2149980.0, ans=0.035 2023-11-22 23:53:31,654 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9900, loss[loss=0.0782, simple_loss=0.1029, pruned_loss=0.01643, audio_tagging_loss=0.0103, over 15515.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09421, pruned_loss=0.01445, audio_tagging_loss=0.009131, over 3053574.37 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:53:41,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2150113.3333333335, ans=0.0 2023-11-22 23:53:58,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2150246.6666666665, ans=0.1 2023-11-22 23:54:08,446 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:54:12,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322550 2023-11-22 23:54:14,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2150313.3333333335, ans=0.0 2023-11-22 23:54:19,025 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:54:26,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2150380.0, ans=0.125 2023-11-22 23:54:27,616 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.162e+01 8.931e+01 9.625e+01 1.127e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-22 23:54:35,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2150446.6666666665, ans=0.0 2023-11-22 23:54:36,340 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 9950, loss[loss=0.06455, simple_loss=0.08091, pruned_loss=0.01337, audio_tagging_loss=0.01072, over 16092.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09471, pruned_loss=0.01449, audio_tagging_loss=0.009038, over 3055188.58 frames. ], batch size: 61, lr: 2.51e-03, grad_scale: 16.0 2023-11-22 23:54:45,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2150446.6666666665, ans=0.125 2023-11-22 23:55:04,260 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:55:05,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2150580.0, ans=0.0 2023-11-22 23:55:08,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2150580.0, ans=0.2 2023-11-22 23:55:11,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2150580.0, ans=0.0 2023-11-22 23:55:16,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322600 2023-11-22 23:55:16,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2150646.6666666665, ans=0.2 2023-11-22 23:55:20,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2150646.6666666665, ans=0.125 2023-11-22 23:55:41,032 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10000, loss[loss=0.07538, simple_loss=0.09547, pruned_loss=0.02025, audio_tagging_loss=0.007387, over 14779.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09438, pruned_loss=0.01468, audio_tagging_loss=0.009111, over 3052072.15 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:55:44,641 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:55:47,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2150780.0, ans=0.125 2023-11-22 23:56:06,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2150913.3333333335, ans=0.2 2023-11-22 23:56:21,705 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322650 2023-11-22 23:56:38,554 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.648e+01 8.099e+01 8.716e+01 9.532e+01 1.465e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-22 23:56:43,036 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-22 23:56:47,936 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10050, loss[loss=0.09002, simple_loss=0.123, pruned_loss=0.02264, audio_tagging_loss=0.005886, over 14343.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09412, pruned_loss=0.0147, audio_tagging_loss=0.00907, over 3043843.58 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:56:58,455 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.41 vs. limit=15.0 2023-11-22 23:57:00,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2151180.0, ans=0.125 2023-11-22 23:57:02,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.71 vs. limit=15.0 2023-11-22 23:57:12,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.75 vs. limit=10.0 2023-11-22 23:57:27,978 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322700 2023-11-22 23:57:44,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2151380.0, ans=0.0 2023-11-22 23:57:51,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2151446.6666666665, ans=0.125 2023-11-22 23:57:52,568 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10100, loss[loss=0.05473, simple_loss=0.07007, pruned_loss=0.01026, audio_tagging_loss=0.009431, over 14726.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09378, pruned_loss=0.01459, audio_tagging_loss=0.009207, over 3042448.90 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:57:59,604 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.55 vs. limit=15.0 2023-11-22 23:58:06,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2151513.3333333335, ans=0.125 2023-11-22 23:58:13,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.93 vs. limit=15.0 2023-11-22 23:58:19,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2151580.0, ans=0.125 2023-11-22 23:58:33,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322750 2023-11-22 23:58:44,952 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 23:58:48,675 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.658e+01 8.081e+01 8.819e+01 9.576e+01 1.489e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-22 23:58:57,831 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10150, loss[loss=0.06364, simple_loss=0.08643, pruned_loss=0.01076, audio_tagging_loss=0.009667, over 14542.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09345, pruned_loss=0.01455, audio_tagging_loss=0.009298, over 3042444.44 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0 2023-11-22 23:59:22,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2151846.6666666665, ans=0.07 2023-11-22 23:59:28,805 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-22 23:59:32,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.55 vs. limit=15.0 2023-11-22 23:59:37,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2151980.0, ans=0.1 2023-11-22 23:59:38,978 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322800 2023-11-22 23:59:48,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2151980.0, ans=0.2 2023-11-23 00:00:04,822 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10200, loss[loss=0.07378, simple_loss=0.09407, pruned_loss=0.01595, audio_tagging_loss=0.01079, over 16511.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09417, pruned_loss=0.01479, audio_tagging_loss=0.009285, over 3051392.99 frames. ], batch size: 60, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:00:16,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2152180.0, ans=0.1 2023-11-23 00:00:27,277 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 00:00:44,050 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322850 2023-11-23 00:00:44,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.43 vs. limit=12.0 2023-11-23 00:00:59,979 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.282e+01 8.917e+01 9.345e+01 1.171e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 00:01:05,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2152380.0, ans=0.0 2023-11-23 00:01:06,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2152380.0, ans=0.0 2023-11-23 00:01:06,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2152380.0, ans=0.2 2023-11-23 00:01:07,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2152446.6666666665, ans=0.125 2023-11-23 00:01:08,759 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10250, loss[loss=0.0915, simple_loss=0.1239, pruned_loss=0.01964, audio_tagging_loss=0.009933, over 16026.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09378, pruned_loss=0.01483, audio_tagging_loss=0.00934, over 3052076.99 frames. ], batch size: 60, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:01:11,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2152446.6666666665, ans=0.0 2023-11-23 00:01:25,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2152513.3333333335, ans=0.5 2023-11-23 00:01:47,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2152646.6666666665, ans=0.125 2023-11-23 00:01:49,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322900 2023-11-23 00:02:02,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2152713.3333333335, ans=0.125 2023-11-23 00:02:12,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2152780.0, ans=0.025 2023-11-23 00:02:13,731 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10300, loss[loss=0.06825, simple_loss=0.08418, pruned_loss=0.01463, audio_tagging_loss=0.01153, over 14840.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09432, pruned_loss=0.01485, audio_tagging_loss=0.009325, over 3053592.51 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:02:13,963 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 00:02:38,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2152846.6666666665, ans=0.125 2023-11-23 00:02:51,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2152980.0, ans=0.125 2023-11-23 00:02:53,990 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 322950 2023-11-23 00:03:00,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2152980.0, ans=0.125 2023-11-23 00:03:10,238 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.589e+01 8.288e+01 8.993e+01 9.603e+01 1.175e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 00:03:10,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2153046.6666666665, ans=0.125 2023-11-23 00:03:13,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2153046.6666666665, ans=0.125 2023-11-23 00:03:15,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2153046.6666666665, ans=10.0 2023-11-23 00:03:16,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2153046.6666666665, ans=0.0 2023-11-23 00:03:19,089 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10350, loss[loss=0.06581, simple_loss=0.08179, pruned_loss=0.01331, audio_tagging_loss=0.0116, over 14701.00 frames. ], tot_loss[loss=0.07142, simple_loss=0.09438, pruned_loss=0.01482, audio_tagging_loss=0.009409, over 3049502.45 frames. ], batch size: 54, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:03:19,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2153113.3333333335, ans=0.1 2023-11-23 00:03:58,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323000 2023-11-23 00:04:24,484 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10400, loss[loss=0.07066, simple_loss=0.09362, pruned_loss=0.01351, audio_tagging_loss=0.01034, over 14963.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09446, pruned_loss=0.01486, audio_tagging_loss=0.009536, over 3046299.33 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:04:31,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2153446.6666666665, ans=0.1 2023-11-23 00:04:32,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2153446.6666666665, ans=0.125 2023-11-23 00:04:32,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2153446.6666666665, ans=0.125 2023-11-23 00:05:01,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2153580.0, ans=0.0 2023-11-23 00:05:05,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323050 2023-11-23 00:05:15,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2153713.3333333335, ans=0.125 2023-11-23 00:05:17,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2153713.3333333335, ans=0.0 2023-11-23 00:05:21,776 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 8.434e+01 9.021e+01 9.797e+01 1.156e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 00:05:29,013 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10450, loss[loss=0.07919, simple_loss=0.1049, pruned_loss=0.016, audio_tagging_loss=0.01073, over 15749.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09457, pruned_loss=0.01489, audio_tagging_loss=0.009443, over 3040042.48 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:05:45,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2153846.6666666665, ans=0.125 2023-11-23 00:05:53,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2153846.6666666665, ans=0.125 2023-11-23 00:06:01,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2153913.3333333335, ans=0.125 2023-11-23 00:06:02,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2153913.3333333335, ans=0.0 2023-11-23 00:06:03,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2153913.3333333335, ans=0.125 2023-11-23 00:06:09,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323100 2023-11-23 00:06:33,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2154113.3333333335, ans=0.125 2023-11-23 00:06:34,663 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10500, loss[loss=0.05997, simple_loss=0.07548, pruned_loss=0.009284, audio_tagging_loss=0.01295, over 14290.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09394, pruned_loss=0.01471, audio_tagging_loss=0.009339, over 3043780.67 frames. ], batch size: 54, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:06:35,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.whiten.whitening_limit, batch_count=2154113.3333333335, ans=12.0 2023-11-23 00:06:36,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2154113.3333333335, ans=0.125 2023-11-23 00:07:03,718 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.59 vs. limit=22.5 2023-11-23 00:07:14,239 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323150 2023-11-23 00:07:29,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.19 vs. limit=15.0 2023-11-23 00:07:32,617 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.185e+01 8.836e+01 9.458e+01 1.353e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 00:07:38,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2154446.6666666665, ans=0.1 2023-11-23 00:07:39,214 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10550, loss[loss=0.06603, simple_loss=0.09583, pruned_loss=0.01174, audio_tagging_loss=0.006376, over 15018.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09244, pruned_loss=0.01438, audio_tagging_loss=0.0093, over 3042624.96 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:08:18,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2154646.6666666665, ans=0.1 2023-11-23 00:08:19,421 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323200 2023-11-23 00:08:43,851 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10600, loss[loss=0.07568, simple_loss=0.1008, pruned_loss=0.01583, audio_tagging_loss=0.009434, over 14702.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09212, pruned_loss=0.01431, audio_tagging_loss=0.009164, over 3035579.01 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:09:03,979 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=15.0 2023-11-23 00:09:24,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323250 2023-11-23 00:09:33,767 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.02 vs. limit=6.0 2023-11-23 00:09:41,926 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 7.982e+01 8.705e+01 9.514e+01 1.291e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-23 00:09:48,202 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10650, loss[loss=0.08821, simple_loss=0.1185, pruned_loss=0.02187, audio_tagging_loss=0.007061, over 15486.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09298, pruned_loss=0.01451, audio_tagging_loss=0.009067, over 3030334.43 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:09:58,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.09 vs. limit=15.0 2023-11-23 00:09:59,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2155113.3333333335, ans=0.05 2023-11-23 00:10:14,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2155246.6666666665, ans=0.125 2023-11-23 00:10:23,956 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 00:10:28,641 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323300 2023-11-23 00:10:47,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2155380.0, ans=0.125 2023-11-23 00:10:47,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2155380.0, ans=0.0 2023-11-23 00:10:52,977 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10700, loss[loss=0.06397, simple_loss=0.08643, pruned_loss=0.0123, audio_tagging_loss=0.008447, over 15633.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.0941, pruned_loss=0.01463, audio_tagging_loss=0.008943, over 3031749.82 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:10:57,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2155446.6666666665, ans=0.0 2023-11-23 00:11:00,046 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.02 vs. limit=15.0 2023-11-23 00:11:33,539 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323350 2023-11-23 00:11:47,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2155713.3333333335, ans=0.125 2023-11-23 00:11:50,554 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.154e+01 8.901e+01 9.766e+01 1.298e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-23 00:11:57,404 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10750, loss[loss=0.07209, simple_loss=0.09696, pruned_loss=0.01441, audio_tagging_loss=0.009193, over 15919.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09266, pruned_loss=0.01425, audio_tagging_loss=0.008996, over 3035991.53 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:11:59,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.79 vs. limit=22.5 2023-11-23 00:12:02,450 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 00:12:06,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2155780.0, ans=0.125 2023-11-23 00:12:13,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2155846.6666666665, ans=0.5 2023-11-23 00:12:17,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2155846.6666666665, ans=0.0 2023-11-23 00:12:37,275 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323400 2023-11-23 00:12:44,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2155980.0, ans=0.125 2023-11-23 00:12:54,797 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.96 vs. limit=15.0 2023-11-23 00:13:01,572 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10800, loss[loss=0.05522, simple_loss=0.07267, pruned_loss=0.009456, audio_tagging_loss=0.009429, over 14525.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09351, pruned_loss=0.01435, audio_tagging_loss=0.008923, over 3035023.87 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:13:12,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2156113.3333333335, ans=0.2 2023-11-23 00:13:33,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2156246.6666666665, ans=0.125 2023-11-23 00:13:33,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2156246.6666666665, ans=0.04949747468305833 2023-11-23 00:13:35,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2156246.6666666665, ans=0.125 2023-11-23 00:13:37,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2156246.6666666665, ans=0.1 2023-11-23 00:13:39,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2156313.3333333335, ans=0.125 2023-11-23 00:13:42,008 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323450 2023-11-23 00:14:00,566 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.240e+01 8.825e+01 9.425e+01 1.162e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-23 00:14:03,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2156380.0, ans=0.125 2023-11-23 00:14:06,736 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10850, loss[loss=0.08029, simple_loss=0.1239, pruned_loss=0.01113, audio_tagging_loss=0.007221, over 15663.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09412, pruned_loss=0.01435, audio_tagging_loss=0.008964, over 3038072.08 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:14:38,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2156580.0, ans=0.125 2023-11-23 00:14:46,699 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323500 2023-11-23 00:14:57,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2156713.3333333335, ans=0.125 2023-11-23 00:15:06,844 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 00:15:10,557 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10900, loss[loss=0.07168, simple_loss=0.1032, pruned_loss=0.01241, audio_tagging_loss=0.007689, over 15999.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09388, pruned_loss=0.01429, audio_tagging_loss=0.009026, over 3036031.26 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:15:36,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.96 vs. limit=15.0 2023-11-23 00:15:51,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323550 2023-11-23 00:15:55,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2156980.0, ans=0.2 2023-11-23 00:16:03,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.94 vs. limit=22.5 2023-11-23 00:16:10,516 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.063e+01 8.887e+01 9.492e+01 1.128e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-23 00:16:15,450 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 10950, loss[loss=0.06903, simple_loss=0.08951, pruned_loss=0.01207, audio_tagging_loss=0.01221, over 15771.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09324, pruned_loss=0.01428, audio_tagging_loss=0.009157, over 3036350.86 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:16:45,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2157246.6666666665, ans=0.0 2023-11-23 00:16:49,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.95 vs. limit=15.0 2023-11-23 00:16:52,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2157313.3333333335, ans=0.0 2023-11-23 00:16:55,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323600 2023-11-23 00:16:57,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.89 vs. limit=15.0 2023-11-23 00:17:03,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2157313.3333333335, ans=0.0 2023-11-23 00:17:05,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2157380.0, ans=0.125 2023-11-23 00:17:09,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2157380.0, ans=0.125 2023-11-23 00:17:12,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2157380.0, ans=0.125 2023-11-23 00:17:14,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2157380.0, ans=0.0 2023-11-23 00:17:20,276 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11000, loss[loss=0.08276, simple_loss=0.1148, pruned_loss=0.01776, audio_tagging_loss=0.007605, over 14644.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09298, pruned_loss=0.01424, audio_tagging_loss=0.009299, over 3043607.75 frames. ], batch size: 53, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:17:20,921 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.55 vs. limit=22.5 2023-11-23 00:17:24,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2157446.6666666665, ans=0.125 2023-11-23 00:17:24,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.89 vs. limit=10.0 2023-11-23 00:17:25,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2157446.6666666665, ans=0.2 2023-11-23 00:17:30,174 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 00:17:32,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2157513.3333333335, ans=0.125 2023-11-23 00:17:34,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2157513.3333333335, ans=0.1 2023-11-23 00:17:50,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2157580.0, ans=0.0 2023-11-23 00:17:53,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2157580.0, ans=0.0 2023-11-23 00:17:59,845 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323650 2023-11-23 00:18:18,918 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.419e+01 9.027e+01 1.014e+02 1.259e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 00:18:22,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2157780.0, ans=0.07 2023-11-23 00:18:24,044 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11050, loss[loss=0.04847, simple_loss=0.05912, pruned_loss=0.008193, audio_tagging_loss=0.01072, over 14338.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09263, pruned_loss=0.01443, audio_tagging_loss=0.009396, over 3051580.80 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:18:24,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2157780.0, ans=0.1 2023-11-23 00:18:40,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.93 vs. limit=15.0 2023-11-23 00:19:02,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2157980.0, ans=0.0 2023-11-23 00:19:04,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323700 2023-11-23 00:19:09,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2157980.0, ans=0.0 2023-11-23 00:19:26,341 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0 2023-11-23 00:19:28,754 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11100, loss[loss=0.0833, simple_loss=0.1151, pruned_loss=0.01879, audio_tagging_loss=0.006956, over 15147.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09325, pruned_loss=0.01442, audio_tagging_loss=0.009382, over 3056149.00 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:19:49,063 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.92 vs. limit=15.0 2023-11-23 00:19:53,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2158246.6666666665, ans=0.0 2023-11-23 00:20:06,566 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.18 vs. limit=15.0 2023-11-23 00:20:08,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323750 2023-11-23 00:20:28,950 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.251e+01 8.340e+01 9.049e+01 9.831e+01 1.206e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 00:20:34,615 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11150, loss[loss=0.09567, simple_loss=0.1346, pruned_loss=0.01981, audio_tagging_loss=0.008562, over 15762.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09338, pruned_loss=0.01452, audio_tagging_loss=0.009529, over 3057064.11 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:20:34,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2158446.6666666665, ans=0.025 2023-11-23 00:20:55,469 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.73 vs. limit=15.0 2023-11-23 00:20:59,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2158580.0, ans=0.0 2023-11-23 00:21:07,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2158580.0, ans=0.125 2023-11-23 00:21:09,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2158580.0, ans=0.0 2023-11-23 00:21:10,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2158580.0, ans=0.125 2023-11-23 00:21:13,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.21 vs. limit=15.0 2023-11-23 00:21:14,171 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323800 2023-11-23 00:21:39,501 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11200, loss[loss=0.06745, simple_loss=0.0859, pruned_loss=0.01441, audio_tagging_loss=0.01008, over 15352.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09337, pruned_loss=0.0145, audio_tagging_loss=0.009537, over 3052617.87 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:21:46,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2158780.0, ans=0.125 2023-11-23 00:21:53,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2158846.6666666665, ans=0.1 2023-11-23 00:22:20,898 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323850 2023-11-23 00:22:39,536 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.459e+01 8.191e+01 8.566e+01 9.520e+01 1.333e+02, threshold=1.713e+02, percent-clipped=0.0 2023-11-23 00:22:44,555 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11250, loss[loss=0.05632, simple_loss=0.07124, pruned_loss=0.009954, audio_tagging_loss=0.01074, over 16127.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09201, pruned_loss=0.01425, audio_tagging_loss=0.009659, over 3047961.37 frames. ], batch size: 61, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:22:56,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2159113.3333333335, ans=0.0 2023-11-23 00:23:14,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2159246.6666666665, ans=0.125 2023-11-23 00:23:20,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2159246.6666666665, ans=0.1 2023-11-23 00:23:25,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323900 2023-11-23 00:23:50,761 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11300, loss[loss=0.0579, simple_loss=0.07702, pruned_loss=0.009844, audio_tagging_loss=0.009552, over 15274.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09288, pruned_loss=0.01464, audio_tagging_loss=0.009453, over 3048625.46 frames. ], batch size: 60, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:24:10,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2159513.3333333335, ans=0.0 2023-11-23 00:24:17,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2159580.0, ans=0.125 2023-11-23 00:24:29,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 323950 2023-11-23 00:24:40,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2159646.6666666665, ans=0.125 2023-11-23 00:24:50,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2159713.3333333335, ans=0.125 2023-11-23 00:24:51,668 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.358e+01 9.001e+01 9.797e+01 1.218e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-23 00:24:53,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2159713.3333333335, ans=0.125 2023-11-23 00:24:55,579 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11350, loss[loss=0.05402, simple_loss=0.07081, pruned_loss=0.008612, audio_tagging_loss=0.01, over 15397.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09321, pruned_loss=0.01476, audio_tagging_loss=0.009247, over 3044439.56 frames. ], batch size: 60, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:25:16,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2159846.6666666665, ans=0.2 2023-11-23 00:25:18,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2159846.6666666665, ans=0.1 2023-11-23 00:25:20,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.10 vs. limit=22.5 2023-11-23 00:25:36,784 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324000 2023-11-23 00:26:03,549 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11400, loss[loss=0.07423, simple_loss=0.102, pruned_loss=0.01742, audio_tagging_loss=0.005825, over 16385.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09272, pruned_loss=0.01456, audio_tagging_loss=0.009158, over 3039183.20 frames. ], batch size: 60, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:26:11,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2160113.3333333335, ans=0.125 2023-11-23 00:26:25,909 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 00:26:32,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2160246.6666666665, ans=0.125 2023-11-23 00:26:42,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2160313.3333333335, ans=0.125 2023-11-23 00:26:44,779 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324050 2023-11-23 00:27:05,049 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.089e+01 8.744e+01 9.515e+01 1.376e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-23 00:27:10,230 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11450, loss[loss=0.1006, simple_loss=0.14, pruned_loss=0.02405, audio_tagging_loss=0.006607, over 16597.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09321, pruned_loss=0.01462, audio_tagging_loss=0.009178, over 3047459.39 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:27:16,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2160446.6666666665, ans=0.1 2023-11-23 00:27:22,413 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.26 vs. limit=15.0 2023-11-23 00:27:25,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2160513.3333333335, ans=0.1 2023-11-23 00:27:42,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2160580.0, ans=0.1 2023-11-23 00:27:50,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324100 2023-11-23 00:28:03,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2160713.3333333335, ans=0.0 2023-11-23 00:28:16,250 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11500, loss[loss=0.07206, simple_loss=0.09666, pruned_loss=0.01499, audio_tagging_loss=0.00874, over 14904.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.093, pruned_loss=0.01452, audio_tagging_loss=0.009057, over 3051438.10 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:28:36,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2160846.6666666665, ans=0.125 2023-11-23 00:28:37,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.52 vs. limit=15.0 2023-11-23 00:28:44,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.02 vs. limit=22.5 2023-11-23 00:28:47,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2160913.3333333335, ans=0.125 2023-11-23 00:28:56,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2160980.0, ans=0.125 2023-11-23 00:28:57,060 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.81 vs. limit=10.0 2023-11-23 00:28:57,853 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324150 2023-11-23 00:28:59,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2160980.0, ans=0.125 2023-11-23 00:29:18,300 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 7.992e+01 8.677e+01 9.342e+01 1.172e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-23 00:29:22,186 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11550, loss[loss=0.07325, simple_loss=0.09955, pruned_loss=0.01223, audio_tagging_loss=0.01124, over 15672.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09333, pruned_loss=0.01454, audio_tagging_loss=0.009049, over 3048890.38 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:29:47,959 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.71 vs. limit=22.5 2023-11-23 00:29:56,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2161246.6666666665, ans=0.125 2023-11-23 00:30:03,379 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 00:30:03,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2161313.3333333335, ans=0.125 2023-11-23 00:30:04,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324200 2023-11-23 00:30:06,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.09 vs. limit=22.5 2023-11-23 00:30:18,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2161380.0, ans=0.0 2023-11-23 00:30:19,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2161380.0, ans=0.1 2023-11-23 00:30:26,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2161380.0, ans=0.04949747468305833 2023-11-23 00:30:29,739 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11600, loss[loss=0.04964, simple_loss=0.06564, pruned_loss=0.005756, audio_tagging_loss=0.01107, over 15772.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09299, pruned_loss=0.0145, audio_tagging_loss=0.009004, over 3046780.45 frames. ], batch size: 60, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:30:30,360 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.22 vs. limit=15.0 2023-11-23 00:30:46,971 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 00:30:56,349 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.43 vs. limit=22.5 2023-11-23 00:31:09,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2161646.6666666665, ans=0.125 2023-11-23 00:31:10,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324250 2023-11-23 00:31:32,563 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.566e+01 8.394e+01 8.884e+01 9.610e+01 1.253e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-23 00:31:36,428 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11650, loss[loss=0.06594, simple_loss=0.08221, pruned_loss=0.01297, audio_tagging_loss=0.01187, over 15359.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09351, pruned_loss=0.01447, audio_tagging_loss=0.009031, over 3044809.48 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:31:46,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2161780.0, ans=0.125 2023-11-23 00:31:53,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2161846.6666666665, ans=0.2 2023-11-23 00:32:00,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2161913.3333333335, ans=0.09899494936611666 2023-11-23 00:32:03,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2161913.3333333335, ans=0.5 2023-11-23 00:32:05,486 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 00:32:09,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.67 vs. limit=10.0 2023-11-23 00:32:17,568 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324300 2023-11-23 00:32:19,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2161980.0, ans=0.125 2023-11-23 00:32:28,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2162046.6666666665, ans=0.2 2023-11-23 00:32:38,936 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.55 vs. limit=22.5 2023-11-23 00:32:42,050 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11700, loss[loss=0.08404, simple_loss=0.1141, pruned_loss=0.01859, audio_tagging_loss=0.008411, over 15949.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09239, pruned_loss=0.01428, audio_tagging_loss=0.009223, over 3045061.09 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:33:00,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2162180.0, ans=0.125 2023-11-23 00:33:08,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.02 vs. limit=10.0 2023-11-23 00:33:21,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2162313.3333333335, ans=0.125 2023-11-23 00:33:23,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324350 2023-11-23 00:33:26,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.90 vs. limit=15.0 2023-11-23 00:33:27,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2162313.3333333335, ans=0.125 2023-11-23 00:33:36,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2162380.0, ans=0.0 2023-11-23 00:33:43,247 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.270e+01 8.836e+01 9.259e+01 1.366e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 00:33:46,983 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11750, loss[loss=0.05748, simple_loss=0.07452, pruned_loss=0.01266, audio_tagging_loss=0.007558, over 14572.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09295, pruned_loss=0.01448, audio_tagging_loss=0.009192, over 3044500.51 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:33:57,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2162446.6666666665, ans=0.125 2023-11-23 00:34:01,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2162513.3333333335, ans=0.0 2023-11-23 00:34:16,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2162580.0, ans=0.125 2023-11-23 00:34:19,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2162580.0, ans=0.125 2023-11-23 00:34:28,629 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324400 2023-11-23 00:34:54,181 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11800, loss[loss=0.06514, simple_loss=0.0904, pruned_loss=0.01188, audio_tagging_loss=0.008052, over 15189.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09304, pruned_loss=0.01441, audio_tagging_loss=0.009163, over 3037007.49 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:34:54,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2162780.0, ans=0.2 2023-11-23 00:35:07,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2162846.6666666665, ans=0.1 2023-11-23 00:35:22,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2162913.3333333335, ans=0.125 2023-11-23 00:35:34,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324450 2023-11-23 00:35:36,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=15.0 2023-11-23 00:35:39,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2162980.0, ans=0.125 2023-11-23 00:35:44,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2162980.0, ans=0.0 2023-11-23 00:35:56,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2163046.6666666665, ans=0.125 2023-11-23 00:35:57,205 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.225e+01 9.023e+01 9.525e+01 1.272e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 00:35:59,720 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11850, loss[loss=0.09333, simple_loss=0.1156, pruned_loss=0.02564, audio_tagging_loss=0.009872, over 15092.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09324, pruned_loss=0.01459, audio_tagging_loss=0.009297, over 3035586.96 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:36:11,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2163180.0, ans=0.2 2023-11-23 00:36:11,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2163180.0, ans=0.1 2023-11-23 00:36:16,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2163180.0, ans=0.035 2023-11-23 00:36:29,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.13 vs. limit=12.0 2023-11-23 00:36:31,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2163246.6666666665, ans=0.2 2023-11-23 00:36:40,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324500 2023-11-23 00:36:46,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2163313.3333333335, ans=0.1 2023-11-23 00:36:57,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2163380.0, ans=0.0 2023-11-23 00:37:04,535 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11900, loss[loss=0.07737, simple_loss=0.1051, pruned_loss=0.01834, audio_tagging_loss=0.006463, over 15031.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09273, pruned_loss=0.01432, audio_tagging_loss=0.009497, over 3038954.51 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:37:21,815 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=12.0 2023-11-23 00:37:45,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324550 2023-11-23 00:37:47,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2163646.6666666665, ans=0.125 2023-11-23 00:38:06,586 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.44 vs. limit=6.0 2023-11-23 00:38:08,412 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.374e+01 8.930e+01 9.704e+01 2.155e+02, threshold=1.786e+02, percent-clipped=1.0 2023-11-23 00:38:10,887 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 11950, loss[loss=0.05727, simple_loss=0.076, pruned_loss=0.009174, audio_tagging_loss=0.0101, over 15577.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09253, pruned_loss=0.01414, audio_tagging_loss=0.009554, over 3041095.31 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0 2023-11-23 00:38:49,172 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.05 vs. limit=15.0 2023-11-23 00:38:49,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324600 2023-11-23 00:39:14,390 INFO [train_asr.py:1221] (3/4) Epoch 27, batch 12000, loss[loss=0.07304, simple_loss=0.09022, pruned_loss=0.01596, audio_tagging_loss=0.01197, over 14306.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09319, pruned_loss=0.01442, audio_tagging_loss=0.009534, over 3042400.36 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0 2023-11-23 00:39:14,391 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 00:39:57,338 INFO [train_asr.py:1253] (3/4) Epoch 27, validation: loss=0.05869, simple_loss=0.05138, pruned_loss=0.005099, audio_tagging_loss=0.0279, over 4681554.00 frames. 2023-11-23 00:39:57,339 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 00:40:00,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.68 vs. limit=15.0 2023-11-23 00:40:06,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2164113.3333333335, ans=0.0 2023-11-23 00:40:08,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.84 vs. limit=10.0 2023-11-23 00:40:15,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2164180.0, ans=0.0 2023-11-23 00:41:02,701 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 0, loss[loss=0.05338, simple_loss=0.04702, pruned_loss=0.005241, audio_tagging_loss=0.02463, over 16563.00 frames. ], tot_loss[loss=0.05338, simple_loss=0.04702, pruned_loss=0.005241, audio_tagging_loss=0.02463, over 16563.00 frames. ], batch size: 66, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:41:02,701 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 00:41:39,756 INFO [train_asr.py:1253] (3/4) Epoch 28, validation: loss=0.0583, simple_loss=0.05139, pruned_loss=0.005161, audio_tagging_loss=0.02744, over 4681554.00 frames. 2023-11-23 00:41:39,757 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 00:41:45,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.38 vs. limit=6.0 2023-11-23 00:41:47,390 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324650 2023-11-23 00:41:54,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.56 vs. limit=15.0 2023-11-23 00:41:54,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2164346.6666666665, ans=0.0 2023-11-23 00:42:09,901 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.350e+01 8.610e+01 9.593e+01 1.048e+02 1.343e+02, threshold=1.919e+02, percent-clipped=0.0 2023-11-23 00:42:31,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2164546.6666666665, ans=0.2 2023-11-23 00:42:43,126 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 50, loss[loss=0.07352, simple_loss=0.07547, pruned_loss=0.01519, audio_tagging_loss=0.02059, over 14634.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.08537, pruned_loss=0.01273, audio_tagging_loss=0.01855, over 686658.69 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:42:50,563 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324700 2023-11-23 00:42:56,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2164680.0, ans=0.125 2023-11-23 00:43:29,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2164813.3333333335, ans=0.2 2023-11-23 00:43:46,718 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 100, loss[loss=0.07902, simple_loss=0.09719, pruned_loss=0.01689, audio_tagging_loss=0.01353, over 15410.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09141, pruned_loss=0.01455, audio_tagging_loss=0.01754, over 1209563.23 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:43:54,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324750 2023-11-23 00:44:01,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2165013.3333333335, ans=0.125 2023-11-23 00:44:12,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2165080.0, ans=0.1 2023-11-23 00:44:17,223 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.407e+01 9.043e+01 9.747e+01 1.056e+02 1.211e+02, threshold=1.949e+02, percent-clipped=0.0 2023-11-23 00:44:18,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2165080.0, ans=0.125 2023-11-23 00:44:21,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2165080.0, ans=0.125 2023-11-23 00:44:30,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2165146.6666666665, ans=0.125 2023-11-23 00:44:49,967 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 150, loss[loss=0.06888, simple_loss=0.09275, pruned_loss=0.01154, audio_tagging_loss=0.01097, over 14843.00 frames. ], tot_loss[loss=0.07414, simple_loss=0.089, pruned_loss=0.01385, audio_tagging_loss=0.01579, over 1620275.59 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:44:58,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324800 2023-11-23 00:45:03,761 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=12.0 2023-11-23 00:45:39,799 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=12.0 2023-11-23 00:45:54,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2165613.3333333335, ans=0.125 2023-11-23 00:45:55,028 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 200, loss[loss=0.0765, simple_loss=0.1022, pruned_loss=0.01832, audio_tagging_loss=0.007096, over 14749.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.08973, pruned_loss=0.01409, audio_tagging_loss=0.01372, over 1930343.22 frames. ], batch size: 55, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:46:01,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2165613.3333333335, ans=0.0 2023-11-23 00:46:02,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324850 2023-11-23 00:46:25,065 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.466e+01 9.149e+01 9.933e+01 1.935e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-23 00:46:29,953 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.77 vs. limit=15.0 2023-11-23 00:46:50,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2165880.0, ans=0.125 2023-11-23 00:46:56,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2165880.0, ans=0.125 2023-11-23 00:46:58,569 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 250, loss[loss=0.06927, simple_loss=0.09579, pruned_loss=0.0137, audio_tagging_loss=0.007672, over 15495.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.093, pruned_loss=0.01452, audio_tagging_loss=0.01219, over 2184471.55 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:47:06,482 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324900 2023-11-23 00:47:10,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2166013.3333333335, ans=0.1 2023-11-23 00:47:20,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2166013.3333333335, ans=6.0 2023-11-23 00:47:29,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0 2023-11-23 00:47:34,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2166080.0, ans=0.125 2023-11-23 00:47:34,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2166080.0, ans=0.1 2023-11-23 00:47:43,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2166146.6666666665, ans=0.2 2023-11-23 00:47:47,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2166146.6666666665, ans=0.2 2023-11-23 00:48:03,089 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 300, loss[loss=0.08859, simple_loss=0.1235, pruned_loss=0.02018, audio_tagging_loss=0.006673, over 16213.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09305, pruned_loss=0.01447, audio_tagging_loss=0.0113, over 2377928.43 frames. ], batch size: 60, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:48:05,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2166280.0, ans=0.125 2023-11-23 00:48:11,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 324950 2023-11-23 00:48:34,180 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.439e+01 8.323e+01 9.001e+01 1.002e+02 1.826e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-23 00:48:38,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2166413.3333333335, ans=0.0 2023-11-23 00:48:45,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2166480.0, ans=0.2 2023-11-23 00:48:47,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2166480.0, ans=0.125 2023-11-23 00:49:02,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2166546.6666666665, ans=0.125 2023-11-23 00:49:09,307 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 350, loss[loss=0.09682, simple_loss=0.1341, pruned_loss=0.02292, audio_tagging_loss=0.006828, over 15634.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.0927, pruned_loss=0.01444, audio_tagging_loss=0.0107, over 2522235.70 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 00:49:12,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2166613.3333333335, ans=0.09899494936611666 2023-11-23 00:49:16,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325000 2023-11-23 00:49:20,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2166613.3333333335, ans=0.1 2023-11-23 00:49:51,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2166813.3333333335, ans=0.1 2023-11-23 00:50:02,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2166880.0, ans=0.1 2023-11-23 00:50:04,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2166880.0, ans=0.0 2023-11-23 00:50:12,988 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 400, loss[loss=0.05279, simple_loss=0.06894, pruned_loss=0.008685, audio_tagging_loss=0.009631, over 15415.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09277, pruned_loss=0.01447, audio_tagging_loss=0.01034, over 2638503.91 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:50:20,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325050 2023-11-23 00:50:23,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2166946.6666666665, ans=0.125 2023-11-23 00:50:24,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2167013.3333333335, ans=0.1 2023-11-23 00:50:27,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2167013.3333333335, ans=0.125 2023-11-23 00:50:29,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2167013.3333333335, ans=0.0 2023-11-23 00:50:40,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2167080.0, ans=0.0 2023-11-23 00:50:45,769 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.175e+01 8.785e+01 9.638e+01 1.202e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-23 00:50:46,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2167080.0, ans=0.1 2023-11-23 00:51:03,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2167213.3333333335, ans=0.0 2023-11-23 00:51:08,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2167213.3333333335, ans=0.1 2023-11-23 00:51:17,313 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 450, loss[loss=0.06761, simple_loss=0.09002, pruned_loss=0.01575, audio_tagging_loss=0.006852, over 16503.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09377, pruned_loss=0.01451, audio_tagging_loss=0.009937, over 2734750.43 frames. ], batch size: 64, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:51:25,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325100 2023-11-23 00:51:26,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2167280.0, ans=0.125 2023-11-23 00:51:26,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2167280.0, ans=0.2 2023-11-23 00:51:28,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2167280.0, ans=0.0 2023-11-23 00:51:40,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2167346.6666666665, ans=0.125 2023-11-23 00:51:42,522 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.49 vs. limit=15.0 2023-11-23 00:51:45,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2167413.3333333335, ans=0.125 2023-11-23 00:51:49,801 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.73 vs. limit=15.0 2023-11-23 00:52:04,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2167480.0, ans=0.0 2023-11-23 00:52:05,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2167480.0, ans=0.09899494936611666 2023-11-23 00:52:07,522 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.28 vs. limit=15.0 2023-11-23 00:52:23,263 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 500, loss[loss=0.0726, simple_loss=0.09635, pruned_loss=0.01609, audio_tagging_loss=0.008339, over 16301.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09406, pruned_loss=0.01452, audio_tagging_loss=0.00969, over 2801828.98 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:52:30,630 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325150 2023-11-23 00:52:34,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2167680.0, ans=0.07 2023-11-23 00:52:54,606 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.145e+01 8.686e+01 9.449e+01 1.268e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-23 00:53:27,163 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 550, loss[loss=0.0878, simple_loss=0.1263, pruned_loss=0.01831, audio_tagging_loss=0.006362, over 15617.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09398, pruned_loss=0.01445, audio_tagging_loss=0.00966, over 2858798.24 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:53:34,568 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325200 2023-11-23 00:53:57,961 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0 2023-11-23 00:54:12,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2168146.6666666665, ans=0.0 2023-11-23 00:54:32,703 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 600, loss[loss=0.07839, simple_loss=0.09632, pruned_loss=0.01801, audio_tagging_loss=0.01222, over 14934.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09419, pruned_loss=0.01456, audio_tagging_loss=0.009569, over 2901652.56 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:54:40,346 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325250 2023-11-23 00:54:53,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2168346.6666666665, ans=0.1 2023-11-23 00:54:56,523 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.06 vs. limit=15.0 2023-11-23 00:55:04,347 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.119e+01 8.698e+01 9.405e+01 1.451e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-23 00:55:09,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2168480.0, ans=0.1 2023-11-23 00:55:19,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2168480.0, ans=0.1 2023-11-23 00:55:24,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2168546.6666666665, ans=0.125 2023-11-23 00:55:34,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2168546.6666666665, ans=0.0 2023-11-23 00:55:36,585 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 650, loss[loss=0.07432, simple_loss=0.104, pruned_loss=0.01175, audio_tagging_loss=0.01057, over 14322.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.0928, pruned_loss=0.01431, audio_tagging_loss=0.009493, over 2924813.16 frames. ], batch size: 53, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:55:44,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325300 2023-11-23 00:56:06,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2168746.6666666665, ans=0.125 2023-11-23 00:56:09,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2168746.6666666665, ans=0.0 2023-11-23 00:56:30,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2168880.0, ans=0.125 2023-11-23 00:56:35,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2168880.0, ans=0.07 2023-11-23 00:56:38,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2168880.0, ans=0.2 2023-11-23 00:56:40,464 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 700, loss[loss=0.06324, simple_loss=0.08768, pruned_loss=0.01216, audio_tagging_loss=0.007249, over 15304.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09316, pruned_loss=0.01429, audio_tagging_loss=0.009414, over 2957566.24 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:56:47,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325350 2023-11-23 00:56:48,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2168946.6666666665, ans=0.2 2023-11-23 00:56:55,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.86 vs. limit=15.0 2023-11-23 00:57:00,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2169013.3333333335, ans=0.0 2023-11-23 00:57:12,510 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.33 vs. limit=15.0 2023-11-23 00:57:13,024 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.373e+01 9.027e+01 9.909e+01 1.159e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 00:57:23,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2169146.6666666665, ans=0.125 2023-11-23 00:57:44,764 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 750, loss[loss=0.06072, simple_loss=0.0816, pruned_loss=0.01167, audio_tagging_loss=0.008259, over 15559.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09311, pruned_loss=0.01432, audio_tagging_loss=0.00946, over 2977585.09 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:57:52,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325400 2023-11-23 00:58:00,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2169346.6666666665, ans=0.0 2023-11-23 00:58:09,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2169346.6666666665, ans=0.0 2023-11-23 00:58:12,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2169413.3333333335, ans=0.2 2023-11-23 00:58:27,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2169480.0, ans=0.125 2023-11-23 00:58:49,715 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 800, loss[loss=0.07261, simple_loss=0.09787, pruned_loss=0.01342, audio_tagging_loss=0.01026, over 16250.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09381, pruned_loss=0.01441, audio_tagging_loss=0.009348, over 3000164.64 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 00:58:57,647 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325450 2023-11-23 00:59:02,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2169680.0, ans=0.125 2023-11-23 00:59:02,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2169680.0, ans=0.2 2023-11-23 00:59:21,177 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.947e+01 8.109e+01 8.929e+01 9.816e+01 1.280e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-23 00:59:29,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2169813.3333333335, ans=0.2 2023-11-23 00:59:45,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2169880.0, ans=0.125 2023-11-23 00:59:47,048 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 00:59:54,380 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 850, loss[loss=0.07813, simple_loss=0.1031, pruned_loss=0.0175, audio_tagging_loss=0.009067, over 15558.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09376, pruned_loss=0.01436, audio_tagging_loss=0.009519, over 3009715.66 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:00:01,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325500 2023-11-23 01:00:07,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2170013.3333333335, ans=0.015 2023-11-23 01:00:09,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2170013.3333333335, ans=0.125 2023-11-23 01:00:24,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2170080.0, ans=0.1 2023-11-23 01:00:35,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2170146.6666666665, ans=0.125 2023-11-23 01:00:57,815 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 900, loss[loss=0.07189, simple_loss=0.09573, pruned_loss=0.01236, audio_tagging_loss=0.01166, over 15894.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09338, pruned_loss=0.01441, audio_tagging_loss=0.009631, over 3020258.02 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:00:58,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.81 vs. limit=22.5 2023-11-23 01:01:06,205 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325550 2023-11-23 01:01:10,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.80 vs. limit=15.0 2023-11-23 01:01:21,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2170346.6666666665, ans=0.2 2023-11-23 01:01:32,253 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.700e+01 8.072e+01 8.678e+01 9.441e+01 1.146e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-23 01:01:34,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.02 vs. limit=15.0 2023-11-23 01:01:51,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2170546.6666666665, ans=0.125 2023-11-23 01:01:55,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2170546.6666666665, ans=0.2 2023-11-23 01:01:58,585 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.68 vs. limit=6.0 2023-11-23 01:02:01,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2170613.3333333335, ans=0.125 2023-11-23 01:02:02,732 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 950, loss[loss=0.07281, simple_loss=0.09775, pruned_loss=0.01466, audio_tagging_loss=0.009266, over 14665.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09396, pruned_loss=0.01451, audio_tagging_loss=0.009431, over 3025434.90 frames. ], batch size: 55, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:02:04,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2170613.3333333335, ans=0.0 2023-11-23 01:02:08,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2170613.3333333335, ans=15.0 2023-11-23 01:02:09,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2170613.3333333335, ans=0.2 2023-11-23 01:02:11,563 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325600 2023-11-23 01:02:19,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2170680.0, ans=0.035 2023-11-23 01:02:29,515 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.00 vs. limit=15.0 2023-11-23 01:02:32,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2170746.6666666665, ans=0.125 2023-11-23 01:02:47,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2170813.3333333335, ans=0.1 2023-11-23 01:03:08,278 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1000, loss[loss=0.08231, simple_loss=0.1146, pruned_loss=0.01887, audio_tagging_loss=0.006138, over 14892.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09423, pruned_loss=0.01465, audio_tagging_loss=0.009234, over 3025226.47 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:03:08,867 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.55 vs. limit=15.0 2023-11-23 01:03:15,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2170946.6666666665, ans=15.0 2023-11-23 01:03:16,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325650 2023-11-23 01:03:31,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2171013.3333333335, ans=0.125 2023-11-23 01:03:34,962 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:03:35,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2171080.0, ans=0.0 2023-11-23 01:03:41,020 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.239e+01 8.945e+01 9.781e+01 1.255e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-23 01:03:49,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2171146.6666666665, ans=0.125 2023-11-23 01:03:51,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2171146.6666666665, ans=0.125 2023-11-23 01:03:59,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2171213.3333333335, ans=10.0 2023-11-23 01:04:06,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2171213.3333333335, ans=0.125 2023-11-23 01:04:06,501 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.91 vs. limit=22.5 2023-11-23 01:04:11,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2171280.0, ans=0.125 2023-11-23 01:04:11,925 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1050, loss[loss=0.06994, simple_loss=0.0914, pruned_loss=0.01244, audio_tagging_loss=0.0118, over 15778.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09358, pruned_loss=0.0146, audio_tagging_loss=0.009197, over 3027415.89 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:04:19,832 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325700 2023-11-23 01:04:22,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2171280.0, ans=0.0 2023-11-23 01:04:32,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.25 vs. limit=15.0 2023-11-23 01:05:09,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2171546.6666666665, ans=0.125 2023-11-23 01:05:15,390 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1100, loss[loss=0.06778, simple_loss=0.09962, pruned_loss=0.01162, audio_tagging_loss=0.006351, over 15783.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.0928, pruned_loss=0.01448, audio_tagging_loss=0.009167, over 3032312.99 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:05:19,140 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:05:23,592 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325750 2023-11-23 01:05:37,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2171680.0, ans=0.2 2023-11-23 01:05:48,720 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.331e+01 9.113e+01 9.844e+01 1.667e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-23 01:05:52,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2171813.3333333335, ans=0.125 2023-11-23 01:05:54,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2171813.3333333335, ans=0.0 2023-11-23 01:06:05,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2171880.0, ans=10.0 2023-11-23 01:06:20,013 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1150, loss[loss=0.08677, simple_loss=0.1154, pruned_loss=0.0231, audio_tagging_loss=0.005995, over 16339.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.0924, pruned_loss=0.01432, audio_tagging_loss=0.009105, over 3036238.67 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:06:27,734 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325800 2023-11-23 01:06:29,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2171946.6666666665, ans=0.09899494936611666 2023-11-23 01:06:50,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2172080.0, ans=0.0 2023-11-23 01:06:53,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2172080.0, ans=0.0 2023-11-23 01:07:06,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.94 vs. limit=15.0 2023-11-23 01:07:14,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2172213.3333333335, ans=0.0 2023-11-23 01:07:24,592 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1200, loss[loss=0.07807, simple_loss=0.106, pruned_loss=0.01772, audio_tagging_loss=0.007349, over 15887.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09356, pruned_loss=0.01459, audio_tagging_loss=0.008981, over 3042736.89 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 01:07:32,290 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325850 2023-11-23 01:07:57,970 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.204e+01 8.430e+01 8.952e+01 9.650e+01 2.402e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-23 01:08:27,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2172613.3333333335, ans=0.125 2023-11-23 01:08:29,015 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1250, loss[loss=0.06685, simple_loss=0.08978, pruned_loss=0.01142, audio_tagging_loss=0.01054, over 15452.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09369, pruned_loss=0.01473, audio_tagging_loss=0.009033, over 3036257.59 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 01:08:37,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325900 2023-11-23 01:08:42,522 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:08:52,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2172680.0, ans=0.125 2023-11-23 01:09:34,058 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1300, loss[loss=0.07502, simple_loss=0.1048, pruned_loss=0.01468, audio_tagging_loss=0.007949, over 15209.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.0938, pruned_loss=0.0146, audio_tagging_loss=0.008993, over 3038845.97 frames. ], batch size: 55, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:09:41,678 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 325950 2023-11-23 01:09:45,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2173013.3333333335, ans=0.2 2023-11-23 01:09:49,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2173013.3333333335, ans=0.125 2023-11-23 01:10:04,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2173080.0, ans=0.125 2023-11-23 01:10:06,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2173080.0, ans=0.0 2023-11-23 01:10:08,885 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.443e+01 8.136e+01 8.697e+01 9.484e+01 1.452e+02, threshold=1.739e+02, percent-clipped=1.0 2023-11-23 01:10:12,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2173146.6666666665, ans=0.125 2023-11-23 01:10:33,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2173213.3333333335, ans=0.125 2023-11-23 01:10:38,105 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1350, loss[loss=0.07033, simple_loss=0.09672, pruned_loss=0.01242, audio_tagging_loss=0.009551, over 15357.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09345, pruned_loss=0.0143, audio_tagging_loss=0.008958, over 3047791.60 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:10:45,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326000 2023-11-23 01:10:48,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2173280.0, ans=0.0 2023-11-23 01:11:01,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2173346.6666666665, ans=0.035 2023-11-23 01:11:07,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2173413.3333333335, ans=0.2 2023-11-23 01:11:17,983 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.31 vs. limit=6.0 2023-11-23 01:11:21,110 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:11:25,638 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:11:33,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2173546.6666666665, ans=0.2 2023-11-23 01:11:39,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2173546.6666666665, ans=0.0 2023-11-23 01:11:42,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2173613.3333333335, ans=0.125 2023-11-23 01:11:42,785 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1400, loss[loss=0.06133, simple_loss=0.07058, pruned_loss=0.01292, audio_tagging_loss=0.01312, over 15035.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09317, pruned_loss=0.01436, audio_tagging_loss=0.009037, over 3044745.18 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:11:50,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326050 2023-11-23 01:12:08,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2173746.6666666665, ans=0.0 2023-11-23 01:12:17,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.070e+01 8.559e+01 9.321e+01 1.215e+02, threshold=1.712e+02, percent-clipped=0.0 2023-11-23 01:12:23,489 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:12:29,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2173813.3333333335, ans=0.125 2023-11-23 01:12:47,046 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1450, loss[loss=0.0688, simple_loss=0.09188, pruned_loss=0.01492, audio_tagging_loss=0.007933, over 14622.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09419, pruned_loss=0.01459, audio_tagging_loss=0.009118, over 3039875.88 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:12:48,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2173946.6666666665, ans=0.2 2023-11-23 01:12:54,563 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326100 2023-11-23 01:13:50,355 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1500, loss[loss=0.06092, simple_loss=0.08111, pruned_loss=0.01031, audio_tagging_loss=0.01006, over 14719.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09364, pruned_loss=0.01455, audio_tagging_loss=0.009202, over 3039844.12 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:13:57,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326150 2023-11-23 01:14:00,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2174280.0, ans=0.1 2023-11-23 01:14:02,202 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.89 vs. limit=15.0 2023-11-23 01:14:16,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2174413.3333333335, ans=0.125 2023-11-23 01:14:26,194 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.046e+01 8.933e+01 9.681e+01 1.186e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 01:14:39,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2174480.0, ans=0.125 2023-11-23 01:14:41,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2174546.6666666665, ans=0.0 2023-11-23 01:14:55,313 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1550, loss[loss=0.07501, simple_loss=0.09627, pruned_loss=0.01832, audio_tagging_loss=0.008558, over 14120.00 frames. ], tot_loss[loss=0.071, simple_loss=0.0942, pruned_loss=0.01466, audio_tagging_loss=0.009237, over 3034869.48 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:15:04,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326200 2023-11-23 01:15:04,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2174613.3333333335, ans=0.0 2023-11-23 01:15:34,926 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:15:54,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2174880.0, ans=0.125 2023-11-23 01:16:03,424 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1600, loss[loss=0.06765, simple_loss=0.08662, pruned_loss=0.01529, audio_tagging_loss=0.009052, over 14692.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09332, pruned_loss=0.01464, audio_tagging_loss=0.009438, over 3037226.20 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 01:16:03,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2174946.6666666665, ans=0.125 2023-11-23 01:16:11,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326250 2023-11-23 01:16:22,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2175013.3333333335, ans=0.1 2023-11-23 01:16:31,128 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:16:33,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2175080.0, ans=0.1 2023-11-23 01:16:37,599 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.240e+01 8.854e+01 9.452e+01 1.276e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-23 01:16:56,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2175213.3333333335, ans=0.125 2023-11-23 01:17:07,137 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1650, loss[loss=0.06214, simple_loss=0.0855, pruned_loss=0.01153, audio_tagging_loss=0.00786, over 15260.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.0932, pruned_loss=0.01457, audio_tagging_loss=0.009388, over 3038550.27 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0 2023-11-23 01:17:14,793 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326300 2023-11-23 01:17:17,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2175280.0, ans=0.0 2023-11-23 01:17:18,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2175346.6666666665, ans=0.125 2023-11-23 01:17:32,263 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0 2023-11-23 01:17:50,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.67 vs. limit=15.0 2023-11-23 01:17:54,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2175480.0, ans=0.125 2023-11-23 01:17:57,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2175546.6666666665, ans=0.0 2023-11-23 01:18:04,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2175546.6666666665, ans=0.0 2023-11-23 01:18:11,452 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1700, loss[loss=0.07832, simple_loss=0.1038, pruned_loss=0.017, audio_tagging_loss=0.009436, over 15516.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09334, pruned_loss=0.0147, audio_tagging_loss=0.009429, over 3040807.41 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:18:19,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326350 2023-11-23 01:18:30,147 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.47 vs. limit=22.5 2023-11-23 01:18:47,668 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.185e+01 8.813e+01 9.746e+01 1.184e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-23 01:19:03,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2175880.0, ans=0.125 2023-11-23 01:19:05,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2175880.0, ans=0.125 2023-11-23 01:19:09,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2175880.0, ans=0.125 2023-11-23 01:19:16,168 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1750, loss[loss=0.09234, simple_loss=0.1216, pruned_loss=0.02184, audio_tagging_loss=0.009708, over 13895.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.0927, pruned_loss=0.0146, audio_tagging_loss=0.009459, over 3041505.88 frames. ], batch size: 52, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:19:23,573 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326400 2023-11-23 01:19:24,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2175946.6666666665, ans=0.0 2023-11-23 01:19:39,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2176080.0, ans=0.125 2023-11-23 01:19:46,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2176080.0, ans=0.125 2023-11-23 01:20:02,619 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.97 vs. limit=15.0 2023-11-23 01:20:20,576 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1800, loss[loss=0.05997, simple_loss=0.07787, pruned_loss=0.008357, audio_tagging_loss=0.01268, over 14963.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09362, pruned_loss=0.01455, audio_tagging_loss=0.009198, over 3045439.46 frames. ], batch size: 55, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:20:22,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2176280.0, ans=0.5 2023-11-23 01:20:28,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326450 2023-11-23 01:20:57,606 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 8.178e+01 8.671e+01 9.411e+01 1.178e+02, threshold=1.734e+02, percent-clipped=0.0 2023-11-23 01:21:17,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2176546.6666666665, ans=0.09899494936611666 2023-11-23 01:21:24,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2176613.3333333335, ans=0.025 2023-11-23 01:21:25,132 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1850, loss[loss=0.04624, simple_loss=0.05049, pruned_loss=0.009014, audio_tagging_loss=0.01198, over 13835.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09334, pruned_loss=0.01437, audio_tagging_loss=0.009234, over 3038657.85 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0 2023-11-23 01:21:33,494 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326500 2023-11-23 01:21:44,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2176680.0, ans=0.125 2023-11-23 01:22:01,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2176746.6666666665, ans=0.125 2023-11-23 01:22:02,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2176746.6666666665, ans=0.1 2023-11-23 01:22:02,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2176746.6666666665, ans=0.1 2023-11-23 01:22:03,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2176813.3333333335, ans=0.125 2023-11-23 01:22:21,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2176880.0, ans=0.2 2023-11-23 01:22:31,058 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1900, loss[loss=0.06599, simple_loss=0.08631, pruned_loss=0.01433, audio_tagging_loss=0.008499, over 15373.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09307, pruned_loss=0.01437, audio_tagging_loss=0.009129, over 3038489.20 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:22:39,646 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326550 2023-11-23 01:23:03,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2177080.0, ans=0.125 2023-11-23 01:23:03,563 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=12.0 2023-11-23 01:23:04,215 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:23:06,401 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.801e+01 8.128e+01 8.928e+01 9.740e+01 2.315e+02, threshold=1.786e+02, percent-clipped=1.0 2023-11-23 01:23:07,053 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.95 vs. limit=15.0 2023-11-23 01:23:18,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2177146.6666666665, ans=0.1 2023-11-23 01:23:36,014 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 1950, loss[loss=0.0707, simple_loss=0.1061, pruned_loss=0.01, audio_tagging_loss=0.007665, over 15049.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09347, pruned_loss=0.01447, audio_tagging_loss=0.009021, over 3040867.63 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:23:43,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326600 2023-11-23 01:23:43,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2177280.0, ans=0.0 2023-11-23 01:24:05,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2177413.3333333335, ans=0.125 2023-11-23 01:24:20,936 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.34 vs. limit=22.5 2023-11-23 01:24:38,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2177546.6666666665, ans=0.1 2023-11-23 01:24:40,692 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2000, loss[loss=0.08987, simple_loss=0.1134, pruned_loss=0.0225, audio_tagging_loss=0.01065, over 14392.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09342, pruned_loss=0.01457, audio_tagging_loss=0.009117, over 3042312.15 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:24:48,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326650 2023-11-23 01:24:59,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2177680.0, ans=0.125 2023-11-23 01:25:02,982 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.77 vs. limit=10.0 2023-11-23 01:25:03,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2177680.0, ans=0.1 2023-11-23 01:25:07,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2177746.6666666665, ans=0.125 2023-11-23 01:25:17,648 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.041e+01 8.683e+01 9.382e+01 1.367e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-23 01:25:33,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2177880.0, ans=0.2 2023-11-23 01:25:39,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2177880.0, ans=0.2 2023-11-23 01:25:45,372 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2050, loss[loss=0.07522, simple_loss=0.09566, pruned_loss=0.01917, audio_tagging_loss=0.008219, over 15199.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09357, pruned_loss=0.01457, audio_tagging_loss=0.009198, over 3042170.84 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:25:45,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2177946.6666666665, ans=0.1 2023-11-23 01:25:48,607 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0 2023-11-23 01:25:53,484 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326700 2023-11-23 01:26:06,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2178013.3333333335, ans=0.0 2023-11-23 01:26:19,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2178080.0, ans=0.125 2023-11-23 01:26:20,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2178080.0, ans=0.125 2023-11-23 01:26:22,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2178146.6666666665, ans=0.1 2023-11-23 01:26:36,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2178213.3333333335, ans=0.0 2023-11-23 01:26:41,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.71 vs. limit=12.0 2023-11-23 01:26:44,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2178213.3333333335, ans=0.125 2023-11-23 01:26:46,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.20 vs. limit=15.0 2023-11-23 01:26:49,581 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2100, loss[loss=0.0705, simple_loss=0.1014, pruned_loss=0.01283, audio_tagging_loss=0.006956, over 14787.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09369, pruned_loss=0.01463, audio_tagging_loss=0.009171, over 3036857.05 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:26:57,533 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326750 2023-11-23 01:27:25,010 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.865e+01 8.249e+01 8.824e+01 9.661e+01 1.236e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-23 01:27:39,483 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:27:47,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2178546.6666666665, ans=0.0 2023-11-23 01:27:53,265 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2150, loss[loss=0.08299, simple_loss=0.1049, pruned_loss=0.02005, audio_tagging_loss=0.01051, over 15539.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09389, pruned_loss=0.01471, audio_tagging_loss=0.009192, over 3042520.77 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:27:53,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2178613.3333333335, ans=0.125 2023-11-23 01:27:54,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2178613.3333333335, ans=0.0 2023-11-23 01:27:57,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2178613.3333333335, ans=0.1 2023-11-23 01:28:00,746 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326800 2023-11-23 01:28:04,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2178613.3333333335, ans=0.2 2023-11-23 01:28:23,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2178746.6666666665, ans=0.1 2023-11-23 01:28:33,809 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:28:58,350 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2200, loss[loss=0.06618, simple_loss=0.08711, pruned_loss=0.01348, audio_tagging_loss=0.009143, over 14388.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09305, pruned_loss=0.01446, audio_tagging_loss=0.009137, over 3039491.11 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:29:00,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2178946.6666666665, ans=0.125 2023-11-23 01:29:05,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2178946.6666666665, ans=0.2 2023-11-23 01:29:06,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326850 2023-11-23 01:29:07,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2178946.6666666665, ans=0.0 2023-11-23 01:29:13,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2179013.3333333335, ans=0.1 2023-11-23 01:29:18,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2179013.3333333335, ans=0.125 2023-11-23 01:29:34,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.469e+01 8.071e+01 8.694e+01 9.748e+01 1.270e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-23 01:29:57,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2179213.3333333335, ans=0.0 2023-11-23 01:29:58,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2179213.3333333335, ans=0.1 2023-11-23 01:30:03,040 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2250, loss[loss=0.07192, simple_loss=0.09372, pruned_loss=0.01474, audio_tagging_loss=0.01032, over 15141.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09368, pruned_loss=0.01461, audio_tagging_loss=0.00919, over 3044928.71 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:30:10,491 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326900 2023-11-23 01:30:22,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2179346.6666666665, ans=0.1 2023-11-23 01:30:24,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2179346.6666666665, ans=0.0 2023-11-23 01:30:26,158 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.79 vs. limit=15.0 2023-11-23 01:30:38,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2179413.3333333335, ans=0.0 2023-11-23 01:30:43,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2179480.0, ans=0.0 2023-11-23 01:30:52,547 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.87 vs. limit=12.0 2023-11-23 01:30:59,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.31 vs. limit=6.0 2023-11-23 01:31:07,420 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2300, loss[loss=0.06866, simple_loss=0.09202, pruned_loss=0.01424, audio_tagging_loss=0.008408, over 15846.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.0931, pruned_loss=0.01447, audio_tagging_loss=0.009257, over 3044092.41 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:31:10,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2179613.3333333335, ans=0.1 2023-11-23 01:31:11,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2179613.3333333335, ans=0.0 2023-11-23 01:31:14,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 326950 2023-11-23 01:31:24,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2179680.0, ans=0.125 2023-11-23 01:31:28,527 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.99 vs. limit=15.0 2023-11-23 01:31:44,820 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.250e+01 8.860e+01 9.394e+01 1.286e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 01:31:49,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2179813.3333333335, ans=0.0 2023-11-23 01:32:04,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2179880.0, ans=0.2 2023-11-23 01:32:05,569 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:32:11,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2179946.6666666665, ans=0.2 2023-11-23 01:32:12,379 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2350, loss[loss=0.07025, simple_loss=0.09385, pruned_loss=0.01482, audio_tagging_loss=0.008496, over 15020.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09348, pruned_loss=0.01451, audio_tagging_loss=0.009286, over 3042937.36 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:32:14,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2179946.6666666665, ans=0.0 2023-11-23 01:32:15,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2179946.6666666665, ans=0.125 2023-11-23 01:32:19,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327000 2023-11-23 01:32:36,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2180013.3333333335, ans=0.125 2023-11-23 01:33:17,549 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2400, loss[loss=0.08465, simple_loss=0.1115, pruned_loss=0.01982, audio_tagging_loss=0.0091, over 16077.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.0947, pruned_loss=0.01475, audio_tagging_loss=0.009355, over 3051392.62 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:33:25,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327050 2023-11-23 01:33:31,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2180346.6666666665, ans=0.1 2023-11-23 01:33:46,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2180413.3333333335, ans=0.2 2023-11-23 01:33:53,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2180413.3333333335, ans=0.125 2023-11-23 01:33:54,369 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.524e+01 9.128e+01 9.621e+01 1.327e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-23 01:34:04,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2180480.0, ans=0.0 2023-11-23 01:34:04,898 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=10.00 vs. limit=10.0 2023-11-23 01:34:21,273 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2450, loss[loss=0.08773, simple_loss=0.1194, pruned_loss=0.01779, audio_tagging_loss=0.01025, over 15693.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09346, pruned_loss=0.01454, audio_tagging_loss=0.009431, over 3047061.70 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:34:25,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2180613.3333333335, ans=0.025 2023-11-23 01:34:28,751 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327100 2023-11-23 01:34:39,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.82 vs. limit=10.0 2023-11-23 01:34:53,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2180746.6666666665, ans=0.125 2023-11-23 01:34:55,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2180746.6666666665, ans=0.05 2023-11-23 01:35:01,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2180813.3333333335, ans=0.125 2023-11-23 01:35:15,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2180880.0, ans=15.0 2023-11-23 01:35:18,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2180880.0, ans=0.125 2023-11-23 01:35:19,141 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.64 vs. limit=15.0 2023-11-23 01:35:21,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2180880.0, ans=0.125 2023-11-23 01:35:25,325 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2500, loss[loss=0.06963, simple_loss=0.08047, pruned_loss=0.01791, audio_tagging_loss=0.01148, over 14336.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09333, pruned_loss=0.01447, audio_tagging_loss=0.009421, over 3044684.69 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:35:33,325 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327150 2023-11-23 01:35:34,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2180946.6666666665, ans=0.2 2023-11-23 01:35:36,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.91 vs. limit=15.0 2023-11-23 01:36:02,401 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.294e+01 8.910e+01 9.572e+01 1.200e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 01:36:06,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2181146.6666666665, ans=0.125 2023-11-23 01:36:22,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2181213.3333333335, ans=0.125 2023-11-23 01:36:26,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2181213.3333333335, ans=0.125 2023-11-23 01:36:30,875 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2550, loss[loss=0.06568, simple_loss=0.08814, pruned_loss=0.01164, audio_tagging_loss=0.009971, over 15492.00 frames. ], tot_loss[loss=0.07, simple_loss=0.0926, pruned_loss=0.01434, audio_tagging_loss=0.009362, over 3043506.92 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:36:38,324 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327200 2023-11-23 01:36:43,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2181346.6666666665, ans=0.125 2023-11-23 01:36:54,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2181413.3333333335, ans=0.125 2023-11-23 01:36:57,602 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.24 vs. limit=12.0 2023-11-23 01:37:25,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2181546.6666666665, ans=0.1 2023-11-23 01:37:28,384 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.71 vs. limit=15.0 2023-11-23 01:37:32,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.32 vs. limit=22.5 2023-11-23 01:37:35,085 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2600, loss[loss=0.09787, simple_loss=0.1325, pruned_loss=0.02369, audio_tagging_loss=0.007946, over 15576.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09213, pruned_loss=0.01431, audio_tagging_loss=0.009312, over 3042761.72 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:37:36,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2181613.3333333335, ans=0.0 2023-11-23 01:37:42,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327250 2023-11-23 01:37:50,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2181680.0, ans=0.0 2023-11-23 01:38:09,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2181746.6666666665, ans=0.125 2023-11-23 01:38:13,509 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.227e+01 8.835e+01 9.553e+01 1.278e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 01:38:16,386 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:38:23,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2181813.3333333335, ans=0.0 2023-11-23 01:38:39,230 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2650, loss[loss=0.06896, simple_loss=0.0921, pruned_loss=0.01339, audio_tagging_loss=0.009522, over 15690.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09219, pruned_loss=0.01443, audio_tagging_loss=0.009246, over 3051916.24 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:38:47,297 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327300 2023-11-23 01:38:52,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2182013.3333333335, ans=0.2 2023-11-23 01:39:25,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2182146.6666666665, ans=0.125 2023-11-23 01:39:31,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2182213.3333333335, ans=0.0 2023-11-23 01:39:35,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.34 vs. limit=15.0 2023-11-23 01:39:41,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2182213.3333333335, ans=0.125 2023-11-23 01:39:44,319 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2700, loss[loss=0.05654, simple_loss=0.07368, pruned_loss=0.01102, audio_tagging_loss=0.008672, over 15554.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09153, pruned_loss=0.01426, audio_tagging_loss=0.009332, over 3057943.94 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:39:52,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327350 2023-11-23 01:40:21,077 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.147e+01 8.099e+01 8.608e+01 9.208e+01 1.380e+02, threshold=1.722e+02, percent-clipped=0.0 2023-11-23 01:40:28,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2182480.0, ans=0.0 2023-11-23 01:40:29,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2182480.0, ans=0.125 2023-11-23 01:40:48,633 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2750, loss[loss=0.07908, simple_loss=0.1085, pruned_loss=0.01744, audio_tagging_loss=0.007375, over 15056.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09265, pruned_loss=0.01424, audio_tagging_loss=0.009266, over 3058177.27 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:40:56,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327400 2023-11-23 01:41:02,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2182680.0, ans=0.1 2023-11-23 01:41:09,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2182680.0, ans=0.5 2023-11-23 01:41:18,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2182746.6666666665, ans=0.125 2023-11-23 01:41:21,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2182746.6666666665, ans=0.95 2023-11-23 01:41:28,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2182813.3333333335, ans=0.125 2023-11-23 01:41:44,323 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:41:52,833 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2800, loss[loss=0.07496, simple_loss=0.1058, pruned_loss=0.01302, audio_tagging_loss=0.009033, over 15827.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09198, pruned_loss=0.01425, audio_tagging_loss=0.009297, over 3056132.73 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:41:57,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2182946.6666666665, ans=0.125 2023-11-23 01:41:59,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2182946.6666666665, ans=0.0 2023-11-23 01:42:00,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327450 2023-11-23 01:42:31,837 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.504e+01 8.188e+01 8.710e+01 9.337e+01 1.091e+02, threshold=1.742e+02, percent-clipped=0.0 2023-11-23 01:42:53,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2183213.3333333335, ans=0.0 2023-11-23 01:42:56,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.51 vs. limit=15.0 2023-11-23 01:42:58,033 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2850, loss[loss=0.0892, simple_loss=0.1213, pruned_loss=0.02152, audio_tagging_loss=0.007014, over 14772.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09307, pruned_loss=0.01445, audio_tagging_loss=0.00916, over 3049231.59 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:42:59,997 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=15.0 2023-11-23 01:43:06,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327500 2023-11-23 01:43:12,039 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.11 vs. limit=15.0 2023-11-23 01:43:17,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2183346.6666666665, ans=0.2 2023-11-23 01:43:52,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2183546.6666666665, ans=0.125 2023-11-23 01:44:01,866 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2900, loss[loss=0.08274, simple_loss=0.1028, pruned_loss=0.01967, audio_tagging_loss=0.01165, over 14186.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09296, pruned_loss=0.01445, audio_tagging_loss=0.009202, over 3041764.08 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:44:05,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2183613.3333333335, ans=0.0 2023-11-23 01:44:09,342 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327550 2023-11-23 01:44:24,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2183680.0, ans=0.0 2023-11-23 01:44:31,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.99 vs. limit=15.0 2023-11-23 01:44:41,224 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.459e+01 9.104e+01 9.833e+01 1.241e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-23 01:44:45,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2183813.3333333335, ans=0.2 2023-11-23 01:44:49,274 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.93 vs. limit=15.0 2023-11-23 01:45:05,862 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 2950, loss[loss=0.04795, simple_loss=0.06365, pruned_loss=0.005324, audio_tagging_loss=0.0108, over 15429.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09294, pruned_loss=0.01434, audio_tagging_loss=0.009208, over 3037192.49 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:45:13,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327600 2023-11-23 01:45:39,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2184080.0, ans=0.07 2023-11-23 01:45:40,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2184080.0, ans=0.0 2023-11-23 01:46:03,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2184213.3333333335, ans=0.125 2023-11-23 01:46:04,708 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.40 vs. limit=15.0 2023-11-23 01:46:09,993 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3000, loss[loss=0.09429, simple_loss=0.1231, pruned_loss=0.02335, audio_tagging_loss=0.009398, over 16152.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09325, pruned_loss=0.01435, audio_tagging_loss=0.009237, over 3035249.73 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:46:09,994 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 01:46:50,800 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.2811, 4.9738, 4.6996, 5.1535], device='cuda:3') 2023-11-23 01:46:53,256 INFO [train_asr.py:1253] (3/4) Epoch 28, validation: loss=0.05807, simple_loss=0.05122, pruned_loss=0.005026, audio_tagging_loss=0.02744, over 4681554.00 frames. 2023-11-23 01:46:53,256 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 01:47:00,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327650 2023-11-23 01:47:01,238 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.41 vs. limit=6.0 2023-11-23 01:47:27,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2184413.3333333335, ans=0.125 2023-11-23 01:47:32,172 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.143e+01 8.821e+01 9.621e+01 1.257e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-23 01:47:33,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2184480.0, ans=0.1 2023-11-23 01:47:38,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.36 vs. limit=15.0 2023-11-23 01:47:42,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2184480.0, ans=0.2 2023-11-23 01:47:56,846 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3050, loss[loss=0.09163, simple_loss=0.122, pruned_loss=0.02234, audio_tagging_loss=0.008268, over 15530.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09372, pruned_loss=0.01455, audio_tagging_loss=0.00921, over 3043709.84 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:48:04,748 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327700 2023-11-23 01:48:06,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2184613.3333333335, ans=0.0 2023-11-23 01:48:09,791 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:48:14,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2184680.0, ans=0.2 2023-11-23 01:48:17,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2184680.0, ans=0.125 2023-11-23 01:48:27,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2184746.6666666665, ans=0.1 2023-11-23 01:48:35,720 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:48:53,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2184880.0, ans=0.0 2023-11-23 01:48:59,063 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.15 vs. limit=22.5 2023-11-23 01:49:01,425 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3100, loss[loss=0.08526, simple_loss=0.1171, pruned_loss=0.0202, audio_tagging_loss=0.006503, over 16374.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.09491, pruned_loss=0.01494, audio_tagging_loss=0.009256, over 3048503.18 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:49:09,497 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327750 2023-11-23 01:49:09,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2184946.6666666665, ans=0.1 2023-11-23 01:49:39,093 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.214e+01 8.129e+01 8.806e+01 9.628e+01 1.604e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-23 01:49:49,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2185146.6666666665, ans=0.125 2023-11-23 01:49:50,873 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.09 vs. limit=15.0 2023-11-23 01:49:51,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2185213.3333333335, ans=0.0 2023-11-23 01:49:57,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2185213.3333333335, ans=0.125 2023-11-23 01:49:58,437 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.95 vs. limit=22.5 2023-11-23 01:50:05,771 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3150, loss[loss=0.06777, simple_loss=0.08881, pruned_loss=0.01174, audio_tagging_loss=0.01162, over 16620.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09485, pruned_loss=0.01481, audio_tagging_loss=0.00935, over 3058284.18 frames. ], batch size: 63, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 01:50:11,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.79 vs. limit=15.0 2023-11-23 01:50:13,416 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327800 2023-11-23 01:50:13,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2185280.0, ans=0.07 2023-11-23 01:50:21,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2185346.6666666665, ans=0.125 2023-11-23 01:50:57,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.01 vs. limit=15.0 2023-11-23 01:51:05,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0 2023-11-23 01:51:09,980 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3200, loss[loss=0.08164, simple_loss=0.1041, pruned_loss=0.01936, audio_tagging_loss=0.01024, over 14857.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09564, pruned_loss=0.0148, audio_tagging_loss=0.009362, over 3058953.57 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:51:17,682 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327850 2023-11-23 01:51:19,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2185613.3333333335, ans=0.07 2023-11-23 01:51:20,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2185613.3333333335, ans=0.125 2023-11-23 01:51:24,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2185680.0, ans=0.125 2023-11-23 01:51:30,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2185680.0, ans=0.125 2023-11-23 01:51:49,573 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.239e+01 8.167e+01 8.814e+01 9.617e+01 1.193e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-23 01:52:01,354 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.24 vs. limit=22.5 2023-11-23 01:52:14,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0 2023-11-23 01:52:14,661 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3250, loss[loss=0.07253, simple_loss=0.09797, pruned_loss=0.01699, audio_tagging_loss=0.006561, over 14425.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09554, pruned_loss=0.01486, audio_tagging_loss=0.009426, over 3056210.27 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:52:20,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2185946.6666666665, ans=0.125 2023-11-23 01:52:23,243 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327900 2023-11-23 01:52:33,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2186013.3333333335, ans=0.05 2023-11-23 01:52:36,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2186013.3333333335, ans=0.125 2023-11-23 01:52:39,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2186080.0, ans=0.0 2023-11-23 01:52:46,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=2186080.0, ans=0.5 2023-11-23 01:52:51,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2186146.6666666665, ans=0.125 2023-11-23 01:53:00,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2186146.6666666665, ans=0.0 2023-11-23 01:53:11,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2186213.3333333335, ans=0.125 2023-11-23 01:53:18,586 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3300, loss[loss=0.07215, simple_loss=0.09075, pruned_loss=0.01785, audio_tagging_loss=0.008928, over 15978.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09431, pruned_loss=0.01466, audio_tagging_loss=0.009444, over 3049765.45 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:53:22,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2186280.0, ans=0.125 2023-11-23 01:53:22,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2186280.0, ans=0.0 2023-11-23 01:53:26,576 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 327950 2023-11-23 01:53:29,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2186280.0, ans=0.125 2023-11-23 01:53:36,816 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:53:46,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2186413.3333333335, ans=0.0 2023-11-23 01:53:47,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2186413.3333333335, ans=0.09899494936611666 2023-11-23 01:53:57,272 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.779e+01 8.400e+01 9.018e+01 9.669e+01 1.285e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 01:53:59,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2186480.0, ans=0.0 2023-11-23 01:54:09,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2186546.6666666665, ans=0.125 2023-11-23 01:54:16,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2186546.6666666665, ans=0.125 2023-11-23 01:54:16,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2186546.6666666665, ans=0.1 2023-11-23 01:54:22,759 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3350, loss[loss=0.09287, simple_loss=0.1221, pruned_loss=0.0223, audio_tagging_loss=0.009547, over 15798.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09382, pruned_loss=0.0146, audio_tagging_loss=0.009403, over 3057795.33 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:54:29,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2186613.3333333335, ans=0.0 2023-11-23 01:54:30,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328000 2023-11-23 01:54:54,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2186746.6666666665, ans=0.0 2023-11-23 01:55:06,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2186813.3333333335, ans=0.2 2023-11-23 01:55:15,880 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 01:55:18,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2186880.0, ans=0.125 2023-11-23 01:55:20,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0 2023-11-23 01:55:30,831 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3400, loss[loss=0.07082, simple_loss=0.0921, pruned_loss=0.01717, audio_tagging_loss=0.007601, over 14105.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09421, pruned_loss=0.01483, audio_tagging_loss=0.009314, over 3057995.85 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:55:31,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2186946.6666666665, ans=0.125 2023-11-23 01:55:32,911 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.49 vs. limit=6.0 2023-11-23 01:55:39,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328050 2023-11-23 01:55:44,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2187013.3333333335, ans=0.125 2023-11-23 01:55:53,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2187013.3333333335, ans=0.0 2023-11-23 01:56:00,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2187080.0, ans=0.2 2023-11-23 01:56:06,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.28 vs. limit=22.5 2023-11-23 01:56:09,477 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.501e+01 8.298e+01 8.922e+01 9.490e+01 1.312e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 01:56:35,856 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3450, loss[loss=0.09003, simple_loss=0.121, pruned_loss=0.02311, audio_tagging_loss=0.006402, over 14150.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09342, pruned_loss=0.0147, audio_tagging_loss=0.009259, over 3046566.96 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:56:43,035 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328100 2023-11-23 01:56:57,297 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.01 vs. limit=15.0 2023-11-23 01:57:02,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2187413.3333333335, ans=0.0 2023-11-23 01:57:25,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2187546.6666666665, ans=0.07 2023-11-23 01:57:29,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2187546.6666666665, ans=0.2 2023-11-23 01:57:39,726 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3500, loss[loss=0.08786, simple_loss=0.1191, pruned_loss=0.01807, audio_tagging_loss=0.01025, over 16285.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09373, pruned_loss=0.01472, audio_tagging_loss=0.009153, over 3044775.73 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:57:47,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328150 2023-11-23 01:58:05,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2187746.6666666665, ans=0.125 2023-11-23 01:58:13,412 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 01:58:18,276 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.297e+01 8.906e+01 9.538e+01 1.458e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-23 01:58:44,535 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3550, loss[loss=0.05305, simple_loss=0.05913, pruned_loss=0.01314, audio_tagging_loss=0.01034, over 14200.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09242, pruned_loss=0.0144, audio_tagging_loss=0.009145, over 3045413.72 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:58:53,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328200 2023-11-23 01:59:19,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2188080.0, ans=0.125 2023-11-23 01:59:34,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2188146.6666666665, ans=0.1 2023-11-23 01:59:44,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2188213.3333333335, ans=0.1 2023-11-23 01:59:48,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2188213.3333333335, ans=15.0 2023-11-23 01:59:50,141 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3600, loss[loss=0.09456, simple_loss=0.1209, pruned_loss=0.02168, audio_tagging_loss=0.01244, over 14267.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09335, pruned_loss=0.01459, audio_tagging_loss=0.00909, over 3046188.94 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 01:59:52,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2188280.0, ans=0.125 2023-11-23 01:59:54,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2188280.0, ans=0.0 2023-11-23 01:59:57,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328250 2023-11-23 01:59:57,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2188280.0, ans=0.1 2023-11-23 02:00:05,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2188346.6666666665, ans=0.0 2023-11-23 02:00:06,712 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.84 vs. limit=10.0 2023-11-23 02:00:08,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2188346.6666666665, ans=0.0 2023-11-23 02:00:15,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2188413.3333333335, ans=0.0 2023-11-23 02:00:16,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2188413.3333333335, ans=0.1 2023-11-23 02:00:28,190 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.210e+01 8.927e+01 9.741e+01 1.268e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 02:00:42,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2188546.6666666665, ans=0.0 2023-11-23 02:00:45,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2188546.6666666665, ans=0.125 2023-11-23 02:00:45,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2188546.6666666665, ans=0.125 2023-11-23 02:00:53,680 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3650, loss[loss=0.07532, simple_loss=0.09527, pruned_loss=0.01767, audio_tagging_loss=0.01001, over 14727.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09287, pruned_loss=0.01463, audio_tagging_loss=0.009084, over 3044685.59 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:01:00,890 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328300 2023-11-23 02:01:17,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2188680.0, ans=0.09899494936611666 2023-11-23 02:01:31,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2188813.3333333335, ans=0.125 2023-11-23 02:01:44,002 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.88 vs. limit=10.0 2023-11-23 02:01:57,307 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3700, loss[loss=0.05933, simple_loss=0.07825, pruned_loss=0.01165, audio_tagging_loss=0.008554, over 15341.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09223, pruned_loss=0.01455, audio_tagging_loss=0.009171, over 3051189.07 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:02:00,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2188946.6666666665, ans=0.125 2023-11-23 02:02:05,400 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328350 2023-11-23 02:02:17,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2189013.3333333335, ans=0.125 2023-11-23 02:02:37,558 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.324e+01 8.915e+01 9.653e+01 1.223e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 02:02:58,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2189213.3333333335, ans=0.125 2023-11-23 02:03:02,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2189280.0, ans=0.125 2023-11-23 02:03:02,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2189280.0, ans=0.125 2023-11-23 02:03:03,362 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3750, loss[loss=0.07054, simple_loss=0.08615, pruned_loss=0.01685, audio_tagging_loss=0.01062, over 15130.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.0932, pruned_loss=0.01466, audio_tagging_loss=0.009099, over 3054910.86 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:03:07,734 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.37 vs. limit=22.5 2023-11-23 02:03:10,866 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328400 2023-11-23 02:03:14,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2189346.6666666665, ans=0.125 2023-11-23 02:03:36,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.32 vs. limit=22.5 2023-11-23 02:03:38,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2189413.3333333335, ans=0.1 2023-11-23 02:03:48,593 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 02:04:06,824 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3800, loss[loss=0.08965, simple_loss=0.1251, pruned_loss=0.02061, audio_tagging_loss=0.006483, over 14725.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09348, pruned_loss=0.01469, audio_tagging_loss=0.009134, over 3060166.93 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:04:10,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2189613.3333333335, ans=0.0 2023-11-23 02:04:12,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2189613.3333333335, ans=0.125 2023-11-23 02:04:14,410 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328450 2023-11-23 02:04:23,461 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0 2023-11-23 02:04:47,089 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.455e+01 8.996e+01 9.813e+01 1.168e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 02:04:54,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2189813.3333333335, ans=0.125 2023-11-23 02:04:55,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2189813.3333333335, ans=0.125 2023-11-23 02:04:56,959 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:04:58,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2189880.0, ans=0.1 2023-11-23 02:05:06,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2189880.0, ans=0.125 2023-11-23 02:05:10,367 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3850, loss[loss=0.08104, simple_loss=0.1072, pruned_loss=0.0162, audio_tagging_loss=0.01125, over 15110.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09333, pruned_loss=0.01471, audio_tagging_loss=0.009245, over 3055211.34 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:05:19,154 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328500 2023-11-23 02:06:03,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2190213.3333333335, ans=0.1 2023-11-23 02:06:15,816 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3900, loss[loss=0.07982, simple_loss=0.1053, pruned_loss=0.01999, audio_tagging_loss=0.007185, over 15065.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09292, pruned_loss=0.01469, audio_tagging_loss=0.009262, over 3052858.10 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:06:23,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328550 2023-11-23 02:06:46,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2190413.3333333335, ans=0.0 2023-11-23 02:06:52,027 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.16 vs. limit=15.0 2023-11-23 02:06:53,556 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.052e+01 8.828e+01 9.660e+01 1.313e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 02:07:18,541 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 3950, loss[loss=0.06521, simple_loss=0.08052, pruned_loss=0.0121, audio_tagging_loss=0.01285, over 14035.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09265, pruned_loss=0.01453, audio_tagging_loss=0.009389, over 3045475.59 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:07:25,046 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:07:25,997 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328600 2023-11-23 02:07:28,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2190613.3333333335, ans=0.125 2023-11-23 02:07:44,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.45 vs. limit=22.5 2023-11-23 02:07:58,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2190813.3333333335, ans=0.125 2023-11-23 02:08:05,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2190813.3333333335, ans=0.0 2023-11-23 02:08:10,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2190880.0, ans=0.1 2023-11-23 02:08:13,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2190880.0, ans=0.0 2023-11-23 02:08:22,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2190946.6666666665, ans=0.2 2023-11-23 02:08:22,774 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4000, loss[loss=0.07898, simple_loss=0.1084, pruned_loss=0.01661, audio_tagging_loss=0.008176, over 16458.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.0934, pruned_loss=0.01471, audio_tagging_loss=0.009381, over 3055639.38 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:08:30,463 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328650 2023-11-23 02:08:38,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2191013.3333333335, ans=0.125 2023-11-23 02:08:41,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.08 vs. limit=15.0 2023-11-23 02:09:03,449 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.297e+01 8.975e+01 9.665e+01 1.273e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-23 02:09:05,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2191146.6666666665, ans=0.1 2023-11-23 02:09:18,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2191213.3333333335, ans=0.1 2023-11-23 02:09:21,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2191213.3333333335, ans=0.125 2023-11-23 02:09:28,281 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4050, loss[loss=0.05469, simple_loss=0.07091, pruned_loss=0.009832, audio_tagging_loss=0.009401, over 15231.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09313, pruned_loss=0.01456, audio_tagging_loss=0.009475, over 3058768.33 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:09:32,525 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 02:09:36,297 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328700 2023-11-23 02:09:40,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2191346.6666666665, ans=0.1 2023-11-23 02:09:45,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2191346.6666666665, ans=0.5 2023-11-23 02:09:47,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2191346.6666666665, ans=0.0 2023-11-23 02:09:54,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2191413.3333333335, ans=0.04949747468305833 2023-11-23 02:10:07,250 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:10:11,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2191480.0, ans=0.1 2023-11-23 02:10:14,299 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.75 vs. limit=22.5 2023-11-23 02:10:32,649 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4100, loss[loss=0.06694, simple_loss=0.08703, pruned_loss=0.0137, audio_tagging_loss=0.009732, over 16651.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09275, pruned_loss=0.01457, audio_tagging_loss=0.009478, over 3048797.82 frames. ], batch size: 62, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:10:40,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328750 2023-11-23 02:11:09,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.57 vs. limit=22.5 2023-11-23 02:11:13,912 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.198e+01 8.662e+01 9.320e+01 1.292e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-23 02:11:18,357 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.37 vs. limit=15.0 2023-11-23 02:11:26,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2191880.0, ans=0.0 2023-11-23 02:11:36,021 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4150, loss[loss=0.07061, simple_loss=0.09043, pruned_loss=0.01427, audio_tagging_loss=0.01112, over 15141.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09405, pruned_loss=0.01465, audio_tagging_loss=0.009287, over 3051545.13 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:11:43,602 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328800 2023-11-23 02:11:47,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2191946.6666666665, ans=0.2 2023-11-23 02:11:49,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2192013.3333333335, ans=0.1 2023-11-23 02:12:09,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2192080.0, ans=0.125 2023-11-23 02:12:10,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2192080.0, ans=0.125 2023-11-23 02:12:14,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2192146.6666666665, ans=0.125 2023-11-23 02:12:21,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2192146.6666666665, ans=0.0 2023-11-23 02:12:22,723 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 02:12:40,180 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4200, loss[loss=0.08051, simple_loss=0.09426, pruned_loss=0.02446, audio_tagging_loss=0.008923, over 15956.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.0928, pruned_loss=0.01452, audio_tagging_loss=0.00916, over 3056157.66 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:12:43,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2192280.0, ans=0.1 2023-11-23 02:12:43,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2192280.0, ans=0.0 2023-11-23 02:12:48,750 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328850 2023-11-23 02:12:59,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2192346.6666666665, ans=0.125 2023-11-23 02:13:07,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2192413.3333333335, ans=0.125 2023-11-23 02:13:08,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2192413.3333333335, ans=0.125 2023-11-23 02:13:20,633 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.473e+01 9.086e+01 9.924e+01 1.385e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-23 02:13:36,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2192546.6666666665, ans=0.2 2023-11-23 02:13:36,411 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.36 vs. limit=15.0 2023-11-23 02:13:39,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2192546.6666666665, ans=0.125 2023-11-23 02:13:44,950 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4250, loss[loss=0.06868, simple_loss=0.09292, pruned_loss=0.01443, audio_tagging_loss=0.007795, over 15702.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09437, pruned_loss=0.01481, audio_tagging_loss=0.008995, over 3054403.45 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:13:52,414 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328900 2023-11-23 02:14:04,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2192680.0, ans=0.125 2023-11-23 02:14:40,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2192880.0, ans=0.0 2023-11-23 02:14:48,994 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4300, loss[loss=0.05783, simple_loss=0.07119, pruned_loss=0.01194, audio_tagging_loss=0.01029, over 14128.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.0947, pruned_loss=0.01495, audio_tagging_loss=0.008967, over 3055252.92 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:14:54,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2192946.6666666665, ans=0.1 2023-11-23 02:14:56,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 328950 2023-11-23 02:14:56,682 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:15:03,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2193013.3333333335, ans=0.125 2023-11-23 02:15:11,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2193013.3333333335, ans=0.125 2023-11-23 02:15:11,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2193013.3333333335, ans=0.125 2023-11-23 02:15:20,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2193080.0, ans=0.125 2023-11-23 02:15:22,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.76 vs. limit=12.0 2023-11-23 02:15:30,846 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.649e+01 8.452e+01 8.978e+01 9.927e+01 1.263e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 02:15:53,671 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4350, loss[loss=0.07289, simple_loss=0.09356, pruned_loss=0.0165, audio_tagging_loss=0.009612, over 15151.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09484, pruned_loss=0.01496, audio_tagging_loss=0.00899, over 3051251.72 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 16.0 2023-11-23 02:16:02,610 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329000 2023-11-23 02:16:26,153 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.05 vs. limit=22.5 2023-11-23 02:16:26,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2193413.3333333335, ans=0.1 2023-11-23 02:16:58,989 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4400, loss[loss=0.0711, simple_loss=0.09501, pruned_loss=0.01403, audio_tagging_loss=0.009561, over 16623.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09394, pruned_loss=0.01463, audio_tagging_loss=0.009048, over 3054577.67 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:17:04,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2193613.3333333335, ans=0.5 2023-11-23 02:17:05,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2193613.3333333335, ans=0.0 2023-11-23 02:17:07,011 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329050 2023-11-23 02:17:07,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2193613.3333333335, ans=0.1 2023-11-23 02:17:18,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2193680.0, ans=0.0 2023-11-23 02:17:25,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2193746.6666666665, ans=0.1 2023-11-23 02:17:40,406 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.171e+01 8.943e+01 9.862e+01 1.169e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-23 02:17:54,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2193880.0, ans=0.0 2023-11-23 02:18:03,080 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4450, loss[loss=0.08584, simple_loss=0.1215, pruned_loss=0.01833, audio_tagging_loss=0.006755, over 15931.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09328, pruned_loss=0.01452, audio_tagging_loss=0.009002, over 3052275.21 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:18:10,636 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329100 2023-11-23 02:18:15,147 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.67 vs. limit=22.5 2023-11-23 02:18:42,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2194146.6666666665, ans=0.125 2023-11-23 02:19:00,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2194213.3333333335, ans=0.2 2023-11-23 02:19:01,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2194213.3333333335, ans=0.125 2023-11-23 02:19:07,664 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4500, loss[loss=0.08207, simple_loss=0.1158, pruned_loss=0.01721, audio_tagging_loss=0.006979, over 15364.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09304, pruned_loss=0.01445, audio_tagging_loss=0.008939, over 3052291.78 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:19:12,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2194280.0, ans=0.04949747468305833 2023-11-23 02:19:16,352 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329150 2023-11-23 02:19:29,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2194346.6666666665, ans=0.0 2023-11-23 02:19:44,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2194413.3333333335, ans=0.125 2023-11-23 02:19:48,965 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.124e+01 8.364e+01 9.070e+01 9.690e+01 1.146e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-23 02:19:50,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2194480.0, ans=0.125 2023-11-23 02:19:57,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.60 vs. limit=10.0 2023-11-23 02:19:58,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2194546.6666666665, ans=0.125 2023-11-23 02:20:02,322 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:20:09,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2194546.6666666665, ans=0.1 2023-11-23 02:20:09,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2194546.6666666665, ans=0.125 2023-11-23 02:20:13,044 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4550, loss[loss=0.06031, simple_loss=0.07383, pruned_loss=0.01385, audio_tagging_loss=0.009542, over 16059.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09324, pruned_loss=0.01446, audio_tagging_loss=0.008982, over 3054812.05 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0 2023-11-23 02:20:19,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2194613.3333333335, ans=0.125 2023-11-23 02:20:20,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329200 2023-11-23 02:20:20,478 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:20:31,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.99 vs. limit=10.0 2023-11-23 02:21:01,791 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 02:21:10,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2194880.0, ans=0.125 2023-11-23 02:21:12,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2194880.0, ans=0.1 2023-11-23 02:21:16,493 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4600, loss[loss=0.05157, simple_loss=0.0657, pruned_loss=0.009173, audio_tagging_loss=0.009542, over 14542.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09355, pruned_loss=0.01442, audio_tagging_loss=0.009066, over 3047382.66 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:21:23,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2194946.6666666665, ans=0.1 2023-11-23 02:21:24,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329250 2023-11-23 02:21:24,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2194946.6666666665, ans=0.125 2023-11-23 02:21:57,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2195146.6666666665, ans=0.125 2023-11-23 02:21:58,558 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.216e+01 8.878e+01 9.478e+01 1.349e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-23 02:22:21,465 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4650, loss[loss=0.07739, simple_loss=0.112, pruned_loss=0.01297, audio_tagging_loss=0.008403, over 15524.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09353, pruned_loss=0.0145, audio_tagging_loss=0.009139, over 3039747.16 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:22:29,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329300 2023-11-23 02:23:01,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2195480.0, ans=0.125 2023-11-23 02:23:27,340 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4700, loss[loss=0.08398, simple_loss=0.1022, pruned_loss=0.02419, audio_tagging_loss=0.008676, over 14569.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09426, pruned_loss=0.01483, audio_tagging_loss=0.009184, over 3044237.21 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:23:34,824 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329350 2023-11-23 02:23:56,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2195746.6666666665, ans=0.125 2023-11-23 02:24:09,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.291e+01 9.016e+01 9.707e+01 1.177e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-23 02:24:31,720 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4750, loss[loss=0.06169, simple_loss=0.08316, pruned_loss=0.009855, audio_tagging_loss=0.01025, over 15698.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09413, pruned_loss=0.01479, audio_tagging_loss=0.009245, over 3041252.48 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:24:39,007 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329400 2023-11-23 02:24:55,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2196013.3333333335, ans=0.125 2023-11-23 02:24:59,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2196080.0, ans=0.125 2023-11-23 02:25:07,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2196080.0, ans=0.05 2023-11-23 02:25:22,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2196146.6666666665, ans=0.0 2023-11-23 02:25:25,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2196213.3333333335, ans=0.0 2023-11-23 02:25:37,113 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4800, loss[loss=0.1009, simple_loss=0.1365, pruned_loss=0.02445, audio_tagging_loss=0.008224, over 14070.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09495, pruned_loss=0.01494, audio_tagging_loss=0.009191, over 3040190.42 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:25:44,701 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329450 2023-11-23 02:25:50,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2196346.6666666665, ans=0.2 2023-11-23 02:25:51,156 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:26:12,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2196413.3333333335, ans=0.0 2023-11-23 02:26:15,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.88 vs. limit=15.0 2023-11-23 02:26:19,706 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.219e+01 8.831e+01 9.548e+01 1.229e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 02:26:21,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.23 vs. limit=6.0 2023-11-23 02:26:28,281 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.88 vs. limit=15.0 2023-11-23 02:26:43,270 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4850, loss[loss=0.06239, simple_loss=0.07613, pruned_loss=0.01329, audio_tagging_loss=0.01104, over 14791.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09484, pruned_loss=0.01486, audio_tagging_loss=0.009276, over 3039028.04 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:26:50,756 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329500 2023-11-23 02:26:54,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2196680.0, ans=0.0 2023-11-23 02:26:55,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.37 vs. limit=15.0 2023-11-23 02:26:58,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2196680.0, ans=0.035 2023-11-23 02:27:00,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2196680.0, ans=0.125 2023-11-23 02:27:05,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2196680.0, ans=0.125 2023-11-23 02:27:19,070 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.98 vs. limit=15.0 2023-11-23 02:27:21,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2196813.3333333335, ans=0.125 2023-11-23 02:27:44,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2196880.0, ans=0.125 2023-11-23 02:27:47,943 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4900, loss[loss=0.07038, simple_loss=0.09377, pruned_loss=0.01506, audio_tagging_loss=0.008438, over 16157.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.0953, pruned_loss=0.01495, audio_tagging_loss=0.009176, over 3038634.15 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:27:55,524 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329550 2023-11-23 02:28:29,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2197146.6666666665, ans=0.125 2023-11-23 02:28:31,621 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.170e+01 8.747e+01 9.254e+01 1.120e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-23 02:28:35,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2197146.6666666665, ans=0.0 2023-11-23 02:28:42,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2197213.3333333335, ans=0.0 2023-11-23 02:28:50,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2197213.3333333335, ans=0.1 2023-11-23 02:28:52,720 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 4950, loss[loss=0.08792, simple_loss=0.1154, pruned_loss=0.02104, audio_tagging_loss=0.009176, over 14484.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09575, pruned_loss=0.01487, audio_tagging_loss=0.009059, over 3041314.53 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:29:00,751 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329600 2023-11-23 02:29:15,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2197346.6666666665, ans=0.125 2023-11-23 02:29:17,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2197346.6666666665, ans=0.2 2023-11-23 02:29:27,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2197413.3333333335, ans=0.125 2023-11-23 02:29:34,118 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:29:51,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2197546.6666666665, ans=0.05 2023-11-23 02:29:52,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2197546.6666666665, ans=0.04949747468305833 2023-11-23 02:29:59,521 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5000, loss[loss=0.06674, simple_loss=0.08793, pruned_loss=0.01377, audio_tagging_loss=0.009001, over 14774.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09616, pruned_loss=0.01479, audio_tagging_loss=0.008923, over 3042334.88 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:30:07,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329650 2023-11-23 02:30:15,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2197680.0, ans=0.125 2023-11-23 02:30:41,660 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.477e+01 9.069e+01 9.699e+01 1.131e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-23 02:30:51,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2197880.0, ans=0.125 2023-11-23 02:31:00,211 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.08 vs. limit=15.0 2023-11-23 02:31:04,438 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5050, loss[loss=0.06113, simple_loss=0.07693, pruned_loss=0.01263, audio_tagging_loss=0.01004, over 13702.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.0953, pruned_loss=0.0147, audio_tagging_loss=0.008907, over 3035735.91 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:31:11,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329700 2023-11-23 02:31:18,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2198013.3333333335, ans=0.125 2023-11-23 02:31:55,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2198213.3333333335, ans=0.125 2023-11-23 02:32:08,946 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5100, loss[loss=0.05117, simple_loss=0.05886, pruned_loss=0.0107, audio_tagging_loss=0.01104, over 14721.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09377, pruned_loss=0.01439, audio_tagging_loss=0.008977, over 3037233.72 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:32:16,351 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329750 2023-11-23 02:32:38,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2198413.3333333335, ans=0.125 2023-11-23 02:32:42,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2198413.3333333335, ans=0.09899494936611666 2023-11-23 02:32:51,755 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.231e+01 8.134e+01 8.796e+01 9.528e+01 1.100e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-23 02:32:52,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2198480.0, ans=0.2 2023-11-23 02:33:08,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0 2023-11-23 02:33:13,776 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5150, loss[loss=0.06963, simple_loss=0.09317, pruned_loss=0.01624, audio_tagging_loss=0.006798, over 15450.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.0932, pruned_loss=0.01415, audio_tagging_loss=0.009036, over 3036022.98 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:33:18,532 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.90 vs. limit=12.0 2023-11-23 02:33:23,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329800 2023-11-23 02:33:23,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2198613.3333333335, ans=0.1 2023-11-23 02:33:32,851 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.74 vs. limit=22.5 2023-11-23 02:34:10,589 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.86 vs. limit=10.0 2023-11-23 02:34:20,431 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5200, loss[loss=0.06758, simple_loss=0.09731, pruned_loss=0.01094, audio_tagging_loss=0.007978, over 15803.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09322, pruned_loss=0.01417, audio_tagging_loss=0.009023, over 3040642.35 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:34:27,883 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329850 2023-11-23 02:34:43,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2199013.3333333335, ans=0.125 2023-11-23 02:34:55,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2199080.0, ans=0.1 2023-11-23 02:35:05,348 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.387e+01 9.079e+01 9.668e+01 1.776e+02, threshold=1.816e+02, percent-clipped=1.0 2023-11-23 02:35:05,925 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.80 vs. limit=10.0 2023-11-23 02:35:09,330 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:35:25,320 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5250, loss[loss=0.06274, simple_loss=0.08447, pruned_loss=0.01092, audio_tagging_loss=0.009584, over 14190.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09309, pruned_loss=0.01418, audio_tagging_loss=0.008983, over 3048607.36 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:35:32,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329900 2023-11-23 02:36:11,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2199480.0, ans=0.025 2023-11-23 02:36:30,319 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5300, loss[loss=0.08693, simple_loss=0.1325, pruned_loss=0.01553, audio_tagging_loss=0.005127, over 15243.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09338, pruned_loss=0.01424, audio_tagging_loss=0.008868, over 3038281.90 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:36:39,682 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 329950 2023-11-23 02:36:46,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2199680.0, ans=0.2 2023-11-23 02:36:58,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.14 vs. limit=15.0 2023-11-23 02:37:02,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.49 vs. limit=15.0 2023-11-23 02:37:14,810 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.403e+01 8.848e+01 9.438e+01 1.127e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-23 02:37:17,795 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:37:22,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2199880.0, ans=0.0 2023-11-23 02:37:37,490 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5350, loss[loss=0.06904, simple_loss=0.09207, pruned_loss=0.01334, audio_tagging_loss=0.009673, over 15868.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09467, pruned_loss=0.01444, audio_tagging_loss=0.008825, over 3046013.78 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:37:44,377 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.49 vs. limit=10.0 2023-11-23 02:37:44,898 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330000 2023-11-23 02:38:24,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2200146.6666666665, ans=0.0 2023-11-23 02:38:26,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2200146.6666666665, ans=0.125 2023-11-23 02:38:42,218 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5400, loss[loss=0.08397, simple_loss=0.1145, pruned_loss=0.01603, audio_tagging_loss=0.01067, over 14296.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09526, pruned_loss=0.01458, audio_tagging_loss=0.008957, over 3045051.89 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:38:46,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2200280.0, ans=0.0 2023-11-23 02:38:49,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330050 2023-11-23 02:38:52,851 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.72 vs. limit=22.5 2023-11-23 02:38:56,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2200346.6666666665, ans=0.1 2023-11-23 02:38:59,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=2200346.6666666665, ans=0.2 2023-11-23 02:39:15,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2200413.3333333335, ans=0.0 2023-11-23 02:39:18,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2200413.3333333335, ans=0.1 2023-11-23 02:39:22,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.96 vs. limit=15.0 2023-11-23 02:39:27,025 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.068e+01 8.843e+01 9.502e+01 1.253e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 02:39:47,578 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5450, loss[loss=0.05509, simple_loss=0.06482, pruned_loss=0.01205, audio_tagging_loss=0.01063, over 14329.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09505, pruned_loss=0.01455, audio_tagging_loss=0.008987, over 3039643.89 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:39:56,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330100 2023-11-23 02:40:00,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2200680.0, ans=0.0 2023-11-23 02:40:15,864 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.41 vs. limit=15.0 2023-11-23 02:40:17,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2200746.6666666665, ans=0.2 2023-11-23 02:40:28,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2200813.3333333335, ans=0.5 2023-11-23 02:40:40,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2200880.0, ans=0.0 2023-11-23 02:40:43,490 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.06 vs. limit=12.0 2023-11-23 02:40:53,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2200946.6666666665, ans=0.125 2023-11-23 02:40:53,895 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5500, loss[loss=0.0791, simple_loss=0.1105, pruned_loss=0.01646, audio_tagging_loss=0.00738, over 17070.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09501, pruned_loss=0.01464, audio_tagging_loss=0.008963, over 3043395.42 frames. ], batch size: 62, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:40:55,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2200946.6666666665, ans=0.125 2023-11-23 02:41:01,943 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330150 2023-11-23 02:41:37,668 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.324e+01 8.941e+01 9.835e+01 1.232e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-23 02:41:40,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2201146.6666666665, ans=0.125 2023-11-23 02:41:43,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2201146.6666666665, ans=0.0 2023-11-23 02:41:44,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2201213.3333333335, ans=0.125 2023-11-23 02:41:50,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2201213.3333333335, ans=0.125 2023-11-23 02:41:57,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2201280.0, ans=0.125 2023-11-23 02:41:58,013 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5550, loss[loss=0.0526, simple_loss=0.06381, pruned_loss=0.008241, audio_tagging_loss=0.01246, over 14145.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09456, pruned_loss=0.01448, audio_tagging_loss=0.009174, over 3042706.25 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:42:05,579 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330200 2023-11-23 02:42:21,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2201346.6666666665, ans=0.1 2023-11-23 02:42:23,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2201413.3333333335, ans=0.125 2023-11-23 02:42:24,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2201413.3333333335, ans=0.2 2023-11-23 02:43:02,342 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5600, loss[loss=0.07108, simple_loss=0.0941, pruned_loss=0.01264, audio_tagging_loss=0.01139, over 14322.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09415, pruned_loss=0.01421, audio_tagging_loss=0.009251, over 3048236.48 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:43:10,563 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330250 2023-11-23 02:43:13,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2201613.3333333335, ans=0.2 2023-11-23 02:43:22,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2201680.0, ans=0.0 2023-11-23 02:43:25,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2201680.0, ans=0.125 2023-11-23 02:43:31,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2201746.6666666665, ans=15.0 2023-11-23 02:43:47,710 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.200e+01 8.820e+01 9.406e+01 1.283e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-23 02:43:49,110 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 02:43:50,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2201813.3333333335, ans=0.0 2023-11-23 02:43:55,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2201880.0, ans=0.125 2023-11-23 02:44:01,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2201880.0, ans=0.125 2023-11-23 02:44:08,236 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5650, loss[loss=0.05941, simple_loss=0.07247, pruned_loss=0.0134, audio_tagging_loss=0.009769, over 14697.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09441, pruned_loss=0.01434, audio_tagging_loss=0.009289, over 3049286.69 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:44:15,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330300 2023-11-23 02:44:23,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2202013.3333333335, ans=0.125 2023-11-23 02:44:41,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2202080.0, ans=0.95 2023-11-23 02:44:53,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.60 vs. limit=15.0 2023-11-23 02:45:04,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2202213.3333333335, ans=0.0 2023-11-23 02:45:06,133 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:45:11,977 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5700, loss[loss=0.07164, simple_loss=0.0938, pruned_loss=0.01695, audio_tagging_loss=0.007788, over 15187.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09319, pruned_loss=0.01424, audio_tagging_loss=0.009329, over 3059057.04 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:45:20,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330350 2023-11-23 02:45:52,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2202480.0, ans=0.125 2023-11-23 02:45:56,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2202480.0, ans=0.0 2023-11-23 02:45:57,499 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.129e+01 8.046e+01 8.620e+01 9.394e+01 1.276e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-23 02:45:57,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2202480.0, ans=10.0 2023-11-23 02:46:16,729 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5750, loss[loss=0.07688, simple_loss=0.09885, pruned_loss=0.01809, audio_tagging_loss=0.009365, over 14943.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09258, pruned_loss=0.01426, audio_tagging_loss=0.009239, over 3058419.84 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:46:19,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2202613.3333333335, ans=0.1 2023-11-23 02:46:19,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2202613.3333333335, ans=0.125 2023-11-23 02:46:24,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330400 2023-11-23 02:46:25,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2202613.3333333335, ans=0.125 2023-11-23 02:46:36,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2202680.0, ans=0.125 2023-11-23 02:46:50,862 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.63 vs. limit=22.5 2023-11-23 02:46:51,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2202746.6666666665, ans=10.0 2023-11-23 02:46:59,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2202813.3333333335, ans=0.125 2023-11-23 02:47:00,436 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.10 vs. limit=15.0 2023-11-23 02:47:07,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2202880.0, ans=0.1 2023-11-23 02:47:13,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2202880.0, ans=0.0 2023-11-23 02:47:22,025 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5800, loss[loss=0.0513, simple_loss=0.06413, pruned_loss=0.01005, audio_tagging_loss=0.009184, over 14898.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09283, pruned_loss=0.01432, audio_tagging_loss=0.009119, over 3047143.61 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:47:24,154 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.38 vs. limit=22.5 2023-11-23 02:47:29,598 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330450 2023-11-23 02:47:44,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2203013.3333333335, ans=0.0 2023-11-23 02:47:56,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2203080.0, ans=0.05 2023-11-23 02:47:57,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2203080.0, ans=0.0 2023-11-23 02:47:57,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2203080.0, ans=0.125 2023-11-23 02:47:59,360 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.95 vs. limit=22.5 2023-11-23 02:48:06,944 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.335e+01 8.418e+01 9.019e+01 9.630e+01 1.229e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 02:48:25,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2203280.0, ans=0.0 2023-11-23 02:48:26,336 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5850, loss[loss=0.05359, simple_loss=0.07485, pruned_loss=0.007103, audio_tagging_loss=0.009056, over 15299.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09383, pruned_loss=0.0144, audio_tagging_loss=0.009003, over 3040506.91 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:48:33,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330500 2023-11-23 02:48:47,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2203346.6666666665, ans=0.1 2023-11-23 02:49:16,991 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.27 vs. limit=15.0 2023-11-23 02:49:18,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.31 vs. limit=22.5 2023-11-23 02:49:30,673 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5900, loss[loss=0.07388, simple_loss=0.09455, pruned_loss=0.01862, audio_tagging_loss=0.007991, over 14318.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09338, pruned_loss=0.01432, audio_tagging_loss=0.008985, over 3038886.48 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:49:32,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2203613.3333333335, ans=0.0 2023-11-23 02:49:32,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.18 vs. limit=10.0 2023-11-23 02:49:38,201 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330550 2023-11-23 02:49:54,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2203680.0, ans=0.125 2023-11-23 02:50:15,292 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.273e+01 9.005e+01 9.559e+01 1.170e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 02:50:35,427 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 5950, loss[loss=0.06266, simple_loss=0.08388, pruned_loss=0.009285, audio_tagging_loss=0.01144, over 15013.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09307, pruned_loss=0.01424, audio_tagging_loss=0.009018, over 3044258.85 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 02:50:43,471 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330600 2023-11-23 02:50:52,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2204013.3333333335, ans=0.0 2023-11-23 02:51:06,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2204080.0, ans=0.125 2023-11-23 02:51:10,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2204080.0, ans=0.0 2023-11-23 02:51:17,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2204146.6666666665, ans=0.2 2023-11-23 02:51:29,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2204213.3333333335, ans=0.1 2023-11-23 02:51:40,723 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6000, loss[loss=0.05893, simple_loss=0.07958, pruned_loss=0.008201, audio_tagging_loss=0.01094, over 15320.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09291, pruned_loss=0.01421, audio_tagging_loss=0.008919, over 3044402.28 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:51:40,724 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 02:52:06,098 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.3165, 4.8133, 5.1930, 4.5719], device='cuda:3') 2023-11-23 02:52:24,759 INFO [train_asr.py:1253] (3/4) Epoch 28, validation: loss=0.05863, simple_loss=0.05128, pruned_loss=0.0051, audio_tagging_loss=0.02789, over 4681554.00 frames. 2023-11-23 02:52:24,760 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 02:52:33,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330650 2023-11-23 02:52:36,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2204346.6666666665, ans=0.125 2023-11-23 02:52:44,615 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.62 vs. limit=10.0 2023-11-23 02:53:06,922 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.52 vs. limit=12.0 2023-11-23 02:53:10,026 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.180e+01 8.677e+01 9.605e+01 1.280e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-23 02:53:11,380 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 02:53:18,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2204546.6666666665, ans=0.0 2023-11-23 02:53:25,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2204546.6666666665, ans=0.95 2023-11-23 02:53:30,849 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6050, loss[loss=0.06728, simple_loss=0.08479, pruned_loss=0.01408, audio_tagging_loss=0.01081, over 14081.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09286, pruned_loss=0.01418, audio_tagging_loss=0.008977, over 3042314.96 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:53:37,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2204613.3333333335, ans=0.2 2023-11-23 02:53:38,470 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330700 2023-11-23 02:53:41,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2204613.3333333335, ans=22.5 2023-11-23 02:53:52,636 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.73 vs. limit=15.0 2023-11-23 02:53:53,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2204680.0, ans=0.125 2023-11-23 02:54:15,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.68 vs. limit=22.5 2023-11-23 02:54:26,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2204880.0, ans=0.1 2023-11-23 02:54:31,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2204880.0, ans=0.125 2023-11-23 02:54:32,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2204880.0, ans=0.0 2023-11-23 02:54:35,080 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6100, loss[loss=0.0643, simple_loss=0.08537, pruned_loss=0.01011, audio_tagging_loss=0.01152, over 16523.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09314, pruned_loss=0.01425, audio_tagging_loss=0.009005, over 3042575.67 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:54:42,624 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330750 2023-11-23 02:54:44,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2204946.6666666665, ans=0.1 2023-11-23 02:55:04,499 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.17 vs. limit=15.0 2023-11-23 02:55:10,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.65 vs. limit=15.0 2023-11-23 02:55:21,020 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 7.997e+01 8.605e+01 9.181e+01 1.192e+02, threshold=1.721e+02, percent-clipped=0.0 2023-11-23 02:55:23,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2205146.6666666665, ans=0.1 2023-11-23 02:55:31,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2205213.3333333335, ans=0.0 2023-11-23 02:55:35,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2205213.3333333335, ans=0.125 2023-11-23 02:55:37,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2205213.3333333335, ans=0.125 2023-11-23 02:55:39,756 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6150, loss[loss=0.06853, simple_loss=0.09507, pruned_loss=0.01263, audio_tagging_loss=0.008363, over 16729.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09285, pruned_loss=0.01419, audio_tagging_loss=0.009084, over 3045395.57 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:55:48,266 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330800 2023-11-23 02:56:09,119 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 02:56:18,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2205480.0, ans=0.125 2023-11-23 02:56:44,804 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.70 vs. limit=15.0 2023-11-23 02:56:45,211 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6200, loss[loss=0.06469, simple_loss=0.07563, pruned_loss=0.01183, audio_tagging_loss=0.01505, over 15385.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09242, pruned_loss=0.01409, audio_tagging_loss=0.009197, over 3045994.82 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:56:53,741 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330850 2023-11-23 02:56:56,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2205613.3333333335, ans=0.05 2023-11-23 02:57:11,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0 2023-11-23 02:57:30,718 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.609e+01 8.249e+01 8.951e+01 9.781e+01 1.178e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-23 02:57:35,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2205813.3333333335, ans=0.125 2023-11-23 02:57:36,950 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.93 vs. limit=22.5 2023-11-23 02:57:48,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2205880.0, ans=0.5 2023-11-23 02:57:49,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2205946.6666666665, ans=0.2 2023-11-23 02:57:50,234 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6250, loss[loss=0.08391, simple_loss=0.1083, pruned_loss=0.02211, audio_tagging_loss=0.007636, over 15466.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09268, pruned_loss=0.0142, audio_tagging_loss=0.009294, over 3046957.19 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:57:53,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.08 vs. limit=10.0 2023-11-23 02:57:57,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330900 2023-11-23 02:58:07,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2206013.3333333335, ans=0.2 2023-11-23 02:58:12,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2206013.3333333335, ans=0.0 2023-11-23 02:58:17,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2206080.0, ans=0.0 2023-11-23 02:58:37,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2206146.6666666665, ans=0.0 2023-11-23 02:58:39,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2206146.6666666665, ans=0.125 2023-11-23 02:58:54,315 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6300, loss[loss=0.07046, simple_loss=0.1006, pruned_loss=0.01051, audio_tagging_loss=0.009645, over 16206.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09292, pruned_loss=0.01413, audio_tagging_loss=0.009294, over 3052443.20 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 02:58:57,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2206280.0, ans=0.125 2023-11-23 02:59:01,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 330950 2023-11-23 02:59:23,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2206413.3333333335, ans=0.04949747468305833 2023-11-23 02:59:26,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2206413.3333333335, ans=0.125 2023-11-23 02:59:40,246 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.187e+01 8.713e+01 9.379e+01 1.255e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-23 02:59:54,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2206546.6666666665, ans=0.125 2023-11-23 02:59:59,685 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6350, loss[loss=0.06159, simple_loss=0.09102, pruned_loss=0.01052, audio_tagging_loss=0.005559, over 15451.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09208, pruned_loss=0.014, audio_tagging_loss=0.009345, over 3050858.97 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 03:00:00,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.31 vs. limit=15.0 2023-11-23 03:00:08,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331000 2023-11-23 03:00:21,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2206680.0, ans=0.125 2023-11-23 03:00:23,529 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.62 vs. limit=22.5 2023-11-23 03:00:24,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2206680.0, ans=0.125 2023-11-23 03:00:25,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2206746.6666666665, ans=0.0 2023-11-23 03:00:49,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2206813.3333333335, ans=0.2 2023-11-23 03:00:56,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2206880.0, ans=0.0 2023-11-23 03:01:04,034 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.66 vs. limit=15.0 2023-11-23 03:01:05,841 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6400, loss[loss=0.0753, simple_loss=0.1041, pruned_loss=0.01496, audio_tagging_loss=0.008269, over 15295.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09191, pruned_loss=0.01397, audio_tagging_loss=0.009418, over 3050661.41 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 03:01:13,431 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331050 2023-11-23 03:01:14,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2206946.6666666665, ans=15.0 2023-11-23 03:01:18,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2207013.3333333335, ans=0.125 2023-11-23 03:01:52,584 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.045e+01 8.704e+01 9.425e+01 1.351e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-23 03:01:52,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2207146.6666666665, ans=0.125 2023-11-23 03:02:09,987 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6450, loss[loss=0.07724, simple_loss=0.09839, pruned_loss=0.01741, audio_tagging_loss=0.01064, over 15491.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09031, pruned_loss=0.01363, audio_tagging_loss=0.009581, over 3040357.11 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:02:17,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331100 2023-11-23 03:02:24,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2207346.6666666665, ans=0.125 2023-11-23 03:02:24,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2207346.6666666665, ans=0.2 2023-11-23 03:02:41,639 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.86 vs. limit=15.0 2023-11-23 03:02:45,649 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.64 vs. limit=8.0 2023-11-23 03:03:15,320 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6500, loss[loss=0.05542, simple_loss=0.07572, pruned_loss=0.006666, audio_tagging_loss=0.0109, over 15524.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09035, pruned_loss=0.01377, audio_tagging_loss=0.009428, over 3046136.67 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:03:23,424 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331150 2023-11-23 03:03:34,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2207680.0, ans=0.0 2023-11-23 03:03:46,998 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:03:51,053 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.59 vs. limit=15.0 2023-11-23 03:03:55,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2207813.3333333335, ans=0.125 2023-11-23 03:03:57,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2207813.3333333335, ans=0.0 2023-11-23 03:04:01,376 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 8.162e+01 8.865e+01 9.552e+01 1.149e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-23 03:04:03,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2207813.3333333335, ans=0.0 2023-11-23 03:04:08,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2207880.0, ans=0.1 2023-11-23 03:04:10,136 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.56 vs. limit=15.0 2023-11-23 03:04:21,244 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6550, loss[loss=0.04815, simple_loss=0.06997, pruned_loss=0.006849, audio_tagging_loss=0.006321, over 14734.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09036, pruned_loss=0.01386, audio_tagging_loss=0.009233, over 3044796.87 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:04:21,831 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0 2023-11-23 03:04:28,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331200 2023-11-23 03:04:48,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2208080.0, ans=0.125 2023-11-23 03:05:09,496 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:05:13,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2208213.3333333335, ans=0.1 2023-11-23 03:05:14,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2208213.3333333335, ans=0.0 2023-11-23 03:05:25,298 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6600, loss[loss=0.06223, simple_loss=0.07913, pruned_loss=0.0136, audio_tagging_loss=0.009062, over 16727.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09124, pruned_loss=0.0141, audio_tagging_loss=0.009095, over 3037431.39 frames. ], batch size: 64, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:05:25,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2208280.0, ans=0.125 2023-11-23 03:05:31,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2208280.0, ans=0.125 2023-11-23 03:05:32,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331250 2023-11-23 03:05:33,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2208280.0, ans=0.125 2023-11-23 03:05:36,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2208346.6666666665, ans=0.2 2023-11-23 03:05:39,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2208346.6666666665, ans=0.1 2023-11-23 03:05:48,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2208346.6666666665, ans=0.0 2023-11-23 03:05:48,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2208346.6666666665, ans=0.2 2023-11-23 03:05:52,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.26 vs. limit=15.0 2023-11-23 03:06:02,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2208413.3333333335, ans=0.125 2023-11-23 03:06:11,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2208480.0, ans=0.1 2023-11-23 03:06:12,274 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.179e+01 8.913e+01 9.440e+01 1.641e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 03:06:29,556 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6650, loss[loss=0.0736, simple_loss=0.09616, pruned_loss=0.01609, audio_tagging_loss=0.00944, over 14956.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09108, pruned_loss=0.014, audio_tagging_loss=0.009131, over 3042494.11 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:06:33,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0 2023-11-23 03:06:37,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331300 2023-11-23 03:06:42,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2208680.0, ans=0.1 2023-11-23 03:06:49,655 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.17 vs. limit=15.0 2023-11-23 03:06:58,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2208746.6666666665, ans=0.125 2023-11-23 03:07:08,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2208813.3333333335, ans=0.125 2023-11-23 03:07:09,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2208813.3333333335, ans=0.0 2023-11-23 03:07:14,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.04 vs. limit=22.5 2023-11-23 03:07:34,619 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6700, loss[loss=0.05814, simple_loss=0.07674, pruned_loss=0.01178, audio_tagging_loss=0.007993, over 14145.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09129, pruned_loss=0.01404, audio_tagging_loss=0.009099, over 3045277.00 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:07:43,174 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331350 2023-11-23 03:07:43,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2208946.6666666665, ans=0.1 2023-11-23 03:07:45,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2208946.6666666665, ans=0.2 2023-11-23 03:08:22,310 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.251e+01 8.844e+01 9.540e+01 1.359e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 03:08:28,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2209213.3333333335, ans=0.0 2023-11-23 03:08:40,203 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6750, loss[loss=0.08205, simple_loss=0.1103, pruned_loss=0.0172, audio_tagging_loss=0.009711, over 14809.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09184, pruned_loss=0.01426, audio_tagging_loss=0.009036, over 3042197.52 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:08:44,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2209280.0, ans=0.05 2023-11-23 03:08:47,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331400 2023-11-23 03:09:23,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2209480.0, ans=0.125 2023-11-23 03:09:27,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2209480.0, ans=0.125 2023-11-23 03:09:41,577 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:09:45,066 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6800, loss[loss=0.05875, simple_loss=0.07638, pruned_loss=0.008722, audio_tagging_loss=0.01183, over 16123.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09272, pruned_loss=0.01421, audio_tagging_loss=0.008991, over 3042912.48 frames. ], batch size: 62, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 03:09:47,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=2209613.3333333335, ans=15.0 2023-11-23 03:09:53,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331450 2023-11-23 03:10:15,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2209746.6666666665, ans=0.125 2023-11-23 03:10:17,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2209746.6666666665, ans=0.0 2023-11-23 03:10:22,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2209746.6666666665, ans=0.125 2023-11-23 03:10:31,116 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:10:31,993 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.723e+01 8.090e+01 8.741e+01 9.523e+01 1.182e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-23 03:10:34,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.26 vs. limit=15.0 2023-11-23 03:10:40,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.77 vs. limit=15.0 2023-11-23 03:10:46,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2209880.0, ans=0.0 2023-11-23 03:10:51,126 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6850, loss[loss=0.06553, simple_loss=0.08305, pruned_loss=0.01652, audio_tagging_loss=0.007481, over 14871.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09357, pruned_loss=0.01446, audio_tagging_loss=0.008912, over 3043104.06 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 03:10:58,577 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331500 2023-11-23 03:11:01,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.55 vs. limit=15.0 2023-11-23 03:11:20,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2210080.0, ans=0.2 2023-11-23 03:11:34,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2210146.6666666665, ans=0.125 2023-11-23 03:11:38,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2210146.6666666665, ans=0.125 2023-11-23 03:11:42,178 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.00 vs. limit=22.5 2023-11-23 03:11:50,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2210213.3333333335, ans=0.035 2023-11-23 03:11:56,056 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6900, loss[loss=0.07286, simple_loss=0.1002, pruned_loss=0.0161, audio_tagging_loss=0.006673, over 15469.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09285, pruned_loss=0.01434, audio_tagging_loss=0.008892, over 3035267.80 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:12:03,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331550 2023-11-23 03:12:03,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2210280.0, ans=0.125 2023-11-23 03:12:43,544 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.094e+01 8.690e+01 9.402e+01 1.184e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-23 03:12:44,874 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 03:13:00,466 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 6950, loss[loss=0.06801, simple_loss=0.0937, pruned_loss=0.0113, audio_tagging_loss=0.009859, over 15502.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09383, pruned_loss=0.01458, audio_tagging_loss=0.008945, over 3039491.96 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:13:01,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2210613.3333333335, ans=0.1 2023-11-23 03:13:02,474 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.46 vs. limit=10.0 2023-11-23 03:13:04,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2210613.3333333335, ans=0.125 2023-11-23 03:13:08,045 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331600 2023-11-23 03:13:25,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2210680.0, ans=0.125 2023-11-23 03:13:29,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2210746.6666666665, ans=0.125 2023-11-23 03:13:48,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2210813.3333333335, ans=0.125 2023-11-23 03:13:50,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2210813.3333333335, ans=0.0 2023-11-23 03:14:06,064 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7000, loss[loss=0.06533, simple_loss=0.07373, pruned_loss=0.01635, audio_tagging_loss=0.01211, over 15461.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09369, pruned_loss=0.01449, audio_tagging_loss=0.009044, over 3040943.39 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:14:06,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2210946.6666666665, ans=0.125 2023-11-23 03:14:14,245 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331650 2023-11-23 03:14:17,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2210946.6666666665, ans=0.0 2023-11-23 03:14:53,597 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.328e+01 8.294e+01 8.963e+01 9.673e+01 1.557e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-23 03:15:01,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2211213.3333333335, ans=0.125 2023-11-23 03:15:10,765 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7050, loss[loss=0.04919, simple_loss=0.06358, pruned_loss=0.007674, audio_tagging_loss=0.009724, over 15297.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09369, pruned_loss=0.0146, audio_tagging_loss=0.009228, over 3046801.13 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:15:18,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331700 2023-11-23 03:15:28,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2211346.6666666665, ans=0.0 2023-11-23 03:15:28,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2211346.6666666665, ans=0.125 2023-11-23 03:15:52,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2211480.0, ans=0.05 2023-11-23 03:16:02,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2211546.6666666665, ans=0.0 2023-11-23 03:16:13,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2211613.3333333335, ans=0.125 2023-11-23 03:16:14,858 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7100, loss[loss=0.05239, simple_loss=0.06007, pruned_loss=0.0101, audio_tagging_loss=0.01225, over 14391.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09302, pruned_loss=0.01436, audio_tagging_loss=0.009332, over 3051183.81 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:16:20,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2211613.3333333335, ans=0.07 2023-11-23 03:16:22,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331750 2023-11-23 03:16:25,898 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.21 vs. limit=15.0 2023-11-23 03:17:02,517 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.317e+01 8.441e+01 9.062e+01 9.663e+01 1.297e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-23 03:17:08,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2211880.0, ans=0.125 2023-11-23 03:17:09,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2211880.0, ans=0.125 2023-11-23 03:17:15,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2211880.0, ans=0.125 2023-11-23 03:17:19,056 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7150, loss[loss=0.05073, simple_loss=0.06042, pruned_loss=0.009293, audio_tagging_loss=0.01123, over 14830.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09364, pruned_loss=0.01456, audio_tagging_loss=0.009283, over 3048800.56 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0 2023-11-23 03:17:26,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331800 2023-11-23 03:17:38,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2212013.3333333335, ans=15.0 2023-11-23 03:17:39,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2212013.3333333335, ans=0.125 2023-11-23 03:17:56,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2212146.6666666665, ans=0.1 2023-11-23 03:18:19,129 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:18:22,456 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7200, loss[loss=0.08356, simple_loss=0.1198, pruned_loss=0.01628, audio_tagging_loss=0.007377, over 16198.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09439, pruned_loss=0.01464, audio_tagging_loss=0.009253, over 3051707.44 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 03:18:29,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331850 2023-11-23 03:18:56,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2212413.3333333335, ans=0.125 2023-11-23 03:19:02,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2212480.0, ans=0.0 2023-11-23 03:19:08,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.73 vs. limit=15.0 2023-11-23 03:19:08,961 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.113e+01 8.957e+01 9.713e+01 1.213e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 03:19:23,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2212613.3333333335, ans=0.1 2023-11-23 03:19:24,863 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7250, loss[loss=0.05925, simple_loss=0.07912, pruned_loss=0.01045, audio_tagging_loss=0.009235, over 15108.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09359, pruned_loss=0.01449, audio_tagging_loss=0.009418, over 3050394.74 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0 2023-11-23 03:19:32,950 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331900 2023-11-23 03:19:34,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2212613.3333333335, ans=0.1 2023-11-23 03:20:08,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2212813.3333333335, ans=0.2 2023-11-23 03:20:13,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2212813.3333333335, ans=0.125 2023-11-23 03:20:29,121 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7300, loss[loss=0.06146, simple_loss=0.0792, pruned_loss=0.01291, audio_tagging_loss=0.008951, over 15161.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09334, pruned_loss=0.01439, audio_tagging_loss=0.009272, over 3047691.23 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:20:36,932 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 331950 2023-11-23 03:20:38,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2212946.6666666665, ans=0.125 2023-11-23 03:20:40,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2213013.3333333335, ans=0.125 2023-11-23 03:20:51,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2213013.3333333335, ans=0.07 2023-11-23 03:21:15,974 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.147e+01 8.723e+01 9.402e+01 1.286e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-23 03:21:18,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2213213.3333333335, ans=0.125 2023-11-23 03:21:33,054 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7350, loss[loss=0.06902, simple_loss=0.09285, pruned_loss=0.01331, audio_tagging_loss=0.009281, over 15923.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09303, pruned_loss=0.01434, audio_tagging_loss=0.009133, over 3049626.45 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:21:40,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332000 2023-11-23 03:21:46,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2213280.0, ans=0.2 2023-11-23 03:21:47,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2213346.6666666665, ans=0.125 2023-11-23 03:21:51,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=2213346.6666666665, ans=0.02 2023-11-23 03:21:56,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2213346.6666666665, ans=0.1 2023-11-23 03:22:39,823 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7400, loss[loss=0.08421, simple_loss=0.1073, pruned_loss=0.02073, audio_tagging_loss=0.009828, over 15235.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09273, pruned_loss=0.01419, audio_tagging_loss=0.009137, over 3050569.28 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:22:47,298 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332050 2023-11-23 03:23:02,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2213680.0, ans=0.0 2023-11-23 03:23:05,540 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:23:12,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2213746.6666666665, ans=0.0 2023-11-23 03:23:21,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2213813.3333333335, ans=0.125 2023-11-23 03:23:27,011 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.31 vs. limit=15.0 2023-11-23 03:23:28,622 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.451e+01 8.955e+01 9.673e+01 1.193e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 03:23:28,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2213813.3333333335, ans=0.125 2023-11-23 03:23:44,657 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7450, loss[loss=0.06256, simple_loss=0.07869, pruned_loss=0.01333, audio_tagging_loss=0.009883, over 15144.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09315, pruned_loss=0.01449, audio_tagging_loss=0.009107, over 3046418.51 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:23:52,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332100 2023-11-23 03:24:30,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2214146.6666666665, ans=0.125 2023-11-23 03:24:49,007 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7500, loss[loss=0.06665, simple_loss=0.08624, pruned_loss=0.01351, audio_tagging_loss=0.01002, over 15404.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09319, pruned_loss=0.01445, audio_tagging_loss=0.008997, over 3056526.65 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:24:56,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332150 2023-11-23 03:25:18,741 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:25:24,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2214413.3333333335, ans=0.5 2023-11-23 03:25:28,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2214480.0, ans=0.1 2023-11-23 03:25:30,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2214480.0, ans=0.0 2023-11-23 03:25:37,860 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.184e+01 8.938e+01 9.763e+01 1.268e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-23 03:25:52,773 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7550, loss[loss=0.07398, simple_loss=0.09051, pruned_loss=0.01729, audio_tagging_loss=0.01144, over 15130.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09297, pruned_loss=0.01453, audio_tagging_loss=0.008972, over 3049136.50 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:25:56,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2214613.3333333335, ans=0.0 2023-11-23 03:26:00,072 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332200 2023-11-23 03:26:02,083 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.58 vs. limit=15.0 2023-11-23 03:26:55,772 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7600, loss[loss=0.05344, simple_loss=0.06634, pruned_loss=0.01057, audio_tagging_loss=0.009697, over 15159.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09207, pruned_loss=0.01432, audio_tagging_loss=0.009084, over 3058292.61 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:27:04,428 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332250 2023-11-23 03:27:13,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2215013.3333333335, ans=0.0 2023-11-23 03:27:13,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2215013.3333333335, ans=0.125 2023-11-23 03:27:26,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2215080.0, ans=0.1 2023-11-23 03:27:26,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.63 vs. limit=15.0 2023-11-23 03:27:27,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2215080.0, ans=0.125 2023-11-23 03:27:32,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2215080.0, ans=0.125 2023-11-23 03:27:33,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2215146.6666666665, ans=0.1 2023-11-23 03:27:33,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2215146.6666666665, ans=0.125 2023-11-23 03:27:34,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0 2023-11-23 03:27:34,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2215146.6666666665, ans=0.0 2023-11-23 03:27:36,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2215146.6666666665, ans=0.125 2023-11-23 03:27:42,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2215146.6666666665, ans=0.125 2023-11-23 03:27:44,364 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.025e+01 8.532e+01 9.208e+01 1.132e+02, threshold=1.706e+02, percent-clipped=0.0 2023-11-23 03:28:01,616 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7650, loss[loss=0.06487, simple_loss=0.07731, pruned_loss=0.01186, audio_tagging_loss=0.01436, over 16314.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09228, pruned_loss=0.01421, audio_tagging_loss=0.00908, over 3056583.39 frames. ], batch size: 63, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:28:09,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332300 2023-11-23 03:28:43,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2215480.0, ans=0.2 2023-11-23 03:28:50,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2215480.0, ans=0.0 2023-11-23 03:29:01,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2215546.6666666665, ans=0.2 2023-11-23 03:29:05,137 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7700, loss[loss=0.05817, simple_loss=0.07138, pruned_loss=0.01274, audio_tagging_loss=0.009741, over 14005.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09286, pruned_loss=0.01423, audio_tagging_loss=0.009048, over 3050483.28 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:29:11,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2215613.3333333335, ans=0.09899494936611666 2023-11-23 03:29:12,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332350 2023-11-23 03:29:16,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2215680.0, ans=0.1 2023-11-23 03:29:55,234 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.060e+01 8.592e+01 9.630e+01 1.129e+02, threshold=1.718e+02, percent-clipped=0.0 2023-11-23 03:30:08,631 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7750, loss[loss=0.09646, simple_loss=0.1305, pruned_loss=0.02441, audio_tagging_loss=0.006821, over 14733.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09317, pruned_loss=0.01431, audio_tagging_loss=0.008982, over 3049243.50 frames. ], batch size: 53, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:30:08,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2215946.6666666665, ans=0.125 2023-11-23 03:30:16,641 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332400 2023-11-23 03:30:17,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=22.5 2023-11-23 03:30:24,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2216013.3333333335, ans=0.0 2023-11-23 03:30:25,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2216013.3333333335, ans=0.0 2023-11-23 03:30:55,770 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.32 vs. limit=22.5 2023-11-23 03:31:14,411 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7800, loss[loss=0.08234, simple_loss=0.1111, pruned_loss=0.01634, audio_tagging_loss=0.01045, over 14037.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09397, pruned_loss=0.01437, audio_tagging_loss=0.008987, over 3046317.44 frames. ], batch size: 52, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:31:21,775 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332450 2023-11-23 03:31:21,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2216280.0, ans=0.1 2023-11-23 03:31:22,622 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.46 vs. limit=15.0 2023-11-23 03:31:32,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2216346.6666666665, ans=0.125 2023-11-23 03:31:39,097 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:31:49,660 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.59 vs. limit=15.0 2023-11-23 03:32:04,598 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.076e+01 8.334e+01 8.987e+01 9.736e+01 1.242e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 03:32:18,094 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7850, loss[loss=0.05957, simple_loss=0.07695, pruned_loss=0.0131, audio_tagging_loss=0.007999, over 14194.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09392, pruned_loss=0.01437, audio_tagging_loss=0.009041, over 3040297.79 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:32:20,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2216613.3333333335, ans=0.07 2023-11-23 03:32:25,690 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332500 2023-11-23 03:33:02,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2216813.3333333335, ans=0.125 2023-11-23 03:33:18,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2216880.0, ans=0.1 2023-11-23 03:33:20,002 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.25 vs. limit=15.0 2023-11-23 03:33:21,823 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7900, loss[loss=0.07822, simple_loss=0.1043, pruned_loss=0.01828, audio_tagging_loss=0.007787, over 16576.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09449, pruned_loss=0.01457, audio_tagging_loss=0.009162, over 3042655.90 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:33:29,931 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332550 2023-11-23 03:33:56,697 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.31 vs. limit=15.0 2023-11-23 03:34:09,687 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.15 vs. limit=15.0 2023-11-23 03:34:11,385 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.317e+01 8.969e+01 9.609e+01 1.150e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-23 03:34:22,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2217213.3333333335, ans=0.125 2023-11-23 03:34:26,051 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 7950, loss[loss=0.04304, simple_loss=0.05446, pruned_loss=0.006906, audio_tagging_loss=0.008909, over 14404.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.0937, pruned_loss=0.01453, audio_tagging_loss=0.0093, over 3045943.36 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:34:33,843 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332600 2023-11-23 03:34:39,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2217346.6666666665, ans=0.1 2023-11-23 03:34:40,838 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 03:34:57,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2217413.3333333335, ans=0.125 2023-11-23 03:34:58,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2217413.3333333335, ans=0.125 2023-11-23 03:35:30,611 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8000, loss[loss=0.0734, simple_loss=0.09761, pruned_loss=0.01607, audio_tagging_loss=0.008525, over 15295.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09305, pruned_loss=0.01449, audio_tagging_loss=0.00945, over 3037947.75 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:35:37,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332650 2023-11-23 03:35:58,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2217746.6666666665, ans=0.05 2023-11-23 03:35:58,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2217746.6666666665, ans=0.125 2023-11-23 03:36:07,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2217813.3333333335, ans=0.05 2023-11-23 03:36:19,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.05 vs. limit=15.0 2023-11-23 03:36:20,158 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.081e+01 8.720e+01 9.413e+01 1.910e+02, threshold=1.744e+02, percent-clipped=1.0 2023-11-23 03:36:25,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.49 vs. limit=22.5 2023-11-23 03:36:27,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2217880.0, ans=0.2 2023-11-23 03:36:33,486 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8050, loss[loss=0.07959, simple_loss=0.1034, pruned_loss=0.01782, audio_tagging_loss=0.01009, over 13995.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09244, pruned_loss=0.01441, audio_tagging_loss=0.009571, over 3036440.06 frames. ], batch size: 53, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:36:40,748 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332700 2023-11-23 03:37:04,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2218080.0, ans=0.1 2023-11-23 03:37:27,585 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.38 vs. limit=22.5 2023-11-23 03:37:36,543 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.08 vs. limit=10.0 2023-11-23 03:37:37,083 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8100, loss[loss=0.06444, simple_loss=0.08804, pruned_loss=0.01263, audio_tagging_loss=0.007793, over 14967.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09233, pruned_loss=0.01435, audio_tagging_loss=0.009489, over 3034199.55 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:37:45,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332750 2023-11-23 03:38:06,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2218413.3333333335, ans=0.0 2023-11-23 03:38:24,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff3.min_abs, batch_count=2218480.0, ans=0.2 2023-11-23 03:38:27,683 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.342e+01 8.867e+01 9.602e+01 1.203e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-23 03:38:29,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2218546.6666666665, ans=0.2 2023-11-23 03:38:40,761 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8150, loss[loss=0.07104, simple_loss=0.09256, pruned_loss=0.01439, audio_tagging_loss=0.01036, over 15343.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09261, pruned_loss=0.01444, audio_tagging_loss=0.009332, over 3037387.21 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:38:48,861 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332800 2023-11-23 03:38:55,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2218680.0, ans=0.125 2023-11-23 03:38:58,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2218680.0, ans=0.125 2023-11-23 03:39:10,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2218746.6666666665, ans=0.125 2023-11-23 03:39:22,402 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.87 vs. limit=22.5 2023-11-23 03:39:45,652 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8200, loss[loss=0.07461, simple_loss=0.1105, pruned_loss=0.0126, audio_tagging_loss=0.006762, over 16079.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09227, pruned_loss=0.01424, audio_tagging_loss=0.009299, over 3044910.07 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:39:45,688 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 03:39:47,461 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.26 vs. limit=10.0 2023-11-23 03:39:50,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2218946.6666666665, ans=0.125 2023-11-23 03:39:53,167 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332850 2023-11-23 03:39:58,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2219013.3333333335, ans=0.0 2023-11-23 03:40:11,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.20 vs. limit=22.5 2023-11-23 03:40:37,033 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.334e+01 8.200e+01 8.804e+01 9.476e+01 1.277e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-23 03:40:44,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2219213.3333333335, ans=0.05 2023-11-23 03:40:49,915 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8250, loss[loss=0.0729, simple_loss=0.08916, pruned_loss=0.01909, audio_tagging_loss=0.009222, over 14316.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.0921, pruned_loss=0.0142, audio_tagging_loss=0.009278, over 3046193.51 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:40:58,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332900 2023-11-23 03:40:58,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2219280.0, ans=0.1 2023-11-23 03:41:35,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2219480.0, ans=0.125 2023-11-23 03:41:38,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2219480.0, ans=0.125 2023-11-23 03:41:54,272 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8300, loss[loss=0.05281, simple_loss=0.06604, pruned_loss=0.008207, audio_tagging_loss=0.01158, over 14228.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09202, pruned_loss=0.0141, audio_tagging_loss=0.00923, over 3050002.92 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:41:54,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2219613.3333333335, ans=0.0 2023-11-23 03:42:01,602 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 332950 2023-11-23 03:42:17,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2219680.0, ans=0.0 2023-11-23 03:42:18,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=2219746.6666666665, ans=0.02 2023-11-23 03:42:19,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2219746.6666666665, ans=0.0 2023-11-23 03:42:25,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.94 vs. limit=15.0 2023-11-23 03:42:43,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2219813.3333333335, ans=0.2 2023-11-23 03:42:44,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2219880.0, ans=0.0 2023-11-23 03:42:46,803 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.213e+01 8.531e+01 9.088e+01 9.532e+01 2.273e+02, threshold=1.818e+02, percent-clipped=2.0 2023-11-23 03:42:57,837 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8350, loss[loss=0.05498, simple_loss=0.07508, pruned_loss=0.009673, audio_tagging_loss=0.007762, over 14561.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09321, pruned_loss=0.01435, audio_tagging_loss=0.009038, over 3048723.99 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 8.0 2023-11-23 03:43:05,996 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333000 2023-11-23 03:43:19,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2220013.3333333335, ans=0.125 2023-11-23 03:43:29,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2220080.0, ans=0.0 2023-11-23 03:43:41,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2220146.6666666665, ans=0.2 2023-11-23 03:43:50,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2220213.3333333335, ans=0.0 2023-11-23 03:44:03,098 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8400, loss[loss=0.06108, simple_loss=0.08418, pruned_loss=0.009033, audio_tagging_loss=0.009953, over 16077.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09328, pruned_loss=0.01428, audio_tagging_loss=0.009054, over 3051066.53 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:44:10,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333050 2023-11-23 03:44:14,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2220280.0, ans=0.1 2023-11-23 03:44:19,469 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.63 vs. limit=15.0 2023-11-23 03:44:35,580 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.37 vs. limit=15.0 2023-11-23 03:44:37,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2220413.3333333335, ans=0.025 2023-11-23 03:44:55,758 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 7.977e+01 8.728e+01 9.553e+01 1.214e+02, threshold=1.746e+02, percent-clipped=0.0 2023-11-23 03:45:04,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2220546.6666666665, ans=0.125 2023-11-23 03:45:07,861 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8450, loss[loss=0.06801, simple_loss=0.1034, pruned_loss=0.009193, audio_tagging_loss=0.007102, over 15254.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09251, pruned_loss=0.01428, audio_tagging_loss=0.00912, over 3048426.29 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:45:15,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333100 2023-11-23 03:45:29,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.19 vs. limit=15.0 2023-11-23 03:45:44,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2220746.6666666665, ans=0.1 2023-11-23 03:45:44,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.40 vs. limit=15.0 2023-11-23 03:46:12,225 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8500, loss[loss=0.05937, simple_loss=0.06975, pruned_loss=0.0154, audio_tagging_loss=0.009094, over 14568.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.092, pruned_loss=0.01428, audio_tagging_loss=0.009155, over 3045021.30 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:46:18,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2220946.6666666665, ans=0.125 2023-11-23 03:46:19,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333150 2023-11-23 03:46:19,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2220946.6666666665, ans=0.09899494936611666 2023-11-23 03:46:53,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.64 vs. limit=6.0 2023-11-23 03:47:01,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2221213.3333333335, ans=0.1 2023-11-23 03:47:03,975 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.153e+01 8.814e+01 9.449e+01 1.976e+02, threshold=1.763e+02, percent-clipped=1.0 2023-11-23 03:47:15,507 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8550, loss[loss=0.09047, simple_loss=0.1288, pruned_loss=0.0176, audio_tagging_loss=0.008471, over 16209.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09322, pruned_loss=0.01456, audio_tagging_loss=0.009105, over 3045858.34 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:47:23,699 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333200 2023-11-23 03:47:23,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2221280.0, ans=0.1 2023-11-23 03:47:30,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2221346.6666666665, ans=0.2 2023-11-23 03:47:37,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.62 vs. limit=15.0 2023-11-23 03:47:49,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2221413.3333333335, ans=0.125 2023-11-23 03:47:57,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2221480.0, ans=0.1 2023-11-23 03:48:16,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2221546.6666666665, ans=0.1 2023-11-23 03:48:18,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2221546.6666666665, ans=0.1 2023-11-23 03:48:20,343 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8600, loss[loss=0.08126, simple_loss=0.1119, pruned_loss=0.01732, audio_tagging_loss=0.007973, over 15499.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09295, pruned_loss=0.01442, audio_tagging_loss=0.009169, over 3052855.51 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:48:20,740 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:48:26,988 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0 2023-11-23 03:48:27,584 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333250 2023-11-23 03:48:30,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2221613.3333333335, ans=0.0 2023-11-23 03:48:37,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2221680.0, ans=0.125 2023-11-23 03:48:38,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2221680.0, ans=0.2 2023-11-23 03:49:10,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2221880.0, ans=0.1 2023-11-23 03:49:12,198 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.558e+01 9.096e+01 9.715e+01 2.705e+02, threshold=1.819e+02, percent-clipped=1.0 2023-11-23 03:49:17,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2221880.0, ans=22.5 2023-11-23 03:49:22,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2221946.6666666665, ans=0.125 2023-11-23 03:49:23,133 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8650, loss[loss=0.06749, simple_loss=0.09939, pruned_loss=0.01114, audio_tagging_loss=0.006658, over 15535.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09305, pruned_loss=0.01441, audio_tagging_loss=0.009299, over 3051440.10 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:49:30,616 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333300 2023-11-23 03:49:40,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.38 vs. limit=22.5 2023-11-23 03:49:56,850 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 03:49:58,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2222080.0, ans=0.1 2023-11-23 03:50:12,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0 2023-11-23 03:50:24,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2222280.0, ans=0.09899494936611666 2023-11-23 03:50:25,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.70 vs. limit=15.0 2023-11-23 03:50:25,868 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8700, loss[loss=0.06162, simple_loss=0.07452, pruned_loss=0.01166, audio_tagging_loss=0.01271, over 15099.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09243, pruned_loss=0.0143, audio_tagging_loss=0.00942, over 3050861.29 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:50:26,636 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.78 vs. limit=15.0 2023-11-23 03:50:34,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333350 2023-11-23 03:50:52,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2222413.3333333335, ans=0.125 2023-11-23 03:50:59,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2222413.3333333335, ans=0.125 2023-11-23 03:51:17,205 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.257e+01 8.537e+01 9.054e+01 9.922e+01 1.181e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-23 03:51:28,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2222613.3333333335, ans=0.2 2023-11-23 03:51:30,137 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8750, loss[loss=0.06486, simple_loss=0.08139, pruned_loss=0.01384, audio_tagging_loss=0.01033, over 14056.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09301, pruned_loss=0.01443, audio_tagging_loss=0.009426, over 3040811.69 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:51:32,078 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.74 vs. limit=10.0 2023-11-23 03:51:34,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2222613.3333333335, ans=0.0 2023-11-23 03:51:37,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333400 2023-11-23 03:52:09,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2222813.3333333335, ans=0.125 2023-11-23 03:52:32,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2222946.6666666665, ans=0.125 2023-11-23 03:52:33,781 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8800, loss[loss=0.06246, simple_loss=0.08977, pruned_loss=0.01076, audio_tagging_loss=0.006814, over 15447.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09211, pruned_loss=0.01414, audio_tagging_loss=0.009597, over 3044095.96 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:52:41,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333450 2023-11-23 03:52:51,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2223013.3333333335, ans=0.125 2023-11-23 03:53:10,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2223080.0, ans=0.0 2023-11-23 03:53:13,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2223146.6666666665, ans=0.125 2023-11-23 03:53:19,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2223146.6666666665, ans=0.125 2023-11-23 03:53:25,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2223213.3333333335, ans=0.07 2023-11-23 03:53:26,100 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.173e+01 8.235e+01 8.890e+01 9.571e+01 2.057e+02, threshold=1.778e+02, percent-clipped=1.0 2023-11-23 03:53:26,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2223213.3333333335, ans=0.0 2023-11-23 03:53:37,233 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8850, loss[loss=0.06139, simple_loss=0.0905, pruned_loss=0.007359, audio_tagging_loss=0.008777, over 15805.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09203, pruned_loss=0.01404, audio_tagging_loss=0.009558, over 3039894.66 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:53:45,281 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333500 2023-11-23 03:53:48,751 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 03:53:51,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2223346.6666666665, ans=0.125 2023-11-23 03:53:59,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.82 vs. limit=22.5 2023-11-23 03:54:31,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2223546.6666666665, ans=0.0 2023-11-23 03:54:33,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2223546.6666666665, ans=0.125 2023-11-23 03:54:37,866 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=9.45 vs. limit=12.0 2023-11-23 03:54:40,992 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8900, loss[loss=0.09683, simple_loss=0.1246, pruned_loss=0.02352, audio_tagging_loss=0.01103, over 15590.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09272, pruned_loss=0.01431, audio_tagging_loss=0.009396, over 3037851.02 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:54:41,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2223613.3333333335, ans=0.125 2023-11-23 03:54:50,380 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333550 2023-11-23 03:54:50,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2223613.3333333335, ans=0.1 2023-11-23 03:54:59,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2223680.0, ans=0.125 2023-11-23 03:55:25,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0 2023-11-23 03:55:34,578 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.471e+01 9.014e+01 9.931e+01 1.268e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-23 03:55:38,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2223880.0, ans=0.125 2023-11-23 03:55:39,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2223880.0, ans=0.0 2023-11-23 03:55:46,447 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 8950, loss[loss=0.07432, simple_loss=0.1072, pruned_loss=0.01355, audio_tagging_loss=0.007173, over 14806.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09279, pruned_loss=0.01426, audio_tagging_loss=0.009215, over 3048145.61 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:55:53,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333600 2023-11-23 03:56:25,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2224146.6666666665, ans=0.125 2023-11-23 03:56:49,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2224280.0, ans=0.1 2023-11-23 03:56:50,597 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9000, loss[loss=0.05817, simple_loss=0.07919, pruned_loss=0.006049, audio_tagging_loss=0.01253, over 14993.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09339, pruned_loss=0.01433, audio_tagging_loss=0.00911, over 3050489.97 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 03:56:50,598 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 03:57:32,315 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.0603, 2.5715, 2.7559, 2.6828], device='cuda:3') 2023-11-23 03:57:33,762 INFO [train_asr.py:1253] (3/4) Epoch 28, validation: loss=0.05919, simple_loss=0.05113, pruned_loss=0.004978, audio_tagging_loss=0.02865, over 4681554.00 frames. 2023-11-23 03:57:33,762 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 03:57:41,881 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333650 2023-11-23 03:58:11,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2224480.0, ans=0.0 2023-11-23 03:58:28,169 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.337e+01 8.839e+01 9.721e+01 1.226e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-23 03:58:37,997 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9050, loss[loss=0.07367, simple_loss=0.1038, pruned_loss=0.01239, audio_tagging_loss=0.009377, over 16250.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09379, pruned_loss=0.01445, audio_tagging_loss=0.009131, over 3049614.28 frames. ], batch size: 61, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:58:45,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333700 2023-11-23 03:59:10,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2224746.6666666665, ans=0.125 2023-11-23 03:59:15,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2224813.3333333335, ans=0.125 2023-11-23 03:59:28,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2224880.0, ans=0.1 2023-11-23 03:59:31,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2224880.0, ans=0.2 2023-11-23 03:59:35,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2224880.0, ans=0.125 2023-11-23 03:59:40,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2224946.6666666665, ans=0.1 2023-11-23 03:59:41,506 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9100, loss[loss=0.06387, simple_loss=0.08119, pruned_loss=0.01074, audio_tagging_loss=0.01254, over 15995.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09389, pruned_loss=0.01436, audio_tagging_loss=0.00903, over 3052569.37 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 03:59:49,518 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333750 2023-11-23 03:59:55,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2225013.3333333335, ans=0.0 2023-11-23 04:00:29,658 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=22.5 2023-11-23 04:00:34,988 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.085e+01 8.276e+01 8.928e+01 9.613e+01 1.240e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-23 04:00:43,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2225213.3333333335, ans=0.0 2023-11-23 04:00:46,299 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9150, loss[loss=0.06323, simple_loss=0.08744, pruned_loss=0.01348, audio_tagging_loss=0.006032, over 16046.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09377, pruned_loss=0.01438, audio_tagging_loss=0.009045, over 3047356.70 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:00:54,089 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333800 2023-11-23 04:01:00,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2225346.6666666665, ans=0.0 2023-11-23 04:01:07,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2225346.6666666665, ans=0.07 2023-11-23 04:01:12,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2225413.3333333335, ans=0.2 2023-11-23 04:01:13,131 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=15.0 2023-11-23 04:01:14,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2225413.3333333335, ans=0.125 2023-11-23 04:01:50,727 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9200, loss[loss=0.07225, simple_loss=0.1012, pruned_loss=0.01334, audio_tagging_loss=0.008325, over 15400.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09352, pruned_loss=0.01432, audio_tagging_loss=0.009075, over 3046912.08 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:01:58,106 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333850 2023-11-23 04:02:13,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.42 vs. limit=15.0 2023-11-23 04:02:38,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2225813.3333333335, ans=0.1 2023-11-23 04:02:44,369 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.331e+01 8.795e+01 9.315e+01 1.180e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-23 04:02:53,983 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9250, loss[loss=0.06027, simple_loss=0.0764, pruned_loss=0.01142, audio_tagging_loss=0.01065, over 15410.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09308, pruned_loss=0.01442, audio_tagging_loss=0.00909, over 3052221.37 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:03:01,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333900 2023-11-23 04:03:12,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2226013.3333333335, ans=0.125 2023-11-23 04:03:13,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.91 vs. limit=15.0 2023-11-23 04:03:21,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2226080.0, ans=0.125 2023-11-23 04:03:23,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2226080.0, ans=0.2 2023-11-23 04:03:40,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2226146.6666666665, ans=0.1 2023-11-23 04:03:57,694 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.86 vs. limit=12.0 2023-11-23 04:03:58,098 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9300, loss[loss=0.09224, simple_loss=0.1295, pruned_loss=0.01971, audio_tagging_loss=0.007808, over 15617.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09331, pruned_loss=0.01448, audio_tagging_loss=0.009053, over 3049049.64 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:04:06,705 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 333950 2023-11-23 04:04:39,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2226480.0, ans=0.1 2023-11-23 04:04:52,644 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.259e+01 8.847e+01 9.648e+01 1.420e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 04:04:58,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2226546.6666666665, ans=15.0 2023-11-23 04:04:58,084 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.07 vs. limit=15.0 2023-11-23 04:05:01,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2226613.3333333335, ans=0.125 2023-11-23 04:05:02,495 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9350, loss[loss=0.06659, simple_loss=0.09281, pruned_loss=0.0127, audio_tagging_loss=0.007489, over 15298.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09289, pruned_loss=0.01433, audio_tagging_loss=0.009178, over 3046854.55 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:05:04,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2226613.3333333335, ans=0.0 2023-11-23 04:05:10,425 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334000 2023-11-23 04:05:31,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.40 vs. limit=15.0 2023-11-23 04:05:38,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2226746.6666666665, ans=0.1 2023-11-23 04:05:51,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2226813.3333333335, ans=0.1 2023-11-23 04:05:53,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2226880.0, ans=0.0 2023-11-23 04:06:00,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2226880.0, ans=0.125 2023-11-23 04:06:06,814 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9400, loss[loss=0.0688, simple_loss=0.08952, pruned_loss=0.01645, audio_tagging_loss=0.007591, over 16346.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09178, pruned_loss=0.01421, audio_tagging_loss=0.009306, over 3043909.05 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:06:14,282 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334050 2023-11-23 04:06:14,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2226946.6666666665, ans=0.125 2023-11-23 04:06:18,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2227013.3333333335, ans=0.2 2023-11-23 04:06:24,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2227013.3333333335, ans=0.0 2023-11-23 04:06:34,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2227080.0, ans=0.1 2023-11-23 04:06:47,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2227146.6666666665, ans=0.2 2023-11-23 04:07:00,201 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.797e+01 8.264e+01 9.005e+01 9.544e+01 1.394e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 04:07:08,140 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:07:08,470 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:07:10,616 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9450, loss[loss=0.08539, simple_loss=0.1156, pruned_loss=0.02112, audio_tagging_loss=0.006451, over 16580.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09228, pruned_loss=0.01421, audio_tagging_loss=0.009358, over 3047598.16 frames. ], batch size: 61, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:07:13,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2227280.0, ans=0.125 2023-11-23 04:07:13,524 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.60 vs. limit=15.0 2023-11-23 04:07:18,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334100 2023-11-23 04:07:58,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2227480.0, ans=10.0 2023-11-23 04:08:04,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2227546.6666666665, ans=0.0 2023-11-23 04:08:06,188 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:08:06,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.19 vs. limit=22.5 2023-11-23 04:08:13,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2227546.6666666665, ans=0.125 2023-11-23 04:08:15,167 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9500, loss[loss=0.05791, simple_loss=0.07652, pruned_loss=0.01147, audio_tagging_loss=0.008183, over 15879.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09109, pruned_loss=0.01401, audio_tagging_loss=0.009434, over 3046283.07 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:08:22,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334150 2023-11-23 04:08:27,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2227680.0, ans=0.125 2023-11-23 04:08:37,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2227680.0, ans=0.125 2023-11-23 04:08:39,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2227746.6666666665, ans=0.125 2023-11-23 04:08:49,189 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.89 vs. limit=15.0 2023-11-23 04:08:51,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2227746.6666666665, ans=0.125 2023-11-23 04:09:05,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2227880.0, ans=0.1 2023-11-23 04:09:08,995 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.198e+01 8.793e+01 9.435e+01 1.623e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-23 04:09:09,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.50 vs. limit=10.0 2023-11-23 04:09:19,595 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9550, loss[loss=0.06866, simple_loss=0.08812, pruned_loss=0.01438, audio_tagging_loss=0.01023, over 14597.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09222, pruned_loss=0.0142, audio_tagging_loss=0.009428, over 3047708.22 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:09:25,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2227946.6666666665, ans=0.125 2023-11-23 04:09:26,968 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334200 2023-11-23 04:09:33,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2228013.3333333335, ans=0.1 2023-11-23 04:09:43,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2228013.3333333335, ans=0.0 2023-11-23 04:09:59,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2228146.6666666665, ans=0.1 2023-11-23 04:10:14,288 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0 2023-11-23 04:10:23,926 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9600, loss[loss=0.09386, simple_loss=0.1376, pruned_loss=0.01952, audio_tagging_loss=0.00556, over 16708.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.0922, pruned_loss=0.01427, audio_tagging_loss=0.009404, over 3049286.50 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:10:31,939 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334250 2023-11-23 04:10:36,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.88 vs. limit=15.0 2023-11-23 04:10:42,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2228346.6666666665, ans=0.05 2023-11-23 04:10:42,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2228346.6666666665, ans=0.0 2023-11-23 04:10:53,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.24 vs. limit=22.5 2023-11-23 04:11:16,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2228546.6666666665, ans=0.09899494936611666 2023-11-23 04:11:17,372 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.386e+01 9.005e+01 9.855e+01 1.264e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 04:11:17,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2228546.6666666665, ans=0.125 2023-11-23 04:11:24,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=2228546.6666666665, ans=0.1 2023-11-23 04:11:27,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.82 vs. limit=22.5 2023-11-23 04:11:28,320 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9650, loss[loss=0.06609, simple_loss=0.07537, pruned_loss=0.01514, audio_tagging_loss=0.01327, over 14479.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09151, pruned_loss=0.01425, audio_tagging_loss=0.009424, over 3048697.87 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:11:33,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2228613.3333333335, ans=0.125 2023-11-23 04:11:35,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334300 2023-11-23 04:11:54,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2228746.6666666665, ans=0.125 2023-11-23 04:12:26,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2228880.0, ans=0.125 2023-11-23 04:12:28,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2228880.0, ans=0.125 2023-11-23 04:12:29,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2228880.0, ans=0.125 2023-11-23 04:12:31,996 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9700, loss[loss=0.06078, simple_loss=0.08403, pruned_loss=0.009022, audio_tagging_loss=0.009743, over 14427.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09191, pruned_loss=0.0142, audio_tagging_loss=0.009351, over 3050902.02 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:12:39,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334350 2023-11-23 04:12:54,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2229013.3333333335, ans=0.125 2023-11-23 04:13:03,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2229080.0, ans=0.125 2023-11-23 04:13:27,065 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.979e+01 8.051e+01 8.815e+01 9.310e+01 1.256e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-23 04:13:34,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2229213.3333333335, ans=0.0 2023-11-23 04:13:36,194 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9750, loss[loss=0.05316, simple_loss=0.06942, pruned_loss=0.009455, audio_tagging_loss=0.008996, over 15256.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09171, pruned_loss=0.01402, audio_tagging_loss=0.009246, over 3047961.33 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:13:38,009 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0 2023-11-23 04:13:44,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334400 2023-11-23 04:13:48,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2229346.6666666665, ans=0.125 2023-11-23 04:13:51,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2229346.6666666665, ans=0.0 2023-11-23 04:14:01,511 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.45 vs. limit=22.5 2023-11-23 04:14:39,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2229546.6666666665, ans=0.125 2023-11-23 04:14:41,489 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9800, loss[loss=0.05783, simple_loss=0.07825, pruned_loss=0.008995, audio_tagging_loss=0.009712, over 14476.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09272, pruned_loss=0.01427, audio_tagging_loss=0.009099, over 3043955.00 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:14:45,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0 2023-11-23 04:14:48,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334450 2023-11-23 04:14:54,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2229680.0, ans=0.125 2023-11-23 04:15:02,015 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.64 vs. limit=22.5 2023-11-23 04:15:36,217 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.857e+01 8.531e+01 9.197e+01 9.757e+01 1.250e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-23 04:15:37,519 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:15:38,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2229880.0, ans=0.125 2023-11-23 04:15:45,003 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9850, loss[loss=0.08472, simple_loss=0.1222, pruned_loss=0.01468, audio_tagging_loss=0.00895, over 15728.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09403, pruned_loss=0.01458, audio_tagging_loss=0.009048, over 3038260.08 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:15:49,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.87 vs. limit=22.5 2023-11-23 04:15:51,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2229946.6666666665, ans=0.1 2023-11-23 04:15:52,432 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334500 2023-11-23 04:16:42,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2230213.3333333335, ans=0.125 2023-11-23 04:16:46,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2230213.3333333335, ans=0.125 2023-11-23 04:16:48,577 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9900, loss[loss=0.06162, simple_loss=0.08306, pruned_loss=0.01007, audio_tagging_loss=0.01002, over 15796.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.0945, pruned_loss=0.01461, audio_tagging_loss=0.008971, over 3039510.69 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:16:51,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2230280.0, ans=0.125 2023-11-23 04:16:57,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334550 2023-11-23 04:17:44,121 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.266e+01 8.858e+01 9.665e+01 1.141e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 04:17:53,601 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 9950, loss[loss=0.04959, simple_loss=0.05731, pruned_loss=0.008472, audio_tagging_loss=0.01246, over 14381.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09371, pruned_loss=0.01444, audio_tagging_loss=0.008942, over 3039146.11 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0 2023-11-23 04:17:53,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2230613.3333333335, ans=0.125 2023-11-23 04:18:00,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334600 2023-11-23 04:18:20,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2230746.6666666665, ans=0.5 2023-11-23 04:18:27,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2230746.6666666665, ans=0.2 2023-11-23 04:18:42,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2230813.3333333335, ans=0.2 2023-11-23 04:18:57,019 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10000, loss[loss=0.04954, simple_loss=0.05885, pruned_loss=0.01006, audio_tagging_loss=0.01005, over 14639.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09356, pruned_loss=0.01443, audio_tagging_loss=0.008938, over 3040870.26 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0 2023-11-23 04:19:01,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.11 vs. limit=6.0 2023-11-23 04:19:02,127 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:19:03,750 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.73 vs. limit=12.0 2023-11-23 04:19:04,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334650 2023-11-23 04:19:13,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2231013.3333333335, ans=0.125 2023-11-23 04:19:23,052 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.92 vs. limit=6.0 2023-11-23 04:19:32,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2231080.0, ans=0.0 2023-11-23 04:19:51,976 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.134e+01 8.342e+01 8.814e+01 9.358e+01 1.133e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-23 04:19:52,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2231213.3333333335, ans=0.125 2023-11-23 04:20:00,535 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10050, loss[loss=0.05112, simple_loss=0.06588, pruned_loss=0.008882, audio_tagging_loss=0.009305, over 15823.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09276, pruned_loss=0.0142, audio_tagging_loss=0.009041, over 3033646.50 frames. ], batch size: 61, lr: 2.41e-03, grad_scale: 32.0 2023-11-23 04:20:08,048 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334700 2023-11-23 04:20:31,712 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.09 vs. limit=15.0 2023-11-23 04:20:44,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2231480.0, ans=0.125 2023-11-23 04:21:06,016 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10100, loss[loss=0.08968, simple_loss=0.1209, pruned_loss=0.02244, audio_tagging_loss=0.006778, over 15548.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.0929, pruned_loss=0.01436, audio_tagging_loss=0.009029, over 3035320.54 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:21:13,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334750 2023-11-23 04:21:19,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2231680.0, ans=0.0 2023-11-23 04:21:22,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2231680.0, ans=0.0 2023-11-23 04:21:25,663 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.01 vs. limit=15.0 2023-11-23 04:21:33,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2231746.6666666665, ans=0.125 2023-11-23 04:21:34,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2231746.6666666665, ans=0.05 2023-11-23 04:21:45,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2231813.3333333335, ans=0.0 2023-11-23 04:21:57,933 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:22:01,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2231880.0, ans=0.125 2023-11-23 04:22:02,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.66 vs. limit=22.5 2023-11-23 04:22:02,743 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.395e+01 8.740e+01 9.651e+01 1.218e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-23 04:22:07,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2231880.0, ans=0.0 2023-11-23 04:22:10,052 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10150, loss[loss=0.07984, simple_loss=0.1082, pruned_loss=0.01701, audio_tagging_loss=0.008732, over 14479.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09392, pruned_loss=0.01453, audio_tagging_loss=0.00907, over 3033472.70 frames. ], batch size: 53, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:22:17,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334800 2023-11-23 04:22:26,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2232013.3333333335, ans=0.2 2023-11-23 04:22:26,863 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.02 vs. limit=22.5 2023-11-23 04:22:29,274 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.46 vs. limit=10.0 2023-11-23 04:22:39,066 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:22:39,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2232080.0, ans=0.2 2023-11-23 04:22:40,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0 2023-11-23 04:22:44,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2232080.0, ans=0.125 2023-11-23 04:22:51,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2232146.6666666665, ans=0.025 2023-11-23 04:22:51,774 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.18 vs. limit=15.0 2023-11-23 04:23:01,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2232213.3333333335, ans=0.125 2023-11-23 04:23:07,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2232213.3333333335, ans=0.0 2023-11-23 04:23:12,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2232280.0, ans=0.1 2023-11-23 04:23:13,278 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10200, loss[loss=0.09084, simple_loss=0.112, pruned_loss=0.02594, audio_tagging_loss=0.008901, over 15164.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09368, pruned_loss=0.0144, audio_tagging_loss=0.009089, over 3041677.17 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:23:20,597 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334850 2023-11-23 04:23:23,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2232280.0, ans=0.125 2023-11-23 04:23:37,536 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:23:44,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2232413.3333333335, ans=0.2 2023-11-23 04:23:50,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2232413.3333333335, ans=0.125 2023-11-23 04:24:09,464 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.292e+01 8.929e+01 9.861e+01 1.255e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-23 04:24:17,294 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10250, loss[loss=0.07226, simple_loss=0.1078, pruned_loss=0.01251, audio_tagging_loss=0.005832, over 15810.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09214, pruned_loss=0.01413, audio_tagging_loss=0.009328, over 3037160.62 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:24:25,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334900 2023-11-23 04:24:49,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2232746.6666666665, ans=0.125 2023-11-23 04:24:56,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2232813.3333333335, ans=0.025 2023-11-23 04:24:58,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2232813.3333333335, ans=0.125 2023-11-23 04:25:16,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2232880.0, ans=0.0 2023-11-23 04:25:22,429 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10300, loss[loss=0.0789, simple_loss=0.1045, pruned_loss=0.01505, audio_tagging_loss=0.01158, over 15161.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09231, pruned_loss=0.01415, audio_tagging_loss=0.009294, over 3041589.51 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:25:29,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 334950 2023-11-23 04:26:19,731 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.243e+01 9.019e+01 9.717e+01 1.166e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 04:26:25,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.05 vs. limit=15.0 2023-11-23 04:26:25,988 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10350, loss[loss=0.06118, simple_loss=0.08169, pruned_loss=0.01237, audio_tagging_loss=0.007963, over 15038.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09208, pruned_loss=0.01405, audio_tagging_loss=0.009456, over 3039961.69 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:26:26,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2233280.0, ans=0.125 2023-11-23 04:26:31,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.06 vs. limit=15.0 2023-11-23 04:26:33,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335000 2023-11-23 04:26:33,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2233280.0, ans=0.0 2023-11-23 04:26:36,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2233280.0, ans=0.1 2023-11-23 04:26:47,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2233346.6666666665, ans=0.1 2023-11-23 04:27:03,844 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.71 vs. limit=15.0 2023-11-23 04:27:06,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2233480.0, ans=0.125 2023-11-23 04:27:17,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.83 vs. limit=15.0 2023-11-23 04:27:28,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2233546.6666666665, ans=0.1 2023-11-23 04:27:30,477 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10400, loss[loss=0.07158, simple_loss=0.0963, pruned_loss=0.01375, audio_tagging_loss=0.009675, over 14981.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09151, pruned_loss=0.01399, audio_tagging_loss=0.00959, over 3040623.35 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:27:39,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335050 2023-11-23 04:27:40,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2233613.3333333335, ans=0.125 2023-11-23 04:27:41,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2233613.3333333335, ans=0.0 2023-11-23 04:27:47,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.45 vs. limit=22.5 2023-11-23 04:27:54,862 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.93 vs. limit=12.0 2023-11-23 04:28:04,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2233746.6666666665, ans=0.2 2023-11-23 04:28:06,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2233746.6666666665, ans=0.1 2023-11-23 04:28:08,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2233813.3333333335, ans=0.07 2023-11-23 04:28:30,417 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.149e+01 8.861e+01 9.460e+01 1.224e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 04:28:35,394 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10450, loss[loss=0.05536, simple_loss=0.06452, pruned_loss=0.01194, audio_tagging_loss=0.01115, over 14801.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.0913, pruned_loss=0.01415, audio_tagging_loss=0.009606, over 3036478.82 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:28:37,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2233946.6666666665, ans=0.025 2023-11-23 04:28:41,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2233946.6666666665, ans=0.125 2023-11-23 04:28:43,402 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335100 2023-11-23 04:28:57,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0 2023-11-23 04:29:16,237 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.04 vs. limit=15.0 2023-11-23 04:29:17,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2234146.6666666665, ans=0.0 2023-11-23 04:29:39,668 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10500, loss[loss=0.06803, simple_loss=0.09815, pruned_loss=0.009979, audio_tagging_loss=0.008978, over 15715.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09159, pruned_loss=0.01421, audio_tagging_loss=0.009521, over 3042634.20 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:29:47,119 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335150 2023-11-23 04:29:52,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2234346.6666666665, ans=0.125 2023-11-23 04:30:00,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2234346.6666666665, ans=0.125 2023-11-23 04:30:03,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2234346.6666666665, ans=0.125 2023-11-23 04:30:12,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2234413.3333333335, ans=0.125 2023-11-23 04:30:20,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2234480.0, ans=0.0 2023-11-23 04:30:21,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2234480.0, ans=0.1 2023-11-23 04:30:27,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2234480.0, ans=0.125 2023-11-23 04:30:37,993 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.079e+01 8.755e+01 9.423e+01 1.196e+02, threshold=1.751e+02, percent-clipped=0.0 2023-11-23 04:30:42,945 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10550, loss[loss=0.06408, simple_loss=0.07715, pruned_loss=0.01497, audio_tagging_loss=0.01054, over 14999.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09147, pruned_loss=0.01417, audio_tagging_loss=0.009429, over 3041444.05 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:30:50,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2234613.3333333335, ans=0.05 2023-11-23 04:30:51,477 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335200 2023-11-23 04:30:57,226 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:31:45,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2234880.0, ans=0.125 2023-11-23 04:31:47,913 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10600, loss[loss=0.05955, simple_loss=0.08092, pruned_loss=0.01178, audio_tagging_loss=0.007313, over 16062.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09096, pruned_loss=0.01415, audio_tagging_loss=0.00929, over 3043537.44 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:31:55,315 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335250 2023-11-23 04:31:56,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2234946.6666666665, ans=0.125 2023-11-23 04:32:07,082 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:32:22,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2235080.0, ans=0.125 2023-11-23 04:32:43,211 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:32:46,336 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.415e+01 8.292e+01 8.923e+01 9.577e+01 1.197e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 04:32:51,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2235280.0, ans=0.125 2023-11-23 04:32:51,943 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10650, loss[loss=0.07209, simple_loss=0.1025, pruned_loss=0.01465, audio_tagging_loss=0.006178, over 14665.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09105, pruned_loss=0.01419, audio_tagging_loss=0.009201, over 3043596.37 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:32:59,230 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335300 2023-11-23 04:33:05,985 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.28 vs. limit=15.0 2023-11-23 04:33:11,581 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.82 vs. limit=6.0 2023-11-23 04:33:13,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2235346.6666666665, ans=0.125 2023-11-23 04:33:20,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2235413.3333333335, ans=0.0 2023-11-23 04:33:25,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2235413.3333333335, ans=0.125 2023-11-23 04:33:29,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2235480.0, ans=0.04949747468305833 2023-11-23 04:33:31,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2235480.0, ans=0.0 2023-11-23 04:33:34,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2235480.0, ans=0.0 2023-11-23 04:33:35,552 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.80 vs. limit=15.0 2023-11-23 04:33:42,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.56 vs. limit=15.0 2023-11-23 04:33:55,250 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10700, loss[loss=0.1113, simple_loss=0.1595, pruned_loss=0.02582, audio_tagging_loss=0.005747, over 15991.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09168, pruned_loss=0.01435, audio_tagging_loss=0.009215, over 3047779.22 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:33:59,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2235613.3333333335, ans=0.2 2023-11-23 04:33:59,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2235613.3333333335, ans=0.125 2023-11-23 04:34:03,142 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335350 2023-11-23 04:34:33,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2235813.3333333335, ans=0.125 2023-11-23 04:34:35,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2235813.3333333335, ans=0.125 2023-11-23 04:34:38,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2235813.3333333335, ans=0.05 2023-11-23 04:34:41,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2235813.3333333335, ans=0.1 2023-11-23 04:34:54,567 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.303e+01 8.893e+01 9.663e+01 2.175e+02, threshold=1.779e+02, percent-clipped=1.0 2023-11-23 04:34:59,982 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10750, loss[loss=0.06401, simple_loss=0.09422, pruned_loss=0.01075, audio_tagging_loss=0.006145, over 15311.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09154, pruned_loss=0.01421, audio_tagging_loss=0.009181, over 3047698.23 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:35:07,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335400 2023-11-23 04:35:43,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2236146.6666666665, ans=0.0 2023-11-23 04:35:44,223 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.04 vs. limit=10.0 2023-11-23 04:36:03,438 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10800, loss[loss=0.07824, simple_loss=0.1016, pruned_loss=0.02009, audio_tagging_loss=0.007349, over 15227.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.0907, pruned_loss=0.01402, audio_tagging_loss=0.009196, over 3042589.26 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:36:11,628 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335450 2023-11-23 04:36:25,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2236346.6666666665, ans=0.125 2023-11-23 04:36:37,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2236413.3333333335, ans=0.0 2023-11-23 04:36:39,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.30 vs. limit=22.5 2023-11-23 04:36:41,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2236480.0, ans=0.1 2023-11-23 04:36:43,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.64 vs. limit=22.5 2023-11-23 04:36:48,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2236480.0, ans=0.125 2023-11-23 04:37:02,342 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.072e+01 8.725e+01 9.352e+01 1.267e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-23 04:37:07,354 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10850, loss[loss=0.06709, simple_loss=0.09075, pruned_loss=0.01214, audio_tagging_loss=0.009573, over 14655.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09176, pruned_loss=0.0141, audio_tagging_loss=0.009005, over 3042435.57 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:37:15,402 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335500 2023-11-23 04:37:19,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2236680.0, ans=0.2 2023-11-23 04:37:43,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2236746.6666666665, ans=0.2 2023-11-23 04:37:52,674 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0 2023-11-23 04:37:53,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2236813.3333333335, ans=0.125 2023-11-23 04:37:59,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2236880.0, ans=0.125 2023-11-23 04:38:07,685 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:38:10,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2236946.6666666665, ans=0.0 2023-11-23 04:38:10,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2236946.6666666665, ans=0.0 2023-11-23 04:38:11,718 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10900, loss[loss=0.05983, simple_loss=0.07334, pruned_loss=0.01321, audio_tagging_loss=0.009947, over 16008.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09193, pruned_loss=0.01426, audio_tagging_loss=0.00905, over 3046982.88 frames. ], batch size: 62, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:38:19,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335550 2023-11-23 04:38:27,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2237013.3333333335, ans=0.125 2023-11-23 04:38:33,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2237013.3333333335, ans=0.1 2023-11-23 04:38:35,586 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.87 vs. limit=15.0 2023-11-23 04:38:36,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.65 vs. limit=15.0 2023-11-23 04:38:41,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2237080.0, ans=0.125 2023-11-23 04:38:51,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.54 vs. limit=22.5 2023-11-23 04:39:01,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2237213.3333333335, ans=0.125 2023-11-23 04:39:10,067 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.359e+01 8.884e+01 9.760e+01 1.616e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-23 04:39:14,995 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 10950, loss[loss=0.09334, simple_loss=0.1309, pruned_loss=0.01856, audio_tagging_loss=0.009307, over 14218.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09259, pruned_loss=0.01429, audio_tagging_loss=0.009112, over 3046573.35 frames. ], batch size: 52, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:39:22,554 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335600 2023-11-23 04:39:48,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.59 vs. limit=10.0 2023-11-23 04:39:51,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2237413.3333333335, ans=0.125 2023-11-23 04:40:05,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2237546.6666666665, ans=0.1 2023-11-23 04:40:19,239 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11000, loss[loss=0.06027, simple_loss=0.0752, pruned_loss=0.01078, audio_tagging_loss=0.01189, over 15787.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09189, pruned_loss=0.01408, audio_tagging_loss=0.009266, over 3047875.82 frames. ], batch size: 60, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:40:25,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2237613.3333333335, ans=0.125 2023-11-23 04:40:26,595 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335650 2023-11-23 04:40:28,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2237613.3333333335, ans=0.125 2023-11-23 04:40:28,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2237613.3333333335, ans=0.0 2023-11-23 04:40:29,663 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:40:48,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2237746.6666666665, ans=0.0 2023-11-23 04:40:54,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2237746.6666666665, ans=0.025 2023-11-23 04:41:13,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2237880.0, ans=0.0 2023-11-23 04:41:16,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2237880.0, ans=0.125 2023-11-23 04:41:19,577 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.268e+01 8.870e+01 9.647e+01 1.227e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-23 04:41:21,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2237880.0, ans=0.125 2023-11-23 04:41:23,741 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11050, loss[loss=0.04322, simple_loss=0.04602, pruned_loss=0.007312, audio_tagging_loss=0.0129, over 15463.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09232, pruned_loss=0.01417, audio_tagging_loss=0.009242, over 3047064.83 frames. ], batch size: 60, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:41:31,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335700 2023-11-23 04:41:56,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2238080.0, ans=0.2 2023-11-23 04:42:05,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.94 vs. limit=15.0 2023-11-23 04:42:22,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2238213.3333333335, ans=0.0 2023-11-23 04:42:27,702 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11100, loss[loss=0.07705, simple_loss=0.09954, pruned_loss=0.02001, audio_tagging_loss=0.007265, over 14292.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09273, pruned_loss=0.0142, audio_tagging_loss=0.009312, over 3048863.30 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:42:35,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335750 2023-11-23 04:42:38,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2238346.6666666665, ans=0.0 2023-11-23 04:42:45,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.78 vs. limit=22.5 2023-11-23 04:42:50,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2238346.6666666665, ans=0.0 2023-11-23 04:42:54,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2238413.3333333335, ans=0.1 2023-11-23 04:43:22,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2238546.6666666665, ans=0.0 2023-11-23 04:43:27,388 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.249e+01 8.341e+01 8.890e+01 9.757e+01 1.222e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-23 04:43:31,112 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11150, loss[loss=0.0657, simple_loss=0.08873, pruned_loss=0.01156, audio_tagging_loss=0.009779, over 16569.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09192, pruned_loss=0.01414, audio_tagging_loss=0.009458, over 3045459.70 frames. ], batch size: 63, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 04:43:39,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335800 2023-11-23 04:43:58,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2238746.6666666665, ans=0.125 2023-11-23 04:44:05,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2238746.6666666665, ans=0.1 2023-11-23 04:44:07,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2238746.6666666665, ans=0.0 2023-11-23 04:44:21,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2238880.0, ans=0.0 2023-11-23 04:44:22,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2238880.0, ans=0.125 2023-11-23 04:44:35,963 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11200, loss[loss=0.09424, simple_loss=0.1271, pruned_loss=0.02208, audio_tagging_loss=0.008591, over 17288.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09213, pruned_loss=0.01416, audio_tagging_loss=0.00953, over 3047048.34 frames. ], batch size: 61, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:44:40,969 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.84 vs. limit=15.0 2023-11-23 04:44:44,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335850 2023-11-23 04:44:56,900 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:45:02,279 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.24 vs. limit=15.0 2023-11-23 04:45:10,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2239080.0, ans=0.025 2023-11-23 04:45:14,337 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.77 vs. limit=15.0 2023-11-23 04:45:17,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2239146.6666666665, ans=0.125 2023-11-23 04:45:19,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.33 vs. limit=12.0 2023-11-23 04:45:29,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2239213.3333333335, ans=6.0 2023-11-23 04:45:36,867 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.304e+01 9.043e+01 9.933e+01 1.318e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 04:45:40,655 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11250, loss[loss=0.05253, simple_loss=0.06433, pruned_loss=0.008659, audio_tagging_loss=0.01171, over 14504.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09164, pruned_loss=0.01418, audio_tagging_loss=0.00955, over 3046794.27 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:45:48,231 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335900 2023-11-23 04:45:58,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2239346.6666666665, ans=0.1 2023-11-23 04:46:05,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2239413.3333333335, ans=0.0 2023-11-23 04:46:12,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2239413.3333333335, ans=0.0 2023-11-23 04:46:18,916 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:46:44,914 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11300, loss[loss=0.06077, simple_loss=0.07962, pruned_loss=0.01419, audio_tagging_loss=0.006763, over 14440.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09161, pruned_loss=0.01413, audio_tagging_loss=0.009396, over 3042778.10 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:46:52,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 335950 2023-11-23 04:47:21,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0 2023-11-23 04:47:41,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2239880.0, ans=0.035 2023-11-23 04:47:43,901 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.333e+01 8.937e+01 9.779e+01 1.201e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 04:47:47,561 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11350, loss[loss=0.08953, simple_loss=0.1211, pruned_loss=0.02307, audio_tagging_loss=0.005925, over 14538.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09223, pruned_loss=0.01419, audio_tagging_loss=0.009218, over 3048680.96 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:47:48,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2239946.6666666665, ans=0.1 2023-11-23 04:47:56,801 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336000 2023-11-23 04:47:57,235 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.25 vs. limit=12.0 2023-11-23 04:47:57,422 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.67 vs. limit=15.0 2023-11-23 04:48:12,672 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.68 vs. limit=15.0 2023-11-23 04:48:34,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2240146.6666666665, ans=0.2 2023-11-23 04:48:43,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2240213.3333333335, ans=0.125 2023-11-23 04:48:56,659 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11400, loss[loss=0.07016, simple_loss=0.1022, pruned_loss=0.01234, audio_tagging_loss=0.006731, over 15953.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09243, pruned_loss=0.01441, audio_tagging_loss=0.009194, over 3043183.08 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:48:56,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2240280.0, ans=0.0 2023-11-23 04:48:57,048 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 04:49:03,968 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336050 2023-11-23 04:49:05,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2240280.0, ans=0.125 2023-11-23 04:49:10,764 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.90 vs. limit=22.5 2023-11-23 04:49:17,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.07 vs. limit=6.0 2023-11-23 04:49:18,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2240346.6666666665, ans=0.2 2023-11-23 04:49:27,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2240413.3333333335, ans=0.0 2023-11-23 04:49:51,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=15.0 2023-11-23 04:49:53,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2240546.6666666665, ans=0.125 2023-11-23 04:49:56,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2240546.6666666665, ans=15.0 2023-11-23 04:49:56,452 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.358e+01 8.216e+01 8.830e+01 9.675e+01 1.432e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 04:50:00,131 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11450, loss[loss=0.06751, simple_loss=0.07854, pruned_loss=0.01714, audio_tagging_loss=0.0111, over 14324.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09342, pruned_loss=0.0145, audio_tagging_loss=0.009162, over 3046578.28 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:50:07,503 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336100 2023-11-23 04:50:07,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2240613.3333333335, ans=0.1 2023-11-23 04:50:07,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2240613.3333333335, ans=0.0 2023-11-23 04:50:39,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2240813.3333333335, ans=0.035 2023-11-23 04:51:02,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2240946.6666666665, ans=0.0 2023-11-23 04:51:02,789 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11500, loss[loss=0.08673, simple_loss=0.1179, pruned_loss=0.02016, audio_tagging_loss=0.007602, over 15475.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09476, pruned_loss=0.01469, audio_tagging_loss=0.009107, over 3051033.73 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:51:11,554 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336150 2023-11-23 04:51:35,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2241080.0, ans=0.025 2023-11-23 04:51:37,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2241080.0, ans=0.0 2023-11-23 04:51:52,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.13 vs. limit=15.0 2023-11-23 04:52:01,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2241213.3333333335, ans=0.0 2023-11-23 04:52:01,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2241213.3333333335, ans=0.125 2023-11-23 04:52:04,945 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.510e+01 8.416e+01 8.974e+01 9.531e+01 1.779e+02, threshold=1.795e+02, percent-clipped=1.0 2023-11-23 04:52:08,772 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11550, loss[loss=0.06119, simple_loss=0.07874, pruned_loss=0.01307, audio_tagging_loss=0.008749, over 15065.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09466, pruned_loss=0.01464, audio_tagging_loss=0.009014, over 3052643.48 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:52:09,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2241280.0, ans=0.0 2023-11-23 04:52:12,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2241280.0, ans=15.0 2023-11-23 04:52:16,214 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336200 2023-11-23 04:52:27,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2241346.6666666665, ans=0.0 2023-11-23 04:52:37,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2241413.3333333335, ans=0.025 2023-11-23 04:52:46,524 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 04:52:52,380 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.69 vs. limit=15.0 2023-11-23 04:53:12,012 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11600, loss[loss=0.05922, simple_loss=0.08026, pruned_loss=0.008953, audio_tagging_loss=0.01013, over 14931.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09501, pruned_loss=0.01467, audio_tagging_loss=0.009068, over 3059069.89 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 32.0 2023-11-23 04:53:15,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2241613.3333333335, ans=0.0 2023-11-23 04:53:19,414 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336250 2023-11-23 04:53:23,686 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.93 vs. limit=15.0 2023-11-23 04:53:41,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2241746.6666666665, ans=0.125 2023-11-23 04:53:42,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2241746.6666666665, ans=0.125 2023-11-23 04:53:53,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2241813.3333333335, ans=0.125 2023-11-23 04:54:12,059 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.276e+01 8.228e+01 9.009e+01 9.457e+01 1.103e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-23 04:54:12,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2241880.0, ans=0.2 2023-11-23 04:54:15,858 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11650, loss[loss=0.0603, simple_loss=0.08179, pruned_loss=0.00959, audio_tagging_loss=0.009811, over 15207.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09429, pruned_loss=0.01449, audio_tagging_loss=0.009211, over 3049912.34 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 32.0 2023-11-23 04:54:23,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336300 2023-11-23 04:54:38,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2242013.3333333335, ans=0.125 2023-11-23 04:54:43,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2242080.0, ans=0.125 2023-11-23 04:55:02,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2242146.6666666665, ans=0.0 2023-11-23 04:55:09,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.86 vs. limit=15.0 2023-11-23 04:55:18,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2242213.3333333335, ans=0.04949747468305833 2023-11-23 04:55:21,589 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11700, loss[loss=0.05397, simple_loss=0.0704, pruned_loss=0.01048, audio_tagging_loss=0.008295, over 14224.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09341, pruned_loss=0.01436, audio_tagging_loss=0.00925, over 3048153.61 frames. ], batch size: 53, lr: 2.41e-03, grad_scale: 32.0 2023-11-23 04:55:29,467 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336350 2023-11-23 04:55:30,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2242280.0, ans=0.1 2023-11-23 04:55:38,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2242346.6666666665, ans=0.125 2023-11-23 04:55:49,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2242413.3333333335, ans=0.05 2023-11-23 04:56:21,993 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.295e+01 8.847e+01 9.516e+01 1.261e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 04:56:24,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2242613.3333333335, ans=0.125 2023-11-23 04:56:25,701 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11750, loss[loss=0.04864, simple_loss=0.06627, pruned_loss=0.006768, audio_tagging_loss=0.008735, over 14753.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09278, pruned_loss=0.01411, audio_tagging_loss=0.0093, over 3047654.72 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 32.0 2023-11-23 04:56:29,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2242613.3333333335, ans=0.125 2023-11-23 04:56:33,242 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336400 2023-11-23 04:56:37,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2242680.0, ans=0.0 2023-11-23 04:56:39,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2242680.0, ans=0.2 2023-11-23 04:56:42,827 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.53 vs. limit=6.0 2023-11-23 04:56:44,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2242680.0, ans=0.0 2023-11-23 04:57:13,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2242813.3333333335, ans=0.95 2023-11-23 04:57:16,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2242880.0, ans=0.0 2023-11-23 04:57:17,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2242880.0, ans=0.125 2023-11-23 04:57:27,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2242880.0, ans=0.125 2023-11-23 04:57:29,439 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11800, loss[loss=0.07882, simple_loss=0.1055, pruned_loss=0.01661, audio_tagging_loss=0.009459, over 14985.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09315, pruned_loss=0.01429, audio_tagging_loss=0.009249, over 3055287.33 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 32.0 2023-11-23 04:57:29,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2242946.6666666665, ans=0.0 2023-11-23 04:57:36,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336450 2023-11-23 04:57:58,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2243080.0, ans=0.2 2023-11-23 04:58:10,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.07 vs. limit=15.0 2023-11-23 04:58:11,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.86 vs. limit=6.0 2023-11-23 04:58:28,125 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.83 vs. limit=15.0 2023-11-23 04:58:31,391 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.315e+01 8.837e+01 9.445e+01 1.101e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 04:58:33,809 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11850, loss[loss=0.06166, simple_loss=0.08444, pruned_loss=0.0109, audio_tagging_loss=0.008542, over 15945.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09383, pruned_loss=0.01448, audio_tagging_loss=0.009162, over 3051882.17 frames. ], batch size: 63, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:58:35,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2243280.0, ans=0.125 2023-11-23 04:58:41,639 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336500 2023-11-23 04:59:07,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2243413.3333333335, ans=0.125 2023-11-23 04:59:10,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.33 vs. limit=6.0 2023-11-23 04:59:13,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2243480.0, ans=0.125 2023-11-23 04:59:38,820 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11900, loss[loss=0.06836, simple_loss=0.08684, pruned_loss=0.01548, audio_tagging_loss=0.009458, over 15633.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09321, pruned_loss=0.01431, audio_tagging_loss=0.009244, over 3046542.90 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 04:59:46,233 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336550 2023-11-23 04:59:52,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2243680.0, ans=0.0 2023-11-23 04:59:56,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2243680.0, ans=0.1 2023-11-23 05:00:09,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2243746.6666666665, ans=0.0 2023-11-23 05:00:16,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2243813.3333333335, ans=0.0 2023-11-23 05:00:40,662 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.380e+01 8.877e+01 9.570e+01 1.262e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-23 05:00:41,904 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 11950, loss[loss=0.06882, simple_loss=0.09314, pruned_loss=0.0145, audio_tagging_loss=0.007741, over 15262.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09305, pruned_loss=0.01431, audio_tagging_loss=0.009362, over 3049691.22 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0 2023-11-23 05:00:49,034 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336600 2023-11-23 05:01:13,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2244080.0, ans=0.125 2023-11-23 05:01:29,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2244146.6666666665, ans=0.1 2023-11-23 05:01:32,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.76 vs. limit=10.0 2023-11-23 05:01:34,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2244213.3333333335, ans=0.0 2023-11-23 05:01:35,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2244213.3333333335, ans=0.125 2023-11-23 05:01:43,546 INFO [train_asr.py:1221] (3/4) Epoch 28, batch 12000, loss[loss=0.06081, simple_loss=0.07757, pruned_loss=0.009846, audio_tagging_loss=0.01218, over 15150.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09283, pruned_loss=0.01434, audio_tagging_loss=0.009439, over 3053568.15 frames. ], batch size: 60, lr: 2.41e-03, grad_scale: 16.0 2023-11-23 05:01:43,549 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 05:02:27,080 INFO [train_asr.py:1253] (3/4) Epoch 28, validation: loss=0.05897, simple_loss=0.05124, pruned_loss=0.005128, audio_tagging_loss=0.02822, over 4681554.00 frames. 2023-11-23 05:02:27,081 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 05:02:34,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336650 2023-11-23 05:02:38,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2244346.6666666665, ans=0.2 2023-11-23 05:03:30,837 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 0, loss[loss=0.08216, simple_loss=0.08789, pruned_loss=0.01562, audio_tagging_loss=0.02259, over 15669.00 frames. ], tot_loss[loss=0.08216, simple_loss=0.08789, pruned_loss=0.01562, audio_tagging_loss=0.02259, over 15669.00 frames. ], batch size: 57, lr: 2.37e-03, grad_scale: 32.0 2023-11-23 05:03:30,838 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 05:04:08,413 INFO [train_asr.py:1253] (3/4) Epoch 29, validation: loss=0.05816, simple_loss=0.05122, pruned_loss=0.005095, audio_tagging_loss=0.02745, over 4681554.00 frames. 2023-11-23 05:04:08,414 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 05:04:40,857 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 8.521e+01 9.209e+01 1.026e+02 2.736e+02, threshold=1.842e+02, percent-clipped=1.0 2023-11-23 05:04:49,534 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336700 2023-11-23 05:05:02,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2244706.6666666665, ans=0.95 2023-11-23 05:05:12,082 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 50, loss[loss=0.08103, simple_loss=0.09672, pruned_loss=0.0157, audio_tagging_loss=0.01697, over 15522.00 frames. ], tot_loss[loss=0.07761, simple_loss=0.09212, pruned_loss=0.01405, audio_tagging_loss=0.0175, over 692477.46 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:05:19,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2244773.3333333335, ans=0.125 2023-11-23 05:05:20,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2244773.3333333335, ans=0.1 2023-11-23 05:05:29,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2244840.0, ans=0.125 2023-11-23 05:05:29,367 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=12.0 2023-11-23 05:05:30,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2244840.0, ans=0.125 2023-11-23 05:05:30,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2244840.0, ans=0.125 2023-11-23 05:05:52,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2244973.3333333335, ans=0.0 2023-11-23 05:05:54,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336750 2023-11-23 05:06:03,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2245040.0, ans=0.0 2023-11-23 05:06:18,182 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 100, loss[loss=0.07744, simple_loss=0.09535, pruned_loss=0.01275, audio_tagging_loss=0.01702, over 16056.00 frames. ], tot_loss[loss=0.07658, simple_loss=0.09127, pruned_loss=0.01405, audio_tagging_loss=0.0169, over 1208570.35 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:06:18,847 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.96 vs. limit=15.0 2023-11-23 05:06:26,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2245106.6666666665, ans=0.125 2023-11-23 05:06:45,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2245240.0, ans=0.1 2023-11-23 05:06:49,674 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.633e+01 8.985e+01 9.652e+01 1.024e+02 1.304e+02, threshold=1.930e+02, percent-clipped=0.0 2023-11-23 05:06:52,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2245240.0, ans=0.2 2023-11-23 05:06:59,117 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336800 2023-11-23 05:07:10,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2245373.3333333335, ans=0.1 2023-11-23 05:07:22,233 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 150, loss[loss=0.06854, simple_loss=0.09249, pruned_loss=0.01274, audio_tagging_loss=0.00956, over 15100.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09163, pruned_loss=0.01398, audio_tagging_loss=0.01507, over 1615268.49 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:07:40,852 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.05 vs. limit=15.0 2023-11-23 05:07:53,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2245573.3333333335, ans=0.125 2023-11-23 05:07:59,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2245573.3333333335, ans=0.125 2023-11-23 05:08:04,421 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336850 2023-11-23 05:08:21,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2245706.6666666665, ans=0.1 2023-11-23 05:08:27,348 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 200, loss[loss=0.07793, simple_loss=0.1026, pruned_loss=0.01657, audio_tagging_loss=0.01009, over 14508.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.0924, pruned_loss=0.01414, audio_tagging_loss=0.01337, over 1939812.01 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:08:45,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2245840.0, ans=0.0 2023-11-23 05:08:56,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2245906.6666666665, ans=0.0 2023-11-23 05:09:00,526 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.383e+01 9.139e+01 9.827e+01 1.313e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-23 05:09:08,670 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336900 2023-11-23 05:09:16,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.78 vs. limit=22.5 2023-11-23 05:09:24,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2246040.0, ans=0.125 2023-11-23 05:09:32,717 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 250, loss[loss=0.06444, simple_loss=0.08222, pruned_loss=0.01341, audio_tagging_loss=0.009924, over 15783.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09315, pruned_loss=0.01446, audio_tagging_loss=0.01213, over 2190411.70 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:09:36,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2246106.6666666665, ans=0.5 2023-11-23 05:09:40,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2246106.6666666665, ans=0.0 2023-11-23 05:09:50,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2246173.3333333335, ans=0.1 2023-11-23 05:10:12,442 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 336950 2023-11-23 05:10:13,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.52 vs. limit=15.0 2023-11-23 05:10:21,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2246306.6666666665, ans=0.125 2023-11-23 05:10:31,999 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0 2023-11-23 05:10:36,214 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 300, loss[loss=0.0658, simple_loss=0.08728, pruned_loss=0.01265, audio_tagging_loss=0.009516, over 14809.00 frames. ], tot_loss[loss=0.07275, simple_loss=0.09403, pruned_loss=0.01448, audio_tagging_loss=0.01125, over 2378401.44 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:10:37,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2246440.0, ans=0.0 2023-11-23 05:10:38,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2246440.0, ans=0.125 2023-11-23 05:10:41,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2246440.0, ans=0.0 2023-11-23 05:11:10,506 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.241e+01 8.912e+01 9.854e+01 1.344e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 05:11:17,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337000 2023-11-23 05:11:34,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2246706.6666666665, ans=0.125 2023-11-23 05:11:40,977 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 350, loss[loss=0.0618, simple_loss=0.08043, pruned_loss=0.0136, audio_tagging_loss=0.007985, over 16615.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09401, pruned_loss=0.01433, audio_tagging_loss=0.01065, over 2532146.56 frames. ], batch size: 62, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:11:45,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2246773.3333333335, ans=0.125 2023-11-23 05:11:47,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2246773.3333333335, ans=0.025 2023-11-23 05:11:49,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2246773.3333333335, ans=0.125 2023-11-23 05:11:54,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2246840.0, ans=0.125 2023-11-23 05:12:07,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2246906.6666666665, ans=0.1 2023-11-23 05:12:12,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2246906.6666666665, ans=0.0 2023-11-23 05:12:19,916 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:12:22,217 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337050 2023-11-23 05:12:28,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2246973.3333333335, ans=0.07 2023-11-23 05:12:38,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2247040.0, ans=0.1 2023-11-23 05:12:39,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2247040.0, ans=0.125 2023-11-23 05:12:39,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2247040.0, ans=10.0 2023-11-23 05:12:42,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2247040.0, ans=0.0 2023-11-23 05:12:44,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2247040.0, ans=0.0 2023-11-23 05:12:46,380 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 400, loss[loss=0.04202, simple_loss=0.04855, pruned_loss=0.0062, audio_tagging_loss=0.01154, over 14926.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09279, pruned_loss=0.01412, audio_tagging_loss=0.01039, over 2646362.80 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:12:56,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.68 vs. limit=15.0 2023-11-23 05:13:04,298 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.24 vs. limit=6.0 2023-11-23 05:13:08,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2247173.3333333335, ans=0.0 2023-11-23 05:13:18,413 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.176e+01 8.720e+01 9.339e+01 1.453e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-23 05:13:26,475 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337100 2023-11-23 05:13:49,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2247440.0, ans=0.0 2023-11-23 05:13:50,279 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 450, loss[loss=0.04808, simple_loss=0.0653, pruned_loss=0.007928, audio_tagging_loss=0.007507, over 15708.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09282, pruned_loss=0.01425, audio_tagging_loss=0.009931, over 2742223.33 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:13:57,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.11 vs. limit=12.0 2023-11-23 05:13:58,172 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.99 vs. limit=15.0 2023-11-23 05:14:21,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2247573.3333333335, ans=0.125 2023-11-23 05:14:24,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.48 vs. limit=12.0 2023-11-23 05:14:31,395 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337150 2023-11-23 05:14:47,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2247706.6666666665, ans=0.2 2023-11-23 05:14:51,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2247706.6666666665, ans=0.0 2023-11-23 05:14:53,602 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 500, loss[loss=0.05854, simple_loss=0.0757, pruned_loss=0.01067, audio_tagging_loss=0.01002, over 16545.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09277, pruned_loss=0.01424, audio_tagging_loss=0.009773, over 2815258.23 frames. ], batch size: 65, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:15:04,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2247773.3333333335, ans=0.0 2023-11-23 05:15:05,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2247840.0, ans=0.0 2023-11-23 05:15:13,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.85 vs. limit=10.0 2023-11-23 05:15:26,969 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.484e+01 9.126e+01 9.707e+01 1.686e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-23 05:15:28,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2247906.6666666665, ans=0.125 2023-11-23 05:15:34,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337200 2023-11-23 05:15:46,770 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.18 vs. limit=8.0 2023-11-23 05:15:57,895 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 550, loss[loss=0.05675, simple_loss=0.07385, pruned_loss=0.009273, audio_tagging_loss=0.01054, over 15231.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09278, pruned_loss=0.01427, audio_tagging_loss=0.009638, over 2865289.11 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:16:00,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2248106.6666666665, ans=0.2 2023-11-23 05:16:13,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2248173.3333333335, ans=0.125 2023-11-23 05:16:19,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2248173.3333333335, ans=0.125 2023-11-23 05:16:25,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2248240.0, ans=0.125 2023-11-23 05:16:38,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337250 2023-11-23 05:16:52,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2248373.3333333335, ans=0.0 2023-11-23 05:17:01,997 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 600, loss[loss=0.07187, simple_loss=0.09437, pruned_loss=0.01668, audio_tagging_loss=0.008014, over 14406.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09258, pruned_loss=0.01419, audio_tagging_loss=0.009503, over 2906353.96 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:17:34,614 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.545e+01 8.479e+01 8.808e+01 9.540e+01 1.178e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-23 05:17:42,776 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337300 2023-11-23 05:17:51,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2248706.6666666665, ans=0.5 2023-11-23 05:17:51,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2248706.6666666665, ans=0.025 2023-11-23 05:17:54,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2248706.6666666665, ans=0.1 2023-11-23 05:18:05,043 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 650, loss[loss=0.07179, simple_loss=0.1039, pruned_loss=0.01252, audio_tagging_loss=0.007306, over 15951.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09322, pruned_loss=0.01412, audio_tagging_loss=0.009405, over 2938777.08 frames. ], batch size: 62, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:18:11,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2248773.3333333335, ans=0.125 2023-11-23 05:18:24,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2248840.0, ans=0.5 2023-11-23 05:18:39,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.44 vs. limit=22.5 2023-11-23 05:18:46,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2248973.3333333335, ans=0.0 2023-11-23 05:18:46,994 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337350 2023-11-23 05:18:49,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2248973.3333333335, ans=0.0 2023-11-23 05:18:55,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2249040.0, ans=0.0 2023-11-23 05:19:09,389 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 700, loss[loss=0.06968, simple_loss=0.09007, pruned_loss=0.01763, audio_tagging_loss=0.007013, over 14247.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09328, pruned_loss=0.01404, audio_tagging_loss=0.00928, over 2965564.06 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:19:18,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.36 vs. limit=15.0 2023-11-23 05:19:30,495 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.45 vs. limit=15.0 2023-11-23 05:19:44,727 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.145e+01 8.167e+01 8.621e+01 9.555e+01 1.511e+02, threshold=1.724e+02, percent-clipped=0.0 2023-11-23 05:19:51,201 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337400 2023-11-23 05:19:56,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2249306.6666666665, ans=0.1 2023-11-23 05:20:00,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2249373.3333333335, ans=0.2 2023-11-23 05:20:15,762 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 750, loss[loss=0.06016, simple_loss=0.08178, pruned_loss=0.01096, audio_tagging_loss=0.00831, over 15866.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09227, pruned_loss=0.01385, audio_tagging_loss=0.009459, over 2988215.54 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:20:33,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2249506.6666666665, ans=0.0 2023-11-23 05:20:37,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2249506.6666666665, ans=0.125 2023-11-23 05:20:50,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2249573.3333333335, ans=0.1 2023-11-23 05:20:53,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2249640.0, ans=0.125 2023-11-23 05:20:57,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337450 2023-11-23 05:20:59,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.27 vs. limit=15.0 2023-11-23 05:21:01,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2249640.0, ans=0.125 2023-11-23 05:21:15,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2249706.6666666665, ans=0.2 2023-11-23 05:21:19,956 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 800, loss[loss=0.06853, simple_loss=0.07633, pruned_loss=0.01704, audio_tagging_loss=0.01332, over 14232.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09288, pruned_loss=0.01401, audio_tagging_loss=0.009455, over 3001425.70 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:21:27,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2249773.3333333335, ans=0.0 2023-11-23 05:21:37,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2249840.0, ans=0.125 2023-11-23 05:21:53,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2249906.6666666665, ans=0.125 2023-11-23 05:21:55,385 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.531e+01 8.991e+01 9.709e+01 1.279e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-23 05:22:01,755 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337500 2023-11-23 05:22:24,139 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 850, loss[loss=0.06125, simple_loss=0.078, pruned_loss=0.0122, audio_tagging_loss=0.01005, over 15624.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09247, pruned_loss=0.01408, audio_tagging_loss=0.009514, over 3006829.73 frames. ], batch size: 61, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:23:02,996 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.57 vs. limit=12.0 2023-11-23 05:23:04,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.51 vs. limit=15.0 2023-11-23 05:23:06,294 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337550 2023-11-23 05:23:25,018 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:23:30,311 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 900, loss[loss=0.06694, simple_loss=0.09257, pruned_loss=0.01196, audio_tagging_loss=0.008696, over 15714.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09237, pruned_loss=0.01411, audio_tagging_loss=0.00962, over 3019583.27 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:23:40,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2250440.0, ans=0.07 2023-11-23 05:24:03,823 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.195e+01 8.718e+01 9.591e+01 1.345e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-23 05:24:11,536 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337600 2023-11-23 05:24:30,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2250706.6666666665, ans=0.125 2023-11-23 05:24:35,034 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 950, loss[loss=0.0459, simple_loss=0.05852, pruned_loss=0.008038, audio_tagging_loss=0.008602, over 15115.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09295, pruned_loss=0.01428, audio_tagging_loss=0.009447, over 3025236.12 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:24:36,985 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0 2023-11-23 05:24:46,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2250840.0, ans=0.0 2023-11-23 05:25:17,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337650 2023-11-23 05:25:21,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2250973.3333333335, ans=0.0 2023-11-23 05:25:39,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.39 vs. limit=15.0 2023-11-23 05:25:39,751 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1000, loss[loss=0.05694, simple_loss=0.07188, pruned_loss=0.01272, audio_tagging_loss=0.008276, over 15853.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09148, pruned_loss=0.01404, audio_tagging_loss=0.00945, over 3026215.08 frames. ], batch size: 61, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:25:41,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2251106.6666666665, ans=0.0 2023-11-23 05:25:46,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2251106.6666666665, ans=0.125 2023-11-23 05:25:52,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2251173.3333333335, ans=0.125 2023-11-23 05:25:56,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2251173.3333333335, ans=0.0 2023-11-23 05:26:07,697 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 05:26:07,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2251240.0, ans=0.0 2023-11-23 05:26:15,019 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.282e+01 9.010e+01 9.966e+01 1.225e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-23 05:26:19,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2251306.6666666665, ans=0.0 2023-11-23 05:26:20,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2251306.6666666665, ans=0.2 2023-11-23 05:26:21,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337700 2023-11-23 05:26:23,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2251306.6666666665, ans=15.0 2023-11-23 05:26:25,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2251306.6666666665, ans=0.1 2023-11-23 05:26:31,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2251373.3333333335, ans=0.0 2023-11-23 05:26:35,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2251373.3333333335, ans=0.0 2023-11-23 05:26:44,941 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1050, loss[loss=0.08899, simple_loss=0.1259, pruned_loss=0.01915, audio_tagging_loss=0.006899, over 14627.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09227, pruned_loss=0.01415, audio_tagging_loss=0.009236, over 3029559.34 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:26:50,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2251440.0, ans=0.125 2023-11-23 05:27:06,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2251506.6666666665, ans=0.05 2023-11-23 05:27:08,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2251506.6666666665, ans=0.125 2023-11-23 05:27:08,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.13 vs. limit=22.5 2023-11-23 05:27:18,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2251573.3333333335, ans=0.0 2023-11-23 05:27:25,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2251640.0, ans=0.125 2023-11-23 05:27:26,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337750 2023-11-23 05:27:33,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2251640.0, ans=0.125 2023-11-23 05:27:35,032 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:27:36,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2251706.6666666665, ans=0.0 2023-11-23 05:27:36,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2251706.6666666665, ans=0.0 2023-11-23 05:27:37,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2251706.6666666665, ans=0.035 2023-11-23 05:27:48,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2251706.6666666665, ans=0.125 2023-11-23 05:27:50,135 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1100, loss[loss=0.06117, simple_loss=0.07287, pruned_loss=0.01323, audio_tagging_loss=0.01151, over 15730.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09219, pruned_loss=0.01419, audio_tagging_loss=0.009103, over 3023902.73 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:27:52,681 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 05:27:54,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2251773.3333333335, ans=0.07 2023-11-23 05:27:57,100 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.89 vs. limit=15.0 2023-11-23 05:28:25,465 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.248e+01 8.961e+01 9.560e+01 1.246e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-23 05:28:31,871 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337800 2023-11-23 05:28:54,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.72 vs. limit=15.0 2023-11-23 05:28:54,928 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1150, loss[loss=0.04182, simple_loss=0.05348, pruned_loss=0.005333, audio_tagging_loss=0.00975, over 14392.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09288, pruned_loss=0.01424, audio_tagging_loss=0.009082, over 3028735.29 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:29:12,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2252173.3333333335, ans=0.125 2023-11-23 05:29:25,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2252240.0, ans=0.2 2023-11-23 05:29:36,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337850 2023-11-23 05:29:41,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2252306.6666666665, ans=0.0 2023-11-23 05:30:00,352 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1200, loss[loss=0.06896, simple_loss=0.08909, pruned_loss=0.01474, audio_tagging_loss=0.009669, over 14682.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09307, pruned_loss=0.0142, audio_tagging_loss=0.009026, over 3035882.62 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:30:03,525 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.44 vs. limit=22.5 2023-11-23 05:30:05,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2252440.0, ans=0.125 2023-11-23 05:30:12,369 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.68 vs. limit=22.5 2023-11-23 05:30:15,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2252506.6666666665, ans=0.125 2023-11-23 05:30:18,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2252506.6666666665, ans=0.2 2023-11-23 05:30:35,273 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.804e+01 8.344e+01 9.032e+01 9.683e+01 1.496e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-23 05:30:40,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337900 2023-11-23 05:31:04,498 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1250, loss[loss=0.06514, simple_loss=0.08491, pruned_loss=0.01395, audio_tagging_loss=0.008739, over 14621.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09296, pruned_loss=0.01428, audio_tagging_loss=0.009066, over 3026375.45 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:31:45,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 337950 2023-11-23 05:32:04,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2253040.0, ans=0.0 2023-11-23 05:32:07,627 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1300, loss[loss=0.05285, simple_loss=0.07596, pruned_loss=0.008417, audio_tagging_loss=0.006456, over 14416.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09232, pruned_loss=0.01408, audio_tagging_loss=0.009089, over 3028928.43 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:32:36,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2253240.0, ans=0.125 2023-11-23 05:32:39,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2253240.0, ans=0.125 2023-11-23 05:32:44,458 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.370e+01 8.150e+01 8.858e+01 9.270e+01 1.252e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 05:32:49,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338000 2023-11-23 05:32:55,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2253306.6666666665, ans=0.1 2023-11-23 05:33:07,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2253373.3333333335, ans=0.125 2023-11-23 05:33:07,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2253373.3333333335, ans=0.2 2023-11-23 05:33:07,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2253373.3333333335, ans=0.025 2023-11-23 05:33:11,586 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:33:11,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2253373.3333333335, ans=0.2 2023-11-23 05:33:13,807 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1350, loss[loss=0.05954, simple_loss=0.06975, pruned_loss=0.00964, audio_tagging_loss=0.01502, over 13843.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09209, pruned_loss=0.0141, audio_tagging_loss=0.009068, over 3032675.03 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:33:17,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2253440.0, ans=0.125 2023-11-23 05:33:19,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2253440.0, ans=0.125 2023-11-23 05:33:25,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2253506.6666666665, ans=0.0 2023-11-23 05:33:42,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.91 vs. limit=10.0 2023-11-23 05:33:46,442 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0 2023-11-23 05:33:53,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338050 2023-11-23 05:33:56,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.87 vs. limit=22.5 2023-11-23 05:33:59,452 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 05:34:08,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.42 vs. limit=15.0 2023-11-23 05:34:17,326 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1400, loss[loss=0.04294, simple_loss=0.04703, pruned_loss=0.00757, audio_tagging_loss=0.01186, over 14695.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09239, pruned_loss=0.01419, audio_tagging_loss=0.009042, over 3028104.89 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:34:26,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2253773.3333333335, ans=0.125 2023-11-23 05:34:27,974 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:34:30,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2253840.0, ans=0.125 2023-11-23 05:34:32,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2253840.0, ans=0.125 2023-11-23 05:34:37,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2253840.0, ans=0.125 2023-11-23 05:34:53,599 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.459e+01 9.010e+01 9.793e+01 1.707e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-23 05:34:58,583 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338100 2023-11-23 05:35:09,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2254040.0, ans=0.1 2023-11-23 05:35:21,284 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1450, loss[loss=0.08912, simple_loss=0.1207, pruned_loss=0.01961, audio_tagging_loss=0.009185, over 14537.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09355, pruned_loss=0.01432, audio_tagging_loss=0.009176, over 3031432.20 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:35:22,126 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0 2023-11-23 05:35:23,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.19 vs. limit=22.5 2023-11-23 05:35:45,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.08 vs. limit=15.0 2023-11-23 05:35:52,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2254240.0, ans=0.0 2023-11-23 05:36:01,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338150 2023-11-23 05:36:24,837 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1500, loss[loss=0.07293, simple_loss=0.09248, pruned_loss=0.01629, audio_tagging_loss=0.0104, over 14847.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09451, pruned_loss=0.01453, audio_tagging_loss=0.009207, over 3037031.60 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:36:28,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2254440.0, ans=0.2 2023-11-23 05:36:34,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2254440.0, ans=0.2 2023-11-23 05:36:38,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2254506.6666666665, ans=0.1 2023-11-23 05:36:39,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2254506.6666666665, ans=0.0 2023-11-23 05:37:00,769 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.437e+01 9.139e+01 9.780e+01 1.863e+02, threshold=1.828e+02, percent-clipped=1.0 2023-11-23 05:37:03,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2254640.0, ans=0.0 2023-11-23 05:37:05,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338200 2023-11-23 05:37:21,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2254706.6666666665, ans=0.125 2023-11-23 05:37:22,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2254706.6666666665, ans=0.125 2023-11-23 05:37:29,301 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1550, loss[loss=0.05534, simple_loss=0.06801, pruned_loss=0.00918, audio_tagging_loss=0.01215, over 15685.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09417, pruned_loss=0.01447, audio_tagging_loss=0.009316, over 3041506.04 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:37:35,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2254773.3333333335, ans=0.125 2023-11-23 05:37:38,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.23 vs. limit=22.5 2023-11-23 05:37:47,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2254840.0, ans=0.125 2023-11-23 05:38:10,775 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338250 2023-11-23 05:38:32,709 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1600, loss[loss=0.07997, simple_loss=0.09989, pruned_loss=0.01777, audio_tagging_loss=0.01226, over 16527.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09268, pruned_loss=0.0143, audio_tagging_loss=0.009432, over 3042095.07 frames. ], batch size: 61, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:38:35,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2255106.6666666665, ans=15.0 2023-11-23 05:39:09,190 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 8.320e+01 8.917e+01 9.601e+01 1.213e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 05:39:13,025 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338300 2023-11-23 05:39:28,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2255373.3333333335, ans=0.0 2023-11-23 05:39:35,861 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1650, loss[loss=0.06112, simple_loss=0.08446, pruned_loss=0.009449, audio_tagging_loss=0.009439, over 15442.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09323, pruned_loss=0.01434, audio_tagging_loss=0.009448, over 3044038.75 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:39:57,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2255506.6666666665, ans=0.0 2023-11-23 05:40:15,721 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338350 2023-11-23 05:40:24,507 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.36 vs. limit=22.5 2023-11-23 05:40:39,311 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1700, loss[loss=0.07983, simple_loss=0.1204, pruned_loss=0.01283, audio_tagging_loss=0.006799, over 15326.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.0935, pruned_loss=0.01431, audio_tagging_loss=0.009392, over 3052617.97 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:41:14,134 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:41:16,148 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.211e+01 9.004e+01 9.824e+01 1.268e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 05:41:17,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2255973.3333333335, ans=0.1 2023-11-23 05:41:19,986 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338400 2023-11-23 05:41:26,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2255973.3333333335, ans=0.0 2023-11-23 05:41:36,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2256040.0, ans=0.2 2023-11-23 05:41:42,299 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1750, loss[loss=0.06886, simple_loss=0.09411, pruned_loss=0.01308, audio_tagging_loss=0.00872, over 14921.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09326, pruned_loss=0.01426, audio_tagging_loss=0.009295, over 3052946.01 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:41:48,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2256106.6666666665, ans=0.125 2023-11-23 05:41:53,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2256173.3333333335, ans=0.125 2023-11-23 05:41:53,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2256173.3333333335, ans=0.125 2023-11-23 05:41:57,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2256173.3333333335, ans=0.125 2023-11-23 05:42:22,891 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338450 2023-11-23 05:42:34,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2256373.3333333335, ans=0.125 2023-11-23 05:42:45,363 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1800, loss[loss=0.07202, simple_loss=0.09555, pruned_loss=0.0131, audio_tagging_loss=0.01114, over 14922.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09351, pruned_loss=0.0143, audio_tagging_loss=0.009155, over 3045445.38 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:42:46,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2256440.0, ans=0.125 2023-11-23 05:43:18,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2256573.3333333335, ans=0.125 2023-11-23 05:43:21,548 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.187e+01 9.056e+01 9.555e+01 1.230e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-23 05:43:25,402 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338500 2023-11-23 05:43:48,552 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1850, loss[loss=0.07177, simple_loss=0.09428, pruned_loss=0.01452, audio_tagging_loss=0.0101, over 15864.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09296, pruned_loss=0.01422, audio_tagging_loss=0.009015, over 3044960.40 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:43:48,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2256773.3333333335, ans=0.0 2023-11-23 05:43:56,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2256773.3333333335, ans=0.0 2023-11-23 05:44:29,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338550 2023-11-23 05:44:29,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2256973.3333333335, ans=0.125 2023-11-23 05:44:30,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2256973.3333333335, ans=0.0 2023-11-23 05:44:40,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2257040.0, ans=0.125 2023-11-23 05:44:51,024 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1900, loss[loss=0.0637, simple_loss=0.0814, pruned_loss=0.012, audio_tagging_loss=0.011, over 14904.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09245, pruned_loss=0.01401, audio_tagging_loss=0.008954, over 3044565.52 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:45:08,610 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:45:28,229 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.552e+01 8.593e+01 9.216e+01 1.001e+02 1.158e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-23 05:45:29,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2257306.6666666665, ans=0.125 2023-11-23 05:45:32,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338600 2023-11-23 05:45:35,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=2257306.6666666665, ans=0.02 2023-11-23 05:45:42,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2257373.3333333335, ans=0.0 2023-11-23 05:45:54,453 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 1950, loss[loss=0.04899, simple_loss=0.06515, pruned_loss=0.008453, audio_tagging_loss=0.007958, over 14943.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09145, pruned_loss=0.01395, audio_tagging_loss=0.008993, over 3040741.53 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:46:05,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=2257440.0, ans=15.0 2023-11-23 05:46:20,601 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.89 vs. limit=22.5 2023-11-23 05:46:34,865 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338650 2023-11-23 05:46:41,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2257640.0, ans=0.125 2023-11-23 05:46:47,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2257706.6666666665, ans=0.125 2023-11-23 05:46:50,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2257706.6666666665, ans=0.0 2023-11-23 05:46:55,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2257706.6666666665, ans=0.125 2023-11-23 05:46:55,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2257706.6666666665, ans=0.125 2023-11-23 05:46:57,983 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2000, loss[loss=0.06872, simple_loss=0.07809, pruned_loss=0.01838, audio_tagging_loss=0.0113, over 15272.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.0916, pruned_loss=0.01411, audio_tagging_loss=0.009109, over 3054243.84 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:47:00,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2257773.3333333335, ans=0.125 2023-11-23 05:47:28,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2257906.6666666665, ans=0.125 2023-11-23 05:47:33,486 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.322e+01 9.117e+01 1.012e+02 1.277e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-23 05:47:37,941 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338700 2023-11-23 05:47:40,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2257973.3333333335, ans=0.1 2023-11-23 05:47:46,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2257973.3333333335, ans=0.0 2023-11-23 05:47:58,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2258040.0, ans=0.07 2023-11-23 05:48:00,702 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2050, loss[loss=0.06015, simple_loss=0.07821, pruned_loss=0.01113, audio_tagging_loss=0.009912, over 14973.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09225, pruned_loss=0.01432, audio_tagging_loss=0.009098, over 3051579.50 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:48:02,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2258106.6666666665, ans=10.0 2023-11-23 05:48:04,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2258106.6666666665, ans=0.0 2023-11-23 05:48:06,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2258106.6666666665, ans=0.125 2023-11-23 05:48:11,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2258106.6666666665, ans=0.125 2023-11-23 05:48:18,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2258173.3333333335, ans=0.2 2023-11-23 05:48:18,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2258173.3333333335, ans=0.0 2023-11-23 05:48:24,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2258240.0, ans=0.125 2023-11-23 05:48:26,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2258240.0, ans=0.0 2023-11-23 05:48:38,287 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.61 vs. limit=22.5 2023-11-23 05:48:41,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338750 2023-11-23 05:48:49,497 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.11 vs. limit=10.0 2023-11-23 05:48:51,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2258373.3333333335, ans=0.0 2023-11-23 05:49:03,160 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2100, loss[loss=0.09058, simple_loss=0.1239, pruned_loss=0.02086, audio_tagging_loss=0.007789, over 16165.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09231, pruned_loss=0.01434, audio_tagging_loss=0.00903, over 3047884.40 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:49:09,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2258440.0, ans=0.125 2023-11-23 05:49:11,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2258440.0, ans=0.125 2023-11-23 05:49:36,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.13 vs. limit=15.0 2023-11-23 05:49:38,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2258573.3333333335, ans=0.125 2023-11-23 05:49:40,176 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.396e+01 9.168e+01 1.020e+02 1.328e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-23 05:49:41,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2258640.0, ans=0.95 2023-11-23 05:49:43,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338800 2023-11-23 05:49:45,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2258640.0, ans=0.0 2023-11-23 05:49:47,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2258640.0, ans=0.125 2023-11-23 05:50:04,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2258706.6666666665, ans=0.125 2023-11-23 05:50:07,661 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2150, loss[loss=0.08682, simple_loss=0.1178, pruned_loss=0.02127, audio_tagging_loss=0.006671, over 15955.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09371, pruned_loss=0.01466, audio_tagging_loss=0.008975, over 3044411.78 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:50:22,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.17 vs. limit=15.0 2023-11-23 05:50:45,277 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 05:50:47,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338850 2023-11-23 05:50:52,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.13 vs. limit=15.0 2023-11-23 05:50:54,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2258973.3333333335, ans=0.125 2023-11-23 05:51:11,696 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2200, loss[loss=0.06199, simple_loss=0.07886, pruned_loss=0.01091, audio_tagging_loss=0.01164, over 16556.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09421, pruned_loss=0.01462, audio_tagging_loss=0.008958, over 3042578.40 frames. ], batch size: 62, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:51:19,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2259106.6666666665, ans=0.0 2023-11-23 05:51:30,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2259173.3333333335, ans=0.125 2023-11-23 05:51:33,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.55 vs. limit=15.0 2023-11-23 05:51:37,154 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.75 vs. limit=6.0 2023-11-23 05:51:45,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2259240.0, ans=0.0 2023-11-23 05:51:50,018 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.527e+01 9.201e+01 9.842e+01 1.279e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-23 05:51:53,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338900 2023-11-23 05:51:58,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2259306.6666666665, ans=0.2 2023-11-23 05:52:08,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2259373.3333333335, ans=0.1 2023-11-23 05:52:16,092 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2250, loss[loss=0.07626, simple_loss=0.1028, pruned_loss=0.01538, audio_tagging_loss=0.009492, over 15302.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09412, pruned_loss=0.01462, audio_tagging_loss=0.009049, over 3051545.73 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:52:18,004 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.34 vs. limit=6.0 2023-11-23 05:52:21,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2259440.0, ans=0.125 2023-11-23 05:52:44,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2259573.3333333335, ans=0.0 2023-11-23 05:52:45,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2259573.3333333335, ans=0.07 2023-11-23 05:52:52,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2259573.3333333335, ans=0.2 2023-11-23 05:52:55,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2259640.0, ans=0.0 2023-11-23 05:52:58,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 338950 2023-11-23 05:53:12,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2259706.6666666665, ans=0.0 2023-11-23 05:53:13,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2259706.6666666665, ans=0.5 2023-11-23 05:53:22,023 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2300, loss[loss=0.06985, simple_loss=0.1001, pruned_loss=0.01436, audio_tagging_loss=0.005444, over 15278.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09415, pruned_loss=0.01443, audio_tagging_loss=0.009106, over 3048811.86 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:53:31,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2259773.3333333335, ans=0.0 2023-11-23 05:54:00,275 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.330e+01 8.807e+01 9.666e+01 1.881e+02, threshold=1.761e+02, percent-clipped=1.0 2023-11-23 05:54:02,891 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339000 2023-11-23 05:54:06,054 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 05:54:19,419 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 05:54:26,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2260106.6666666665, ans=0.2 2023-11-23 05:54:27,450 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2350, loss[loss=0.06529, simple_loss=0.0898, pruned_loss=0.01339, audio_tagging_loss=0.006994, over 15408.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09361, pruned_loss=0.0143, audio_tagging_loss=0.009154, over 3051851.47 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:54:32,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2260106.6666666665, ans=0.0 2023-11-23 05:54:53,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2260240.0, ans=0.125 2023-11-23 05:55:01,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2260240.0, ans=0.0 2023-11-23 05:55:08,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339050 2023-11-23 05:55:08,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2260306.6666666665, ans=0.1 2023-11-23 05:55:10,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2260306.6666666665, ans=0.125 2023-11-23 05:55:30,918 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2400, loss[loss=0.05857, simple_loss=0.07592, pruned_loss=0.0129, audio_tagging_loss=0.007713, over 14173.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09431, pruned_loss=0.01454, audio_tagging_loss=0.009213, over 3046521.33 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:56:09,725 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.287e+01 8.924e+01 9.833e+01 1.260e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 05:56:10,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2260640.0, ans=0.125 2023-11-23 05:56:12,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339100 2023-11-23 05:56:30,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2260706.6666666665, ans=0.1 2023-11-23 05:56:34,913 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2450, loss[loss=0.06908, simple_loss=0.08569, pruned_loss=0.01583, audio_tagging_loss=0.01041, over 14587.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.0942, pruned_loss=0.01445, audio_tagging_loss=0.009315, over 3043701.07 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 05:56:52,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2260840.0, ans=0.125 2023-11-23 05:57:14,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.74 vs. limit=15.0 2023-11-23 05:57:15,360 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339150 2023-11-23 05:57:38,410 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2500, loss[loss=0.05947, simple_loss=0.08039, pruned_loss=0.01233, audio_tagging_loss=0.006943, over 14974.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09333, pruned_loss=0.01445, audio_tagging_loss=0.009304, over 3037388.31 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:57:41,699 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.45 vs. limit=22.5 2023-11-23 05:57:54,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0 2023-11-23 05:58:18,162 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.151e+01 8.943e+01 9.911e+01 1.178e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-23 05:58:19,507 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339200 2023-11-23 05:58:27,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2261306.6666666665, ans=0.04949747468305833 2023-11-23 05:58:31,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2261373.3333333335, ans=0.0 2023-11-23 05:58:32,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2261373.3333333335, ans=0.1 2023-11-23 05:58:41,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2261440.0, ans=0.0 2023-11-23 05:58:42,712 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2550, loss[loss=0.04749, simple_loss=0.06595, pruned_loss=0.006933, audio_tagging_loss=0.007584, over 14490.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09344, pruned_loss=0.01451, audio_tagging_loss=0.00925, over 3031442.32 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:58:44,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2261440.0, ans=0.0 2023-11-23 05:59:15,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2261573.3333333335, ans=0.125 2023-11-23 05:59:23,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339250 2023-11-23 05:59:23,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2261640.0, ans=0.1 2023-11-23 05:59:45,927 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2600, loss[loss=0.069, simple_loss=0.09837, pruned_loss=0.01472, audio_tagging_loss=0.005091, over 14415.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09353, pruned_loss=0.01439, audio_tagging_loss=0.009091, over 3034193.04 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 05:59:52,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2261773.3333333335, ans=0.0 2023-11-23 05:59:56,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2261773.3333333335, ans=0.2 2023-11-23 06:00:04,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2261840.0, ans=0.05 2023-11-23 06:00:25,731 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.329e+01 8.953e+01 9.650e+01 1.435e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 06:00:27,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339300 2023-11-23 06:00:30,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.84 vs. limit=15.0 2023-11-23 06:00:35,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2262040.0, ans=0.125 2023-11-23 06:00:50,471 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2650, loss[loss=0.08185, simple_loss=0.1074, pruned_loss=0.01958, audio_tagging_loss=0.008576, over 15629.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09284, pruned_loss=0.01421, audio_tagging_loss=0.009121, over 3032532.42 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 06:01:01,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2262173.3333333335, ans=0.2 2023-11-23 06:01:04,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2262173.3333333335, ans=0.125 2023-11-23 06:01:30,847 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.98 vs. limit=15.0 2023-11-23 06:01:31,609 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339350 2023-11-23 06:01:47,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2262373.3333333335, ans=0.07 2023-11-23 06:01:53,641 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2700, loss[loss=0.0828, simple_loss=0.1184, pruned_loss=0.01505, audio_tagging_loss=0.008544, over 15694.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09282, pruned_loss=0.01419, audio_tagging_loss=0.009123, over 3043873.01 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 06:01:55,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2262440.0, ans=0.125 2023-11-23 06:02:02,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.51 vs. limit=15.0 2023-11-23 06:02:20,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2262573.3333333335, ans=0.0 2023-11-23 06:02:33,770 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.218e+01 8.946e+01 9.636e+01 1.335e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-23 06:02:35,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339400 2023-11-23 06:02:57,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2262773.3333333335, ans=0.2 2023-11-23 06:02:57,726 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.03 vs. limit=6.0 2023-11-23 06:02:58,437 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2750, loss[loss=0.06035, simple_loss=0.07974, pruned_loss=0.01019, audio_tagging_loss=0.01029, over 14441.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09241, pruned_loss=0.01408, audio_tagging_loss=0.009265, over 3042949.02 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 06:03:39,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339450 2023-11-23 06:03:52,615 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 06:04:03,033 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2800, loss[loss=0.06039, simple_loss=0.07674, pruned_loss=0.01136, audio_tagging_loss=0.01066, over 15190.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09307, pruned_loss=0.01407, audio_tagging_loss=0.009193, over 3045236.08 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0 2023-11-23 06:04:17,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.27 vs. limit=15.0 2023-11-23 06:04:18,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2263173.3333333335, ans=0.125 2023-11-23 06:04:19,681 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:04:28,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2263240.0, ans=0.0 2023-11-23 06:04:44,003 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.163e+01 8.713e+01 9.405e+01 1.279e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-23 06:04:44,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339500 2023-11-23 06:04:58,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2263373.3333333335, ans=0.125 2023-11-23 06:04:58,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2263373.3333333335, ans=0.1 2023-11-23 06:05:06,473 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2850, loss[loss=0.07287, simple_loss=0.1004, pruned_loss=0.01462, audio_tagging_loss=0.008033, over 14871.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09305, pruned_loss=0.01398, audio_tagging_loss=0.009124, over 3042013.73 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 06:05:09,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2263440.0, ans=0.09899494936611666 2023-11-23 06:05:11,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2263440.0, ans=0.125 2023-11-23 06:05:20,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2263506.6666666665, ans=0.1 2023-11-23 06:05:37,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2263573.3333333335, ans=0.125 2023-11-23 06:05:40,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2263573.3333333335, ans=0.0 2023-11-23 06:05:42,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2263573.3333333335, ans=0.0 2023-11-23 06:05:48,745 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339550 2023-11-23 06:06:11,466 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2900, loss[loss=0.05274, simple_loss=0.06551, pruned_loss=0.01164, audio_tagging_loss=0.008353, over 15765.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09294, pruned_loss=0.01396, audio_tagging_loss=0.009144, over 3040825.53 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0 2023-11-23 06:06:23,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.31 vs. limit=22.5 2023-11-23 06:06:32,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2263840.0, ans=0.125 2023-11-23 06:06:52,980 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.150e+01 8.750e+01 9.470e+01 1.345e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-23 06:06:53,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339600 2023-11-23 06:06:59,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2263973.3333333335, ans=0.125 2023-11-23 06:06:59,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2263973.3333333335, ans=0.125 2023-11-23 06:07:03,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2264040.0, ans=0.125 2023-11-23 06:07:17,756 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 2950, loss[loss=0.0685, simple_loss=0.08891, pruned_loss=0.01556, audio_tagging_loss=0.008486, over 14453.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09282, pruned_loss=0.01402, audio_tagging_loss=0.009201, over 3043005.28 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:07:18,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2264106.6666666665, ans=0.0 2023-11-23 06:07:55,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2264306.6666666665, ans=0.125 2023-11-23 06:07:58,452 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.34 vs. limit=6.0 2023-11-23 06:07:59,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339650 2023-11-23 06:08:22,225 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3000, loss[loss=0.06269, simple_loss=0.07956, pruned_loss=0.01351, audio_tagging_loss=0.0094, over 14863.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09329, pruned_loss=0.01439, audio_tagging_loss=0.009202, over 3043802.29 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:08:22,226 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 06:08:54,188 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.0625, 3.0076, 3.2427, 2.9600, 3.7063, 3.7602, 3.2853, 3.1172], device='cuda:3') 2023-11-23 06:09:05,311 INFO [train_asr.py:1253] (3/4) Epoch 29, validation: loss=0.05823, simple_loss=0.05127, pruned_loss=0.005185, audio_tagging_loss=0.02741, over 4681554.00 frames. 2023-11-23 06:09:05,312 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 06:09:13,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2264440.0, ans=0.0 2023-11-23 06:09:15,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2264440.0, ans=0.125 2023-11-23 06:09:19,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2264506.6666666665, ans=0.125 2023-11-23 06:09:46,383 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.265e+01 8.432e+01 8.978e+01 9.876e+01 1.396e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 06:09:46,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339700 2023-11-23 06:09:46,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2264640.0, ans=0.0 2023-11-23 06:09:54,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2264640.0, ans=0.0 2023-11-23 06:10:10,834 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3050, loss[loss=0.08312, simple_loss=0.1161, pruned_loss=0.01796, audio_tagging_loss=0.007083, over 14615.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09427, pruned_loss=0.01448, audio_tagging_loss=0.009241, over 3045789.77 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:10:21,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.85 vs. limit=15.0 2023-11-23 06:10:23,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2264840.0, ans=0.1 2023-11-23 06:10:30,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2264840.0, ans=0.09899494936611666 2023-11-23 06:10:34,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2264906.6666666665, ans=0.125 2023-11-23 06:10:46,348 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 06:10:52,466 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339750 2023-11-23 06:10:57,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2264973.3333333335, ans=0.125 2023-11-23 06:11:00,686 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.57 vs. limit=8.0 2023-11-23 06:11:02,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2265040.0, ans=0.2 2023-11-23 06:11:14,617 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3100, loss[loss=0.07556, simple_loss=0.1021, pruned_loss=0.01298, audio_tagging_loss=0.01154, over 16015.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09352, pruned_loss=0.0144, audio_tagging_loss=0.009366, over 3046817.54 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 8.0 2023-11-23 06:11:19,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2265106.6666666665, ans=0.0 2023-11-23 06:11:19,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2265106.6666666665, ans=0.0 2023-11-23 06:11:33,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2265173.3333333335, ans=0.0 2023-11-23 06:11:49,891 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.85 vs. limit=22.5 2023-11-23 06:11:55,546 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339800 2023-11-23 06:11:56,016 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.18 vs. limit=22.5 2023-11-23 06:11:56,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.703e+01 8.232e+01 8.826e+01 9.441e+01 1.388e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-23 06:12:03,802 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.63 vs. limit=10.0 2023-11-23 06:12:08,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2265373.3333333335, ans=0.05 2023-11-23 06:12:12,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2265373.3333333335, ans=10.0 2023-11-23 06:12:16,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2265373.3333333335, ans=0.04949747468305833 2023-11-23 06:12:18,035 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3150, loss[loss=0.0799, simple_loss=0.1045, pruned_loss=0.01851, audio_tagging_loss=0.009135, over 14293.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09407, pruned_loss=0.01448, audio_tagging_loss=0.009374, over 3047790.94 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 8.0 2023-11-23 06:12:25,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2265440.0, ans=0.2 2023-11-23 06:12:31,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2265506.6666666665, ans=0.0 2023-11-23 06:12:59,777 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339850 2023-11-23 06:13:21,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2265706.6666666665, ans=0.125 2023-11-23 06:13:23,921 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3200, loss[loss=0.05955, simple_loss=0.08255, pruned_loss=0.009033, audio_tagging_loss=0.009237, over 14087.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09311, pruned_loss=0.01421, audio_tagging_loss=0.009418, over 3047379.20 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:13:24,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2265773.3333333335, ans=0.2 2023-11-23 06:13:48,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2265906.6666666665, ans=0.125 2023-11-23 06:14:04,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339900 2023-11-23 06:14:06,194 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.473e+01 8.333e+01 9.166e+01 9.819e+01 1.242e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-23 06:14:21,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2266040.0, ans=0.1 2023-11-23 06:14:23,933 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.79 vs. limit=6.0 2023-11-23 06:14:24,985 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.84 vs. limit=22.5 2023-11-23 06:14:26,821 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3250, loss[loss=0.06482, simple_loss=0.08309, pruned_loss=0.01262, audio_tagging_loss=0.01065, over 15913.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09202, pruned_loss=0.0141, audio_tagging_loss=0.009533, over 3047677.83 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:14:40,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2266173.3333333335, ans=0.1 2023-11-23 06:14:40,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2266173.3333333335, ans=0.2 2023-11-23 06:14:44,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2266173.3333333335, ans=0.2 2023-11-23 06:14:45,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2266173.3333333335, ans=0.04949747468305833 2023-11-23 06:14:48,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2266173.3333333335, ans=0.1 2023-11-23 06:14:54,821 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.67 vs. limit=22.5 2023-11-23 06:14:58,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2266240.0, ans=0.125 2023-11-23 06:15:07,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2266306.6666666665, ans=0.125 2023-11-23 06:15:07,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2266306.6666666665, ans=0.1 2023-11-23 06:15:08,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 339950 2023-11-23 06:15:26,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2266373.3333333335, ans=0.0 2023-11-23 06:15:30,531 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3300, loss[loss=0.06804, simple_loss=0.09411, pruned_loss=0.01343, audio_tagging_loss=0.007557, over 15199.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09204, pruned_loss=0.01401, audio_tagging_loss=0.009463, over 3049167.14 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:15:34,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2266440.0, ans=0.125 2023-11-23 06:15:36,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.73 vs. limit=22.5 2023-11-23 06:16:07,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2266573.3333333335, ans=0.125 2023-11-23 06:16:09,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2266640.0, ans=0.125 2023-11-23 06:16:12,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340000 2023-11-23 06:16:13,718 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.298e+01 8.900e+01 9.803e+01 1.322e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-23 06:16:21,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2266640.0, ans=0.0 2023-11-23 06:16:40,086 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3350, loss[loss=0.06368, simple_loss=0.07529, pruned_loss=0.01425, audio_tagging_loss=0.01179, over 14323.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09336, pruned_loss=0.01432, audio_tagging_loss=0.009345, over 3048876.93 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:16:56,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2266840.0, ans=0.125 2023-11-23 06:16:57,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2266840.0, ans=0.0 2023-11-23 06:17:04,060 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.28 vs. limit=10.0 2023-11-23 06:17:04,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2266906.6666666665, ans=0.05 2023-11-23 06:17:20,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340050 2023-11-23 06:17:40,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2267040.0, ans=0.125 2023-11-23 06:17:41,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2267040.0, ans=0.0 2023-11-23 06:17:43,978 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3400, loss[loss=0.07036, simple_loss=0.1009, pruned_loss=0.01404, audio_tagging_loss=0.005865, over 15755.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09335, pruned_loss=0.01424, audio_tagging_loss=0.00919, over 3050458.80 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:17:57,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2267173.3333333335, ans=0.125 2023-11-23 06:18:00,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2267173.3333333335, ans=0.0 2023-11-23 06:18:16,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2267240.0, ans=0.125 2023-11-23 06:18:19,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2267240.0, ans=0.1 2023-11-23 06:18:23,758 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.13 vs. limit=22.5 2023-11-23 06:18:25,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340100 2023-11-23 06:18:26,638 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.359e+01 8.997e+01 9.733e+01 1.260e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 06:18:47,379 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3450, loss[loss=0.06317, simple_loss=0.08087, pruned_loss=0.01105, audio_tagging_loss=0.01168, over 15921.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09275, pruned_loss=0.01413, audio_tagging_loss=0.009185, over 3056786.84 frames. ], batch size: 62, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:18:50,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2267440.0, ans=0.0 2023-11-23 06:18:53,908 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:18:59,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2267506.6666666665, ans=0.0 2023-11-23 06:19:02,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.83 vs. limit=15.0 2023-11-23 06:19:17,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2267573.3333333335, ans=0.0 2023-11-23 06:19:29,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340150 2023-11-23 06:19:33,671 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:19:52,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.24 vs. limit=15.0 2023-11-23 06:19:53,646 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3500, loss[loss=0.06681, simple_loss=0.08441, pruned_loss=0.01277, audio_tagging_loss=0.01183, over 15120.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09275, pruned_loss=0.01427, audio_tagging_loss=0.009101, over 3055726.90 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:19:56,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.56 vs. limit=22.5 2023-11-23 06:20:01,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2267773.3333333335, ans=0.125 2023-11-23 06:20:06,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2267840.0, ans=0.05 2023-11-23 06:20:25,107 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 06:20:34,343 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340200 2023-11-23 06:20:35,329 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.175e+01 8.873e+01 9.452e+01 1.244e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-23 06:20:58,520 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3550, loss[loss=0.05261, simple_loss=0.06588, pruned_loss=0.01104, audio_tagging_loss=0.008631, over 14643.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.0928, pruned_loss=0.01425, audio_tagging_loss=0.00903, over 3047425.65 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:21:06,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2268106.6666666665, ans=0.1 2023-11-23 06:21:16,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2268173.3333333335, ans=0.1 2023-11-23 06:21:16,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2268173.3333333335, ans=0.1 2023-11-23 06:21:39,474 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340250 2023-11-23 06:21:39,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2268306.6666666665, ans=0.04949747468305833 2023-11-23 06:21:43,213 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.83 vs. limit=6.0 2023-11-23 06:22:01,963 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3600, loss[loss=0.06057, simple_loss=0.0815, pruned_loss=0.01195, audio_tagging_loss=0.007874, over 16559.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09242, pruned_loss=0.01424, audio_tagging_loss=0.009016, over 3050019.00 frames. ], batch size: 64, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 06:22:06,430 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.30 vs. limit=22.5 2023-11-23 06:22:18,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2268506.6666666665, ans=0.1 2023-11-23 06:22:28,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2268573.3333333335, ans=0.1 2023-11-23 06:22:32,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2268573.3333333335, ans=0.125 2023-11-23 06:22:35,420 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.43 vs. limit=15.0 2023-11-23 06:22:43,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340300 2023-11-23 06:22:44,649 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.233e+01 8.921e+01 9.911e+01 1.235e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 06:22:45,348 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.65 vs. limit=15.0 2023-11-23 06:23:07,300 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3650, loss[loss=0.05861, simple_loss=0.06942, pruned_loss=0.01172, audio_tagging_loss=0.01218, over 14374.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09331, pruned_loss=0.01441, audio_tagging_loss=0.008972, over 3052417.46 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 06:23:12,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2268773.3333333335, ans=0.125 2023-11-23 06:23:25,729 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:23:29,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2268840.0, ans=0.125 2023-11-23 06:23:30,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2268840.0, ans=0.05 2023-11-23 06:23:37,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2268906.6666666665, ans=0.125 2023-11-23 06:23:39,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2268906.6666666665, ans=0.1 2023-11-23 06:23:47,654 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340350 2023-11-23 06:24:05,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2269040.0, ans=0.125 2023-11-23 06:24:11,267 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3700, loss[loss=0.06513, simple_loss=0.08311, pruned_loss=0.01427, audio_tagging_loss=0.009311, over 15632.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09405, pruned_loss=0.0145, audio_tagging_loss=0.008955, over 3057061.73 frames. ], batch size: 61, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 06:24:19,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2269106.6666666665, ans=0.125 2023-11-23 06:24:42,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.77 vs. limit=12.0 2023-11-23 06:24:46,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2269240.0, ans=0.125 2023-11-23 06:24:52,637 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340400 2023-11-23 06:24:55,306 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.367e+01 8.831e+01 9.767e+01 1.153e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 06:25:02,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2269373.3333333335, ans=0.1 2023-11-23 06:25:12,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2269373.3333333335, ans=0.025 2023-11-23 06:25:13,855 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:25:16,083 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3750, loss[loss=0.07318, simple_loss=0.1021, pruned_loss=0.01419, audio_tagging_loss=0.007947, over 15322.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09572, pruned_loss=0.01493, audio_tagging_loss=0.008873, over 3069743.97 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:25:24,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.00 vs. limit=22.5 2023-11-23 06:25:27,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2269506.6666666665, ans=0.1 2023-11-23 06:25:57,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340450 2023-11-23 06:25:59,028 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 06:26:20,469 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3800, loss[loss=0.07605, simple_loss=0.1059, pruned_loss=0.016, audio_tagging_loss=0.007089, over 15585.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09534, pruned_loss=0.01492, audio_tagging_loss=0.008944, over 3068827.92 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:26:24,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2269773.3333333335, ans=0.05 2023-11-23 06:26:37,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.91 vs. limit=12.0 2023-11-23 06:26:52,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2269906.6666666665, ans=0.125 2023-11-23 06:27:02,007 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340500 2023-11-23 06:27:04,358 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.380e+01 8.994e+01 9.826e+01 1.163e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 06:27:21,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2270040.0, ans=0.125 2023-11-23 06:27:26,028 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3850, loss[loss=0.07872, simple_loss=0.1136, pruned_loss=0.01579, audio_tagging_loss=0.006137, over 14979.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09462, pruned_loss=0.01466, audio_tagging_loss=0.008981, over 3055093.12 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:27:50,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2270240.0, ans=0.125 2023-11-23 06:27:51,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2270240.0, ans=0.1 2023-11-23 06:27:58,870 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.28 vs. limit=6.0 2023-11-23 06:27:59,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2270240.0, ans=0.125 2023-11-23 06:28:07,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2270306.6666666665, ans=0.0 2023-11-23 06:28:08,206 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340550 2023-11-23 06:28:15,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2270306.6666666665, ans=0.95 2023-11-23 06:28:15,815 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:28:22,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2270373.3333333335, ans=0.0 2023-11-23 06:28:31,090 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3900, loss[loss=0.07989, simple_loss=0.1138, pruned_loss=0.01712, audio_tagging_loss=0.005855, over 15334.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09345, pruned_loss=0.01446, audio_tagging_loss=0.009143, over 3056441.46 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:28:36,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2270440.0, ans=0.125 2023-11-23 06:28:47,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2270506.6666666665, ans=0.09899494936611666 2023-11-23 06:29:08,827 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.88 vs. limit=15.0 2023-11-23 06:29:11,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340600 2023-11-23 06:29:14,351 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.444e+01 8.394e+01 8.868e+01 9.858e+01 1.292e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-23 06:29:25,844 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=15.0 2023-11-23 06:29:35,336 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 3950, loss[loss=0.08224, simple_loss=0.1102, pruned_loss=0.02002, audio_tagging_loss=0.007125, over 15649.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09244, pruned_loss=0.01407, audio_tagging_loss=0.009278, over 3061210.49 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:29:36,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2270773.3333333335, ans=0.125 2023-11-23 06:29:49,628 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.02 vs. limit=15.0 2023-11-23 06:30:05,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2270906.6666666665, ans=0.125 2023-11-23 06:30:08,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2270906.6666666665, ans=0.0 2023-11-23 06:30:16,659 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340650 2023-11-23 06:30:30,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2271040.0, ans=10.0 2023-11-23 06:30:32,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2271040.0, ans=0.2 2023-11-23 06:30:34,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.12 vs. limit=10.0 2023-11-23 06:30:36,430 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.05 vs. limit=10.0 2023-11-23 06:30:40,572 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4000, loss[loss=0.07612, simple_loss=0.1015, pruned_loss=0.01543, audio_tagging_loss=0.009958, over 15932.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09295, pruned_loss=0.01421, audio_tagging_loss=0.009382, over 3049232.23 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:30:49,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.20 vs. limit=15.0 2023-11-23 06:31:05,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2271240.0, ans=0.125 2023-11-23 06:31:22,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340700 2023-11-23 06:31:25,687 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.305e+01 8.425e+01 8.951e+01 9.689e+01 1.650e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-23 06:31:31,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.31 vs. limit=10.0 2023-11-23 06:31:44,118 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4050, loss[loss=0.06624, simple_loss=0.08292, pruned_loss=0.01466, audio_tagging_loss=0.01013, over 15881.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09247, pruned_loss=0.01424, audio_tagging_loss=0.009495, over 3045898.53 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:31:45,422 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 06:31:48,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2271440.0, ans=0.125 2023-11-23 06:31:49,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2271440.0, ans=0.125 2023-11-23 06:31:52,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2271440.0, ans=0.0 2023-11-23 06:31:57,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2271506.6666666665, ans=0.2 2023-11-23 06:32:24,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2271640.0, ans=0.125 2023-11-23 06:32:26,197 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340750 2023-11-23 06:32:28,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2271640.0, ans=0.125 2023-11-23 06:32:36,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2271706.6666666665, ans=0.125 2023-11-23 06:32:48,642 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4100, loss[loss=0.06721, simple_loss=0.09022, pruned_loss=0.01277, audio_tagging_loss=0.009327, over 14597.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.0931, pruned_loss=0.01438, audio_tagging_loss=0.009487, over 3048647.11 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:33:08,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2271840.0, ans=15.0 2023-11-23 06:33:20,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2271906.6666666665, ans=0.1 2023-11-23 06:33:24,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.58 vs. limit=22.5 2023-11-23 06:33:30,039 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340800 2023-11-23 06:33:31,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.21 vs. limit=12.0 2023-11-23 06:33:34,565 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.442e+01 8.896e+01 9.495e+01 1.100e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 06:33:37,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2271973.3333333335, ans=0.125 2023-11-23 06:33:42,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2272040.0, ans=0.1 2023-11-23 06:33:54,754 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4150, loss[loss=0.07726, simple_loss=0.1051, pruned_loss=0.01575, audio_tagging_loss=0.008954, over 15239.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09305, pruned_loss=0.01429, audio_tagging_loss=0.009355, over 3049776.52 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:33:58,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2272106.6666666665, ans=10.0 2023-11-23 06:34:07,629 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0 2023-11-23 06:34:19,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2272240.0, ans=0.0 2023-11-23 06:34:27,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2272240.0, ans=0.1 2023-11-23 06:34:36,357 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340850 2023-11-23 06:34:38,731 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 06:34:44,367 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.32 vs. limit=22.5 2023-11-23 06:34:48,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2272373.3333333335, ans=0.125 2023-11-23 06:34:58,258 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4200, loss[loss=0.07662, simple_loss=0.09881, pruned_loss=0.01904, audio_tagging_loss=0.008173, over 16200.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09349, pruned_loss=0.01437, audio_tagging_loss=0.009087, over 3057895.55 frames. ], batch size: 63, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:34:59,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2272440.0, ans=0.125 2023-11-23 06:35:07,138 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:35:38,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2272640.0, ans=0.05 2023-11-23 06:35:40,241 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340900 2023-11-23 06:35:43,747 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.311e+01 8.855e+01 9.584e+01 1.424e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-23 06:35:50,672 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.44 vs. limit=22.5 2023-11-23 06:36:02,084 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4250, loss[loss=0.06378, simple_loss=0.08558, pruned_loss=0.01224, audio_tagging_loss=0.008747, over 15493.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09408, pruned_loss=0.01464, audio_tagging_loss=0.00907, over 3065605.45 frames. ], batch size: 61, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:36:43,957 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 340950 2023-11-23 06:36:49,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2272973.3333333335, ans=0.125 2023-11-23 06:36:52,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2273040.0, ans=0.0 2023-11-23 06:36:53,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2273040.0, ans=0.0 2023-11-23 06:37:07,799 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4300, loss[loss=0.06501, simple_loss=0.08886, pruned_loss=0.013, audio_tagging_loss=0.007573, over 15728.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09484, pruned_loss=0.01463, audio_tagging_loss=0.008954, over 3065068.97 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:37:49,251 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341000 2023-11-23 06:37:53,080 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.426e+01 8.312e+01 8.993e+01 9.823e+01 1.635e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 06:37:55,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2273306.6666666665, ans=0.2 2023-11-23 06:38:03,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2273373.3333333335, ans=0.2 2023-11-23 06:38:11,582 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4350, loss[loss=0.06146, simple_loss=0.0829, pruned_loss=0.01159, audio_tagging_loss=0.008427, over 14783.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09395, pruned_loss=0.01448, audio_tagging_loss=0.008914, over 3060120.91 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:38:40,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2273573.3333333335, ans=0.0 2023-11-23 06:38:43,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2273573.3333333335, ans=0.1 2023-11-23 06:38:46,893 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.80 vs. limit=15.0 2023-11-23 06:38:46,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.40 vs. limit=15.0 2023-11-23 06:38:52,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341050 2023-11-23 06:38:55,691 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.50 vs. limit=12.0 2023-11-23 06:39:04,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2273706.6666666665, ans=0.125 2023-11-23 06:39:14,644 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4400, loss[loss=0.0846, simple_loss=0.1178, pruned_loss=0.01942, audio_tagging_loss=0.006267, over 14931.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09225, pruned_loss=0.01429, audio_tagging_loss=0.008977, over 3057361.87 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 06:39:42,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2273906.6666666665, ans=0.0 2023-11-23 06:39:49,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.60 vs. limit=15.0 2023-11-23 06:39:55,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341100 2023-11-23 06:40:00,198 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.327e+01 8.740e+01 9.683e+01 1.205e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-23 06:40:00,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2273973.3333333335, ans=0.1 2023-11-23 06:40:10,627 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.28 vs. limit=22.5 2023-11-23 06:40:19,128 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4450, loss[loss=0.05716, simple_loss=0.07371, pruned_loss=0.008735, audio_tagging_loss=0.01157, over 16044.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09247, pruned_loss=0.01423, audio_tagging_loss=0.008937, over 3062300.73 frames. ], batch size: 63, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:40:56,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2274306.6666666665, ans=0.125 2023-11-23 06:40:57,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.93 vs. limit=12.0 2023-11-23 06:40:58,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341150 2023-11-23 06:40:58,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2274306.6666666665, ans=0.0 2023-11-23 06:41:06,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2274306.6666666665, ans=0.125 2023-11-23 06:41:08,055 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.61 vs. limit=6.0 2023-11-23 06:41:22,094 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4500, loss[loss=0.05498, simple_loss=0.06799, pruned_loss=0.00985, audio_tagging_loss=0.01113, over 14792.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09357, pruned_loss=0.01441, audio_tagging_loss=0.008931, over 3057279.76 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:41:24,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0 2023-11-23 06:41:25,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.65 vs. limit=12.0 2023-11-23 06:41:26,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2274440.0, ans=0.125 2023-11-23 06:41:29,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2274440.0, ans=0.0 2023-11-23 06:41:33,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2274506.6666666665, ans=0.1 2023-11-23 06:41:36,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.07 vs. limit=12.0 2023-11-23 06:41:40,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2274506.6666666665, ans=0.125 2023-11-23 06:42:03,425 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341200 2023-11-23 06:42:08,627 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.193e+01 8.685e+01 9.597e+01 1.191e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-23 06:42:15,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2274706.6666666665, ans=0.0 2023-11-23 06:42:25,795 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4550, loss[loss=0.06147, simple_loss=0.07341, pruned_loss=0.01036, audio_tagging_loss=0.01441, over 14715.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09273, pruned_loss=0.01434, audio_tagging_loss=0.009143, over 3047943.51 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:42:26,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2274773.3333333335, ans=0.0 2023-11-23 06:42:54,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2274906.6666666665, ans=10.0 2023-11-23 06:42:55,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.80 vs. limit=10.0 2023-11-23 06:42:55,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2274906.6666666665, ans=0.2 2023-11-23 06:43:06,638 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341250 2023-11-23 06:43:06,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2274973.3333333335, ans=0.125 2023-11-23 06:43:11,436 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 06:43:28,949 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4600, loss[loss=0.06555, simple_loss=0.091, pruned_loss=0.01151, audio_tagging_loss=0.00854, over 14795.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09255, pruned_loss=0.01424, audio_tagging_loss=0.009125, over 3045243.79 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:43:47,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.38 vs. limit=15.0 2023-11-23 06:43:49,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2275173.3333333335, ans=0.125 2023-11-23 06:44:08,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341300 2023-11-23 06:44:09,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2275306.6666666665, ans=0.125 2023-11-23 06:44:13,568 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 8.306e+01 8.944e+01 9.725e+01 1.153e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-23 06:44:13,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2275306.6666666665, ans=10.0 2023-11-23 06:44:26,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2275373.3333333335, ans=0.1 2023-11-23 06:44:32,746 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4650, loss[loss=0.0692, simple_loss=0.08765, pruned_loss=0.01356, audio_tagging_loss=0.01182, over 15287.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09258, pruned_loss=0.01423, audio_tagging_loss=0.00928, over 3041348.55 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:44:34,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2275440.0, ans=0.2 2023-11-23 06:44:36,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2275440.0, ans=0.2 2023-11-23 06:44:44,306 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:45:08,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2275573.3333333335, ans=0.0 2023-11-23 06:45:13,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341350 2023-11-23 06:45:29,440 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.16 vs. limit=15.0 2023-11-23 06:45:35,687 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.84 vs. limit=15.0 2023-11-23 06:45:36,209 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4700, loss[loss=0.08004, simple_loss=0.1101, pruned_loss=0.01855, audio_tagging_loss=0.006458, over 14397.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09272, pruned_loss=0.01432, audio_tagging_loss=0.009374, over 3047318.54 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:46:10,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2275906.6666666665, ans=0.1 2023-11-23 06:46:18,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341400 2023-11-23 06:46:23,121 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.995e+01 8.437e+01 9.019e+01 9.512e+01 1.168e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 06:46:24,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2275973.3333333335, ans=0.04949747468305833 2023-11-23 06:46:40,592 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4750, loss[loss=0.04871, simple_loss=0.05658, pruned_loss=0.00956, audio_tagging_loss=0.01086, over 14890.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09203, pruned_loss=0.01408, audio_tagging_loss=0.009342, over 3057017.39 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:46:55,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2276173.3333333335, ans=0.0 2023-11-23 06:47:17,001 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.28 vs. limit=15.0 2023-11-23 06:47:22,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341450 2023-11-23 06:47:44,594 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.18 vs. limit=15.0 2023-11-23 06:47:46,389 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4800, loss[loss=0.07434, simple_loss=0.1046, pruned_loss=0.01439, audio_tagging_loss=0.007655, over 15445.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09211, pruned_loss=0.01425, audio_tagging_loss=0.009347, over 3059527.47 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 06:47:46,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2276440.0, ans=0.05 2023-11-23 06:47:59,856 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:48:02,186 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 06:48:07,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2276506.6666666665, ans=0.0 2023-11-23 06:48:18,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2276573.3333333335, ans=0.125 2023-11-23 06:48:27,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341500 2023-11-23 06:48:33,942 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.380e+01 8.789e+01 9.333e+01 1.267e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-23 06:48:47,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.11 vs. limit=15.0 2023-11-23 06:48:50,500 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4850, loss[loss=0.07146, simple_loss=0.09558, pruned_loss=0.0125, audio_tagging_loss=0.01117, over 14805.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09194, pruned_loss=0.0141, audio_tagging_loss=0.009382, over 3059290.36 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:49:03,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2276840.0, ans=0.2 2023-11-23 06:49:15,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2276906.6666666665, ans=0.0 2023-11-23 06:49:32,089 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341550 2023-11-23 06:49:43,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2277040.0, ans=0.1 2023-11-23 06:49:49,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2277040.0, ans=0.07 2023-11-23 06:49:53,999 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4900, loss[loss=0.06061, simple_loss=0.08689, pruned_loss=0.009417, audio_tagging_loss=0.007746, over 16390.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09149, pruned_loss=0.01419, audio_tagging_loss=0.009405, over 3042099.38 frames. ], batch size: 62, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:50:01,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2277106.6666666665, ans=0.125 2023-11-23 06:50:35,583 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341600 2023-11-23 06:50:41,947 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.445e+01 8.899e+01 9.661e+01 1.338e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-23 06:50:59,294 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 4950, loss[loss=0.07319, simple_loss=0.1041, pruned_loss=0.01282, audio_tagging_loss=0.008305, over 15561.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09224, pruned_loss=0.01428, audio_tagging_loss=0.009237, over 3046690.37 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:51:22,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2277506.6666666665, ans=0.05 2023-11-23 06:51:23,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2277573.3333333335, ans=0.125 2023-11-23 06:51:37,454 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.54 vs. limit=15.0 2023-11-23 06:51:38,895 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.88 vs. limit=5.0 2023-11-23 06:51:40,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341650 2023-11-23 06:52:03,298 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5000, loss[loss=0.06726, simple_loss=0.08887, pruned_loss=0.01337, audio_tagging_loss=0.009457, over 14169.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09263, pruned_loss=0.01436, audio_tagging_loss=0.009122, over 3039901.37 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:52:14,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2277840.0, ans=0.125 2023-11-23 06:52:17,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2277840.0, ans=0.125 2023-11-23 06:52:26,958 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.08 vs. limit=15.0 2023-11-23 06:52:32,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2277906.6666666665, ans=0.125 2023-11-23 06:52:37,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2277906.6666666665, ans=0.125 2023-11-23 06:52:43,658 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341700 2023-11-23 06:52:43,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2277973.3333333335, ans=0.1 2023-11-23 06:52:50,295 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.065e+01 8.894e+01 1.001e+02 1.433e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 06:52:56,001 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.46 vs. limit=15.0 2023-11-23 06:53:06,459 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5050, loss[loss=0.06663, simple_loss=0.09267, pruned_loss=0.01076, audio_tagging_loss=0.009539, over 15267.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09277, pruned_loss=0.01431, audio_tagging_loss=0.009002, over 3038098.45 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:53:23,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2278173.3333333335, ans=0.0 2023-11-23 06:53:41,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2278240.0, ans=0.2 2023-11-23 06:53:47,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341750 2023-11-23 06:53:59,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2278373.3333333335, ans=0.035 2023-11-23 06:54:08,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2278373.3333333335, ans=0.125 2023-11-23 06:54:09,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.15 vs. limit=15.0 2023-11-23 06:54:10,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2278440.0, ans=0.0 2023-11-23 06:54:10,954 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5100, loss[loss=0.07017, simple_loss=0.09492, pruned_loss=0.01576, audio_tagging_loss=0.006958, over 14236.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09222, pruned_loss=0.01416, audio_tagging_loss=0.009042, over 3032736.01 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:54:24,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2278506.6666666665, ans=0.025 2023-11-23 06:54:25,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0 2023-11-23 06:54:27,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2278506.6666666665, ans=0.0 2023-11-23 06:54:32,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2278506.6666666665, ans=0.125 2023-11-23 06:54:38,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2278573.3333333335, ans=0.95 2023-11-23 06:54:52,243 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341800 2023-11-23 06:54:53,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2278640.0, ans=0.125 2023-11-23 06:54:57,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2278640.0, ans=0.5 2023-11-23 06:54:59,233 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 7.874e+01 8.463e+01 9.175e+01 1.215e+02, threshold=1.693e+02, percent-clipped=0.0 2023-11-23 06:55:04,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2278706.6666666665, ans=0.0 2023-11-23 06:55:05,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2278706.6666666665, ans=0.0 2023-11-23 06:55:15,523 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5150, loss[loss=0.07207, simple_loss=0.09481, pruned_loss=0.0166, audio_tagging_loss=0.00807, over 15388.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09229, pruned_loss=0.01406, audio_tagging_loss=0.009066, over 3031502.06 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:55:27,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2278840.0, ans=0.125 2023-11-23 06:55:27,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2278840.0, ans=0.2 2023-11-23 06:55:57,804 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341850 2023-11-23 06:56:19,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2279106.6666666665, ans=0.035 2023-11-23 06:56:20,354 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5200, loss[loss=0.07991, simple_loss=0.1155, pruned_loss=0.0164, audio_tagging_loss=0.005742, over 15913.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09285, pruned_loss=0.01425, audio_tagging_loss=0.009019, over 3031302.87 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 06:56:20,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2279106.6666666665, ans=0.125 2023-11-23 06:56:26,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2279106.6666666665, ans=0.1 2023-11-23 06:56:26,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2279106.6666666665, ans=0.2 2023-11-23 06:56:34,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2279173.3333333335, ans=0.2 2023-11-23 06:56:36,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2279173.3333333335, ans=0.5 2023-11-23 06:56:42,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2279173.3333333335, ans=0.0 2023-11-23 06:57:01,379 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341900 2023-11-23 06:57:02,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2279306.6666666665, ans=0.0 2023-11-23 06:57:09,212 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.360e+01 8.790e+01 9.384e+01 1.181e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-23 06:57:15,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2279373.3333333335, ans=0.125 2023-11-23 06:57:25,757 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5250, loss[loss=0.08168, simple_loss=0.1195, pruned_loss=0.01666, audio_tagging_loss=0.005255, over 16637.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09247, pruned_loss=0.01408, audio_tagging_loss=0.009003, over 3043896.61 frames. ], batch size: 61, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:57:40,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2279506.6666666665, ans=0.125 2023-11-23 06:57:48,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2279506.6666666665, ans=0.1 2023-11-23 06:57:50,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2279573.3333333335, ans=0.025 2023-11-23 06:57:55,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2279573.3333333335, ans=0.1 2023-11-23 06:58:06,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 341950 2023-11-23 06:58:15,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2279640.0, ans=0.125 2023-11-23 06:58:15,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2279640.0, ans=0.125 2023-11-23 06:58:29,714 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5300, loss[loss=0.07345, simple_loss=0.09763, pruned_loss=0.01412, audio_tagging_loss=0.01051, over 14635.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09331, pruned_loss=0.01412, audio_tagging_loss=0.009004, over 3042803.23 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:58:32,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2279773.3333333335, ans=0.125 2023-11-23 06:58:33,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2279773.3333333335, ans=0.2 2023-11-23 06:58:37,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2279773.3333333335, ans=0.05 2023-11-23 06:59:11,247 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342000 2023-11-23 06:59:14,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2279973.3333333335, ans=0.125 2023-11-23 06:59:18,964 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.762e+01 8.363e+01 8.739e+01 9.478e+01 1.273e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-23 06:59:33,721 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5350, loss[loss=0.0664, simple_loss=0.08849, pruned_loss=0.01438, audio_tagging_loss=0.007768, over 15426.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09219, pruned_loss=0.01379, audio_tagging_loss=0.009076, over 3035411.89 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 06:59:37,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2280106.6666666665, ans=0.125 2023-11-23 06:59:44,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2280106.6666666665, ans=0.125 2023-11-23 06:59:59,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2280240.0, ans=0.1 2023-11-23 07:00:13,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2280306.6666666665, ans=0.05 2023-11-23 07:00:13,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2280306.6666666665, ans=0.0 2023-11-23 07:00:15,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342050 2023-11-23 07:00:21,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2280306.6666666665, ans=0.0 2023-11-23 07:00:32,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2280373.3333333335, ans=0.5 2023-11-23 07:00:38,814 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5400, loss[loss=0.08092, simple_loss=0.1171, pruned_loss=0.01306, audio_tagging_loss=0.009291, over 15201.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09324, pruned_loss=0.014, audio_tagging_loss=0.009018, over 3034140.05 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 07:00:54,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2280506.6666666665, ans=0.125 2023-11-23 07:01:00,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2280506.6666666665, ans=0.125 2023-11-23 07:01:04,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2280573.3333333335, ans=0.125 2023-11-23 07:01:15,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2280640.0, ans=0.125 2023-11-23 07:01:19,583 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342100 2023-11-23 07:01:27,622 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.345e+01 9.025e+01 9.492e+01 1.163e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 07:01:31,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2280706.6666666665, ans=0.125 2023-11-23 07:01:43,287 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5450, loss[loss=0.0727, simple_loss=0.1005, pruned_loss=0.01557, audio_tagging_loss=0.006863, over 16063.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09346, pruned_loss=0.01433, audio_tagging_loss=0.009118, over 3037156.55 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 07:01:47,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2280773.3333333335, ans=0.125 2023-11-23 07:01:54,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2280840.0, ans=0.0 2023-11-23 07:02:00,207 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.93 vs. limit=15.0 2023-11-23 07:02:03,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2280840.0, ans=0.05 2023-11-23 07:02:14,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2280906.6666666665, ans=0.125 2023-11-23 07:02:21,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2280973.3333333335, ans=0.2 2023-11-23 07:02:24,759 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342150 2023-11-23 07:02:41,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2281040.0, ans=0.125 2023-11-23 07:02:46,738 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5500, loss[loss=0.09103, simple_loss=0.1294, pruned_loss=0.01769, audio_tagging_loss=0.008657, over 16547.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09349, pruned_loss=0.01426, audio_tagging_loss=0.009144, over 3036208.99 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 07:03:20,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2281240.0, ans=0.2 2023-11-23 07:03:28,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342200 2023-11-23 07:03:29,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2281306.6666666665, ans=0.5 2023-11-23 07:03:31,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2281306.6666666665, ans=0.07 2023-11-23 07:03:35,745 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.376e+01 8.901e+01 9.455e+01 1.073e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-23 07:03:51,691 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5550, loss[loss=0.0909, simple_loss=0.1178, pruned_loss=0.02106, audio_tagging_loss=0.01094, over 15117.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09362, pruned_loss=0.01437, audio_tagging_loss=0.009275, over 3032064.32 frames. ], batch size: 52, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 07:04:02,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2281440.0, ans=0.125 2023-11-23 07:04:04,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2281506.6666666665, ans=0.1 2023-11-23 07:04:06,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2281506.6666666665, ans=0.125 2023-11-23 07:04:08,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.64 vs. limit=15.0 2023-11-23 07:04:19,092 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.73 vs. limit=8.0 2023-11-23 07:04:25,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2281573.3333333335, ans=0.125 2023-11-23 07:04:31,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342250 2023-11-23 07:04:55,525 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5600, loss[loss=0.07072, simple_loss=0.09163, pruned_loss=0.01603, audio_tagging_loss=0.008877, over 14480.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09419, pruned_loss=0.01435, audio_tagging_loss=0.00933, over 3039989.18 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 07:04:56,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=12.0 2023-11-23 07:04:56,284 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0 2023-11-23 07:05:08,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2281840.0, ans=0.0 2023-11-23 07:05:21,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2281906.6666666665, ans=0.125 2023-11-23 07:05:30,334 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 07:05:36,321 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342300 2023-11-23 07:05:38,737 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 07:05:43,437 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.528e+01 8.225e+01 8.879e+01 9.451e+01 1.680e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-23 07:05:55,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2282040.0, ans=0.1 2023-11-23 07:05:57,946 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5650, loss[loss=0.06371, simple_loss=0.08435, pruned_loss=0.01107, audio_tagging_loss=0.01047, over 15114.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09426, pruned_loss=0.01432, audio_tagging_loss=0.0093, over 3043121.82 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 32.0 2023-11-23 07:06:14,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2282173.3333333335, ans=0.1 2023-11-23 07:06:18,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2282173.3333333335, ans=0.0 2023-11-23 07:06:25,863 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.97 vs. limit=15.0 2023-11-23 07:06:39,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342350 2023-11-23 07:06:48,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2282373.3333333335, ans=0.1 2023-11-23 07:06:55,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2282373.3333333335, ans=0.1 2023-11-23 07:07:00,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2282440.0, ans=0.125 2023-11-23 07:07:01,221 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5700, loss[loss=0.07176, simple_loss=0.09854, pruned_loss=0.01132, audio_tagging_loss=0.01118, over 15641.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.09387, pruned_loss=0.01427, audio_tagging_loss=0.009415, over 3046578.61 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 07:07:41,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342400 2023-11-23 07:07:50,745 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.447e+01 8.964e+01 9.780e+01 1.182e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-23 07:08:00,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2282706.6666666665, ans=0.0 2023-11-23 07:08:06,333 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5750, loss[loss=0.07106, simple_loss=0.09709, pruned_loss=0.01517, audio_tagging_loss=0.007343, over 14936.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09261, pruned_loss=0.01407, audio_tagging_loss=0.009287, over 3051505.50 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 07:08:21,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2282840.0, ans=0.0 2023-11-23 07:08:23,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2282840.0, ans=0.1 2023-11-23 07:08:26,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2282840.0, ans=0.0 2023-11-23 07:08:32,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2282906.6666666665, ans=0.125 2023-11-23 07:08:46,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2282973.3333333335, ans=0.0 2023-11-23 07:08:48,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342450 2023-11-23 07:08:54,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.whiten.whitening_limit, batch_count=2282973.3333333335, ans=12.0 2023-11-23 07:09:00,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.68 vs. limit=15.0 2023-11-23 07:09:07,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2283040.0, ans=0.125 2023-11-23 07:09:09,969 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5800, loss[loss=0.08873, simple_loss=0.116, pruned_loss=0.01978, audio_tagging_loss=0.01097, over 16356.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09275, pruned_loss=0.01415, audio_tagging_loss=0.009207, over 3056708.28 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0 2023-11-23 07:09:12,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2283106.6666666665, ans=0.1 2023-11-23 07:09:18,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2283106.6666666665, ans=0.125 2023-11-23 07:09:23,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2283173.3333333335, ans=0.125 2023-11-23 07:09:34,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.11 vs. limit=15.0 2023-11-23 07:09:51,939 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342500 2023-11-23 07:09:58,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2283306.6666666665, ans=0.1 2023-11-23 07:10:00,545 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.596e+01 8.186e+01 8.952e+01 9.751e+01 1.304e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-23 07:10:00,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2283373.3333333335, ans=0.125 2023-11-23 07:10:14,088 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5850, loss[loss=0.06961, simple_loss=0.08952, pruned_loss=0.01514, audio_tagging_loss=0.009706, over 16791.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09191, pruned_loss=0.01389, audio_tagging_loss=0.009245, over 3049804.12 frames. ], batch size: 66, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:10:29,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2283506.6666666665, ans=0.0 2023-11-23 07:10:31,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2283506.6666666665, ans=0.125 2023-11-23 07:10:31,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2283506.6666666665, ans=0.125 2023-11-23 07:10:44,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2283573.3333333335, ans=0.2 2023-11-23 07:10:47,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2283573.3333333335, ans=0.0 2023-11-23 07:10:53,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2283640.0, ans=10.0 2023-11-23 07:10:55,921 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342550 2023-11-23 07:11:03,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2283640.0, ans=0.125 2023-11-23 07:11:20,067 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5900, loss[loss=0.06825, simple_loss=0.0962, pruned_loss=0.01237, audio_tagging_loss=0.007787, over 14825.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09174, pruned_loss=0.01396, audio_tagging_loss=0.009187, over 3045282.49 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:11:31,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2283840.0, ans=0.0 2023-11-23 07:11:39,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.70 vs. limit=22.5 2023-11-23 07:11:42,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2283840.0, ans=0.125 2023-11-23 07:11:52,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2283906.6666666665, ans=0.125 2023-11-23 07:12:01,274 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342600 2023-11-23 07:12:03,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0 2023-11-23 07:12:10,812 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.390e+01 8.881e+01 9.843e+01 1.391e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-23 07:12:19,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2284040.0, ans=0.0 2023-11-23 07:12:22,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2284040.0, ans=0.2 2023-11-23 07:12:22,433 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.15 vs. limit=15.0 2023-11-23 07:12:24,237 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 5950, loss[loss=0.07586, simple_loss=0.1019, pruned_loss=0.01742, audio_tagging_loss=0.007509, over 16552.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09223, pruned_loss=0.01416, audio_tagging_loss=0.009123, over 3049492.91 frames. ], batch size: 61, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:12:31,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2284106.6666666665, ans=0.125 2023-11-23 07:12:51,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2284240.0, ans=0.09899494936611666 2023-11-23 07:13:05,409 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342650 2023-11-23 07:13:12,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2284306.6666666665, ans=0.125 2023-11-23 07:13:20,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2284373.3333333335, ans=0.05 2023-11-23 07:13:23,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2284373.3333333335, ans=0.125 2023-11-23 07:13:27,476 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6000, loss[loss=0.1003, simple_loss=0.1339, pruned_loss=0.02501, audio_tagging_loss=0.008308, over 14850.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09244, pruned_loss=0.01414, audio_tagging_loss=0.009084, over 3048417.88 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:13:27,477 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 07:13:57,095 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.8880, 0.9869, 3.6065, 3.0271, 2.8765, 3.1733, 3.0211, 3.1859], device='cuda:3') 2023-11-23 07:14:00,460 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9342, 3.7196, 4.8990, 4.4635], device='cuda:3') 2023-11-23 07:14:10,558 INFO [train_asr.py:1253] (3/4) Epoch 29, validation: loss=0.05847, simple_loss=0.05123, pruned_loss=0.005068, audio_tagging_loss=0.02778, over 4681554.00 frames. 2023-11-23 07:14:10,559 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 07:14:12,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2284440.0, ans=0.125 2023-11-23 07:14:21,961 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.99 vs. limit=12.0 2023-11-23 07:14:47,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2284640.0, ans=0.125 2023-11-23 07:14:51,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342700 2023-11-23 07:14:55,082 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 07:15:00,585 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.161e+01 8.758e+01 9.491e+01 1.345e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-23 07:15:13,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2284773.3333333335, ans=0.0 2023-11-23 07:15:14,054 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6050, loss[loss=0.0766, simple_loss=0.1025, pruned_loss=0.0182, audio_tagging_loss=0.007137, over 15682.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09264, pruned_loss=0.01435, audio_tagging_loss=0.009013, over 3052528.74 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:15:55,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342750 2023-11-23 07:16:04,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2285040.0, ans=0.0 2023-11-23 07:16:17,513 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6100, loss[loss=0.06289, simple_loss=0.08339, pruned_loss=0.01185, audio_tagging_loss=0.009344, over 15465.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09212, pruned_loss=0.01432, audio_tagging_loss=0.008996, over 3042792.03 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:16:19,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2285106.6666666665, ans=0.05 2023-11-23 07:16:30,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2285173.3333333335, ans=0.0 2023-11-23 07:16:34,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2285173.3333333335, ans=0.1 2023-11-23 07:16:43,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2285240.0, ans=0.0 2023-11-23 07:16:43,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2285240.0, ans=0.025 2023-11-23 07:16:59,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342800 2023-11-23 07:17:02,960 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.95 vs. limit=6.0 2023-11-23 07:17:03,113 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.95 vs. limit=15.0 2023-11-23 07:17:08,166 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.337e+01 9.020e+01 9.718e+01 1.256e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 07:17:09,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2285373.3333333335, ans=0.0 2023-11-23 07:17:15,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2285373.3333333335, ans=0.0 2023-11-23 07:17:20,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2285373.3333333335, ans=0.0 2023-11-23 07:17:20,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.whiten.whitening_limit, batch_count=2285373.3333333335, ans=15.0 2023-11-23 07:17:22,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2285440.0, ans=0.1 2023-11-23 07:17:23,757 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6150, loss[loss=0.08904, simple_loss=0.1235, pruned_loss=0.01725, audio_tagging_loss=0.01006, over 14979.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.0916, pruned_loss=0.01414, audio_tagging_loss=0.009119, over 3038436.10 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:17:26,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2285440.0, ans=0.0 2023-11-23 07:17:35,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2285506.6666666665, ans=0.125 2023-11-23 07:17:40,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2285506.6666666665, ans=0.125 2023-11-23 07:17:48,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2285573.3333333335, ans=0.0 2023-11-23 07:17:54,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2285573.3333333335, ans=0.125 2023-11-23 07:17:59,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=2285573.3333333335, ans=0.1 2023-11-23 07:18:05,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342850 2023-11-23 07:18:11,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2285640.0, ans=0.0 2023-11-23 07:18:28,761 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6200, loss[loss=0.07472, simple_loss=0.09521, pruned_loss=0.0178, audio_tagging_loss=0.009322, over 14907.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09252, pruned_loss=0.01432, audio_tagging_loss=0.009128, over 3044717.02 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:18:33,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2285773.3333333335, ans=0.1 2023-11-23 07:18:36,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2285773.3333333335, ans=0.2 2023-11-23 07:18:44,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2285840.0, ans=0.0 2023-11-23 07:18:54,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2285906.6666666665, ans=0.0 2023-11-23 07:19:09,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342900 2023-11-23 07:19:19,547 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.276e+01 8.850e+01 9.566e+01 1.179e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-23 07:19:19,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2286040.0, ans=0.0 2023-11-23 07:19:23,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2286040.0, ans=0.125 2023-11-23 07:19:32,011 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6250, loss[loss=0.07452, simple_loss=0.103, pruned_loss=0.01686, audio_tagging_loss=0.00616, over 15410.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09275, pruned_loss=0.01439, audio_tagging_loss=0.009132, over 3045151.99 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:19:38,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.39 vs. limit=12.0 2023-11-23 07:19:42,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2286106.6666666665, ans=0.0 2023-11-23 07:19:51,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2286173.3333333335, ans=0.125 2023-11-23 07:20:09,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=2286240.0, ans=0.1 2023-11-23 07:20:14,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 342950 2023-11-23 07:20:37,726 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6300, loss[loss=0.06946, simple_loss=0.09648, pruned_loss=0.01292, audio_tagging_loss=0.008304, over 16416.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09255, pruned_loss=0.01427, audio_tagging_loss=0.009226, over 3053726.81 frames. ], batch size: 61, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:20:53,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2286506.6666666665, ans=0.125 2023-11-23 07:21:18,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343000 2023-11-23 07:21:25,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.94 vs. limit=15.0 2023-11-23 07:21:29,496 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.995e+01 8.227e+01 8.680e+01 9.495e+01 1.255e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-23 07:21:30,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2286706.6666666665, ans=0.125 2023-11-23 07:21:38,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2286706.6666666665, ans=0.1 2023-11-23 07:21:42,451 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6350, loss[loss=0.06674, simple_loss=0.09179, pruned_loss=0.01225, audio_tagging_loss=0.008604, over 16794.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09155, pruned_loss=0.01409, audio_tagging_loss=0.009334, over 3055625.33 frames. ], batch size: 62, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:21:56,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2286840.0, ans=0.125 2023-11-23 07:22:23,160 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.74 vs. limit=15.0 2023-11-23 07:22:23,585 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343050 2023-11-23 07:22:24,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2286973.3333333335, ans=0.125 2023-11-23 07:22:46,268 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6400, loss[loss=0.06418, simple_loss=0.08938, pruned_loss=0.009122, audio_tagging_loss=0.01037, over 15506.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09234, pruned_loss=0.01431, audio_tagging_loss=0.009415, over 3056032.86 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:22:53,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2287106.6666666665, ans=0.125 2023-11-23 07:23:03,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.30 vs. limit=15.0 2023-11-23 07:23:09,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2287173.3333333335, ans=0.125 2023-11-23 07:23:10,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2287173.3333333335, ans=0.1 2023-11-23 07:23:17,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2287240.0, ans=0.2 2023-11-23 07:23:26,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2287306.6666666665, ans=0.1 2023-11-23 07:23:28,283 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343100 2023-11-23 07:23:31,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2287306.6666666665, ans=0.125 2023-11-23 07:23:37,930 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.257e+01 8.871e+01 9.376e+01 1.262e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-23 07:23:51,289 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6450, loss[loss=0.06129, simple_loss=0.07708, pruned_loss=0.01358, audio_tagging_loss=0.009168, over 15846.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09261, pruned_loss=0.01436, audio_tagging_loss=0.009394, over 3051664.57 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:23:56,165 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.30 vs. limit=15.0 2023-11-23 07:24:01,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2287440.0, ans=0.0 2023-11-23 07:24:13,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2287506.6666666665, ans=0.125 2023-11-23 07:24:28,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2287640.0, ans=0.1 2023-11-23 07:24:32,190 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343150 2023-11-23 07:24:34,064 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.75 vs. limit=15.0 2023-11-23 07:24:43,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2287706.6666666665, ans=0.125 2023-11-23 07:24:55,842 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6500, loss[loss=0.05375, simple_loss=0.07221, pruned_loss=0.009636, audio_tagging_loss=0.008016, over 14515.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09255, pruned_loss=0.01433, audio_tagging_loss=0.009338, over 3049065.43 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:25:03,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2287773.3333333335, ans=0.125 2023-11-23 07:25:16,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2287840.0, ans=0.125 2023-11-23 07:25:31,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2287906.6666666665, ans=0.1 2023-11-23 07:25:34,508 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.50 vs. limit=15.0 2023-11-23 07:25:37,694 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343200 2023-11-23 07:25:45,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2287973.3333333335, ans=0.125 2023-11-23 07:25:47,698 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.151e+01 8.781e+01 9.682e+01 1.216e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-23 07:25:53,368 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=15.0 2023-11-23 07:25:58,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2288040.0, ans=0.125 2023-11-23 07:26:00,748 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6550, loss[loss=0.07306, simple_loss=0.09549, pruned_loss=0.0162, audio_tagging_loss=0.009119, over 14286.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09154, pruned_loss=0.0141, audio_tagging_loss=0.009351, over 3042763.05 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:26:34,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.40 vs. limit=22.5 2023-11-23 07:26:39,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2288306.6666666665, ans=0.5 2023-11-23 07:26:41,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2288306.6666666665, ans=0.0 2023-11-23 07:26:42,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343250 2023-11-23 07:26:49,641 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.51 vs. limit=12.0 2023-11-23 07:27:05,410 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6600, loss[loss=0.0839, simple_loss=0.1158, pruned_loss=0.01775, audio_tagging_loss=0.008223, over 14687.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09149, pruned_loss=0.01405, audio_tagging_loss=0.009219, over 3039472.94 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:27:11,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2288440.0, ans=0.0 2023-11-23 07:27:43,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2288640.0, ans=0.2 2023-11-23 07:27:46,974 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343300 2023-11-23 07:27:59,916 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.340e+01 9.003e+01 9.613e+01 1.234e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 07:28:11,238 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6650, loss[loss=0.0816, simple_loss=0.1093, pruned_loss=0.01912, audio_tagging_loss=0.007834, over 15394.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09044, pruned_loss=0.01386, audio_tagging_loss=0.009132, over 3038277.77 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:28:11,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2288773.3333333335, ans=0.125 2023-11-23 07:28:21,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2288773.3333333335, ans=0.0 2023-11-23 07:28:30,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2288840.0, ans=0.0 2023-11-23 07:28:48,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2288906.6666666665, ans=0.0 2023-11-23 07:28:52,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343350 2023-11-23 07:29:02,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2289040.0, ans=0.125 2023-11-23 07:29:14,909 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6700, loss[loss=0.07987, simple_loss=0.1051, pruned_loss=0.01788, audio_tagging_loss=0.009448, over 16051.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09083, pruned_loss=0.01382, audio_tagging_loss=0.009041, over 3044407.08 frames. ], batch size: 61, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:29:45,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2289240.0, ans=0.125 2023-11-23 07:29:55,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2289306.6666666665, ans=0.2 2023-11-23 07:29:56,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343400 2023-11-23 07:30:06,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2289373.3333333335, ans=0.125 2023-11-23 07:30:07,563 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.148e+01 8.906e+01 9.681e+01 1.677e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-23 07:30:17,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2289373.3333333335, ans=0.2 2023-11-23 07:30:19,390 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6750, loss[loss=0.06188, simple_loss=0.07486, pruned_loss=0.01224, audio_tagging_loss=0.01221, over 15535.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09198, pruned_loss=0.01407, audio_tagging_loss=0.00896, over 3039292.37 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:30:19,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2289440.0, ans=0.1 2023-11-23 07:30:27,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2289440.0, ans=0.1 2023-11-23 07:30:30,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.58 vs. limit=22.5 2023-11-23 07:30:31,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2289440.0, ans=0.125 2023-11-23 07:30:37,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2289506.6666666665, ans=0.0 2023-11-23 07:30:57,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.24 vs. limit=15.0 2023-11-23 07:31:00,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343450 2023-11-23 07:31:01,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2289640.0, ans=0.0 2023-11-23 07:31:07,340 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 07:31:21,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2289706.6666666665, ans=0.125 2023-11-23 07:31:24,912 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6800, loss[loss=0.07364, simple_loss=0.09933, pruned_loss=0.01467, audio_tagging_loss=0.009302, over 15443.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09167, pruned_loss=0.01401, audio_tagging_loss=0.009036, over 3040877.80 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:31:40,252 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.80 vs. limit=6.0 2023-11-23 07:31:42,766 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.75 vs. limit=15.0 2023-11-23 07:31:51,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.31 vs. limit=15.0 2023-11-23 07:32:06,730 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343500 2023-11-23 07:32:13,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2289973.3333333335, ans=0.125 2023-11-23 07:32:17,721 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.151e+01 8.874e+01 9.532e+01 1.488e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-23 07:32:18,254 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 07:32:18,821 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.17 vs. limit=8.0 2023-11-23 07:32:28,858 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6850, loss[loss=0.08163, simple_loss=0.1071, pruned_loss=0.02007, audio_tagging_loss=0.008004, over 15015.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.0917, pruned_loss=0.0139, audio_tagging_loss=0.009044, over 3038935.08 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:32:31,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2290106.6666666665, ans=0.0 2023-11-23 07:32:34,359 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.47 vs. limit=15.0 2023-11-23 07:32:43,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2290173.3333333335, ans=0.1 2023-11-23 07:33:11,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343550 2023-11-23 07:33:30,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2290373.3333333335, ans=0.125 2023-11-23 07:33:33,155 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6900, loss[loss=0.05849, simple_loss=0.07981, pruned_loss=0.01083, audio_tagging_loss=0.00776, over 15445.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09169, pruned_loss=0.0139, audio_tagging_loss=0.009032, over 3045723.57 frames. ], batch size: 61, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:33:42,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2290440.0, ans=0.1 2023-11-23 07:33:51,189 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 07:33:56,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.82 vs. limit=15.0 2023-11-23 07:33:57,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2290506.6666666665, ans=0.2 2023-11-23 07:34:14,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343600 2023-11-23 07:34:22,074 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 07:34:26,251 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.590e+01 8.208e+01 8.939e+01 9.660e+01 1.242e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-23 07:34:31,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2290706.6666666665, ans=0.125 2023-11-23 07:34:39,264 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 6950, loss[loss=0.07711, simple_loss=0.1041, pruned_loss=0.01828, audio_tagging_loss=0.006768, over 15602.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.092, pruned_loss=0.01395, audio_tagging_loss=0.009091, over 3055036.55 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:34:48,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2290773.3333333335, ans=0.0 2023-11-23 07:34:50,951 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.40 vs. limit=22.5 2023-11-23 07:34:54,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2290840.0, ans=0.0 2023-11-23 07:35:00,592 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.91 vs. limit=15.0 2023-11-23 07:35:09,501 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.31 vs. limit=15.0 2023-11-23 07:35:19,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343650 2023-11-23 07:35:26,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2290973.3333333335, ans=0.125 2023-11-23 07:35:26,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2290973.3333333335, ans=0.0 2023-11-23 07:35:29,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2291040.0, ans=0.1 2023-11-23 07:35:40,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2291040.0, ans=0.125 2023-11-23 07:35:42,394 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7000, loss[loss=0.05865, simple_loss=0.06825, pruned_loss=0.01305, audio_tagging_loss=0.01148, over 14518.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.0917, pruned_loss=0.01401, audio_tagging_loss=0.009194, over 3053303.51 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:35:45,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.26 vs. limit=15.0 2023-11-23 07:35:47,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2291106.6666666665, ans=0.2 2023-11-23 07:35:51,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.63 vs. limit=15.0 2023-11-23 07:35:56,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2291173.3333333335, ans=0.125 2023-11-23 07:36:05,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2291173.3333333335, ans=0.5 2023-11-23 07:36:19,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2291240.0, ans=0.0 2023-11-23 07:36:24,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343700 2023-11-23 07:36:34,904 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.255e+01 8.925e+01 9.735e+01 1.377e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 07:36:37,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2291373.3333333335, ans=0.0 2023-11-23 07:36:45,878 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7050, loss[loss=0.08156, simple_loss=0.1156, pruned_loss=0.01853, audio_tagging_loss=0.00524, over 15484.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09197, pruned_loss=0.01399, audio_tagging_loss=0.009157, over 3056336.08 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:36:50,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2291440.0, ans=0.2 2023-11-23 07:37:02,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2291506.6666666665, ans=0.125 2023-11-23 07:37:15,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2291573.3333333335, ans=0.125 2023-11-23 07:37:27,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343750 2023-11-23 07:37:52,040 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7100, loss[loss=0.09659, simple_loss=0.1323, pruned_loss=0.02214, audio_tagging_loss=0.008283, over 16405.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09121, pruned_loss=0.01382, audio_tagging_loss=0.009324, over 3054551.25 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:38:32,225 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343800 2023-11-23 07:38:35,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2291973.3333333335, ans=0.2 2023-11-23 07:38:36,192 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.57 vs. limit=22.5 2023-11-23 07:38:41,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2291973.3333333335, ans=0.125 2023-11-23 07:38:45,444 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.319e+01 8.862e+01 9.684e+01 1.135e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 07:38:56,648 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7150, loss[loss=0.06583, simple_loss=0.08533, pruned_loss=0.01268, audio_tagging_loss=0.01049, over 14989.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09095, pruned_loss=0.01383, audio_tagging_loss=0.00942, over 3052313.99 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:39:20,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2292240.0, ans=0.125 2023-11-23 07:39:28,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2292240.0, ans=0.1 2023-11-23 07:39:32,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.15 vs. limit=15.0 2023-11-23 07:39:38,222 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343850 2023-11-23 07:39:46,043 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.06 vs. limit=15.0 2023-11-23 07:39:49,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.44 vs. limit=15.0 2023-11-23 07:39:52,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2292373.3333333335, ans=0.125 2023-11-23 07:39:54,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2292373.3333333335, ans=0.2 2023-11-23 07:40:00,110 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7200, loss[loss=0.06019, simple_loss=0.07776, pruned_loss=0.009136, audio_tagging_loss=0.01218, over 14913.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.0919, pruned_loss=0.01404, audio_tagging_loss=0.009369, over 3047617.60 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:40:15,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2292506.6666666665, ans=0.125 2023-11-23 07:40:20,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2292506.6666666665, ans=22.5 2023-11-23 07:40:22,827 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0 2023-11-23 07:40:33,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2292573.3333333335, ans=0.0 2023-11-23 07:40:41,592 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343900 2023-11-23 07:40:52,466 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.353e+01 8.830e+01 9.779e+01 1.772e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 07:40:54,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.13 vs. limit=10.0 2023-11-23 07:40:58,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2292706.6666666665, ans=0.125 2023-11-23 07:40:58,531 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 07:41:05,598 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7250, loss[loss=0.08393, simple_loss=0.1162, pruned_loss=0.01828, audio_tagging_loss=0.007567, over 14928.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09242, pruned_loss=0.01416, audio_tagging_loss=0.009352, over 3050100.86 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:41:11,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.74 vs. limit=15.0 2023-11-23 07:41:22,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2292840.0, ans=0.125 2023-11-23 07:41:25,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2292840.0, ans=0.125 2023-11-23 07:41:37,685 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.35 vs. limit=15.0 2023-11-23 07:41:45,477 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 343950 2023-11-23 07:41:48,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2292973.3333333335, ans=0.1 2023-11-23 07:42:07,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0 2023-11-23 07:42:10,327 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7300, loss[loss=0.06456, simple_loss=0.08152, pruned_loss=0.01318, audio_tagging_loss=0.01063, over 14591.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.0928, pruned_loss=0.01416, audio_tagging_loss=0.009235, over 3044864.36 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:42:13,168 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 07:42:19,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2293106.6666666665, ans=0.125 2023-11-23 07:42:24,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2293173.3333333335, ans=0.125 2023-11-23 07:42:47,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2293306.6666666665, ans=0.0 2023-11-23 07:42:51,457 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344000 2023-11-23 07:42:59,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2293306.6666666665, ans=0.0 2023-11-23 07:43:06,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.353e+01 8.994e+01 9.658e+01 1.142e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 07:43:17,903 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7350, loss[loss=0.06928, simple_loss=0.08493, pruned_loss=0.01842, audio_tagging_loss=0.008392, over 15762.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09322, pruned_loss=0.01436, audio_tagging_loss=0.009096, over 3041948.78 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:43:38,871 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.32 vs. limit=15.0 2023-11-23 07:43:39,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2293506.6666666665, ans=0.125 2023-11-23 07:43:40,356 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.36 vs. limit=15.0 2023-11-23 07:43:51,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2293573.3333333335, ans=0.2 2023-11-23 07:43:55,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2293573.3333333335, ans=0.125 2023-11-23 07:44:00,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344050 2023-11-23 07:44:06,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2293640.0, ans=0.0 2023-11-23 07:44:12,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2293706.6666666665, ans=0.125 2023-11-23 07:44:16,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2293706.6666666665, ans=0.0 2023-11-23 07:44:23,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.19 vs. limit=10.0 2023-11-23 07:44:23,612 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7400, loss[loss=0.06391, simple_loss=0.0875, pruned_loss=0.01282, audio_tagging_loss=0.00734, over 15741.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09319, pruned_loss=0.01422, audio_tagging_loss=0.00898, over 3042653.86 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:44:28,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2293773.3333333335, ans=0.125 2023-11-23 07:44:29,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2293773.3333333335, ans=0.2 2023-11-23 07:44:32,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2293773.3333333335, ans=0.125 2023-11-23 07:44:47,584 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.21 vs. limit=15.0 2023-11-23 07:44:48,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2293906.6666666665, ans=0.125 2023-11-23 07:45:04,052 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344100 2023-11-23 07:45:06,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2293973.3333333335, ans=0.0 2023-11-23 07:45:17,737 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.228e+01 8.684e+01 9.636e+01 1.290e+02, threshold=1.737e+02, percent-clipped=0.0 2023-11-23 07:45:28,119 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7450, loss[loss=0.08191, simple_loss=0.1124, pruned_loss=0.01815, audio_tagging_loss=0.007537, over 16590.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09357, pruned_loss=0.01424, audio_tagging_loss=0.008857, over 3045558.55 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:45:46,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2294173.3333333335, ans=0.2 2023-11-23 07:46:04,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2294240.0, ans=0.0 2023-11-23 07:46:05,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2294306.6666666665, ans=0.035 2023-11-23 07:46:05,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2294306.6666666665, ans=0.1 2023-11-23 07:46:09,189 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344150 2023-11-23 07:46:14,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2294306.6666666665, ans=0.1 2023-11-23 07:46:15,669 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=16.74 vs. limit=15.0 2023-11-23 07:46:24,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2294373.3333333335, ans=0.125 2023-11-23 07:46:31,667 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7500, loss[loss=0.06487, simple_loss=0.08442, pruned_loss=0.01317, audio_tagging_loss=0.009493, over 15740.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09412, pruned_loss=0.01435, audio_tagging_loss=0.008955, over 3044180.56 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:47:00,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2294573.3333333335, ans=0.125 2023-11-23 07:47:13,261 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344200 2023-11-23 07:47:20,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2294640.0, ans=0.0 2023-11-23 07:47:25,792 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.286e+01 8.811e+01 9.468e+01 1.775e+02, threshold=1.762e+02, percent-clipped=1.0 2023-11-23 07:47:36,018 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7550, loss[loss=0.06029, simple_loss=0.07977, pruned_loss=0.0123, audio_tagging_loss=0.008114, over 16560.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.0951, pruned_loss=0.01459, audio_tagging_loss=0.008832, over 3046405.53 frames. ], batch size: 62, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:47:43,738 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 07:47:43,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2294773.3333333335, ans=0.125 2023-11-23 07:47:46,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2294773.3333333335, ans=0.125 2023-11-23 07:48:17,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344250 2023-11-23 07:48:21,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2294973.3333333335, ans=0.0 2023-11-23 07:48:24,180 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.42 vs. limit=15.0 2023-11-23 07:48:25,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2294973.3333333335, ans=0.125 2023-11-23 07:48:25,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2294973.3333333335, ans=0.0 2023-11-23 07:48:41,121 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7600, loss[loss=0.05748, simple_loss=0.07674, pruned_loss=0.01018, audio_tagging_loss=0.008931, over 15047.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09488, pruned_loss=0.01454, audio_tagging_loss=0.008891, over 3040129.53 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:48:47,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2295106.6666666665, ans=0.125 2023-11-23 07:49:06,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2295240.0, ans=0.1 2023-11-23 07:49:22,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344300 2023-11-23 07:49:24,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2295306.6666666665, ans=0.07 2023-11-23 07:49:35,026 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.274e+01 8.656e+01 9.484e+01 1.340e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-23 07:49:45,394 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7650, loss[loss=0.05877, simple_loss=0.0795, pruned_loss=0.009073, audio_tagging_loss=0.009952, over 15069.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09335, pruned_loss=0.01414, audio_tagging_loss=0.008892, over 3038167.21 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:49:47,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2295440.0, ans=0.125 2023-11-23 07:49:54,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2295440.0, ans=0.0 2023-11-23 07:50:25,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2295640.0, ans=0.1 2023-11-23 07:50:26,586 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344350 2023-11-23 07:50:29,481 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.49 vs. limit=22.5 2023-11-23 07:50:39,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2295706.6666666665, ans=0.125 2023-11-23 07:50:48,880 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7700, loss[loss=0.08164, simple_loss=0.1082, pruned_loss=0.01979, audio_tagging_loss=0.007756, over 15229.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09282, pruned_loss=0.01413, audio_tagging_loss=0.008875, over 3034610.39 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:50:56,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2295773.3333333335, ans=0.0 2023-11-23 07:51:03,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2295840.0, ans=0.125 2023-11-23 07:51:24,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2295906.6666666665, ans=0.1 2023-11-23 07:51:30,224 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344400 2023-11-23 07:51:34,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2295973.3333333335, ans=0.125 2023-11-23 07:51:44,895 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.308e+01 8.349e+01 8.922e+01 9.569e+01 1.181e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 07:51:54,184 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7750, loss[loss=0.06233, simple_loss=0.07923, pruned_loss=0.01162, audio_tagging_loss=0.0111, over 15326.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09266, pruned_loss=0.0141, audio_tagging_loss=0.008984, over 3035025.65 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:52:11,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2296173.3333333335, ans=0.125 2023-11-23 07:52:14,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=15.0 2023-11-23 07:52:20,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2296240.0, ans=0.125 2023-11-23 07:52:25,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2296240.0, ans=0.0 2023-11-23 07:52:35,729 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344450 2023-11-23 07:52:41,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.16 vs. limit=22.5 2023-11-23 07:52:57,561 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7800, loss[loss=0.07776, simple_loss=0.1085, pruned_loss=0.01452, audio_tagging_loss=0.009011, over 16420.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09329, pruned_loss=0.01421, audio_tagging_loss=0.009093, over 3039898.55 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:52:57,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2296440.0, ans=0.2 2023-11-23 07:53:07,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2296440.0, ans=0.125 2023-11-23 07:53:09,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2296506.6666666665, ans=0.1 2023-11-23 07:53:17,512 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.66 vs. limit=10.0 2023-11-23 07:53:18,588 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.54 vs. limit=10.0 2023-11-23 07:53:21,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2296506.6666666665, ans=0.0 2023-11-23 07:53:26,706 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0 2023-11-23 07:53:39,383 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344500 2023-11-23 07:53:48,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2296706.6666666665, ans=0.125 2023-11-23 07:53:49,682 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.86 vs. limit=15.0 2023-11-23 07:53:53,402 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.147e+01 8.634e+01 9.399e+01 1.178e+02, threshold=1.727e+02, percent-clipped=0.0 2023-11-23 07:54:01,882 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7850, loss[loss=0.09609, simple_loss=0.1285, pruned_loss=0.02649, audio_tagging_loss=0.005354, over 15317.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09372, pruned_loss=0.01419, audio_tagging_loss=0.009085, over 3042974.03 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:54:28,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2296906.6666666665, ans=0.2 2023-11-23 07:54:31,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.80 vs. limit=15.0 2023-11-23 07:54:35,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2296906.6666666665, ans=0.125 2023-11-23 07:54:43,726 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344550 2023-11-23 07:54:56,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2297040.0, ans=0.125 2023-11-23 07:55:07,319 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7900, loss[loss=0.0756, simple_loss=0.1047, pruned_loss=0.01503, audio_tagging_loss=0.008202, over 15644.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09336, pruned_loss=0.01415, audio_tagging_loss=0.009169, over 3048795.50 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:55:39,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2297240.0, ans=0.125 2023-11-23 07:55:48,169 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344600 2023-11-23 07:55:57,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=2297306.6666666665, ans=0.1 2023-11-23 07:55:58,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2297373.3333333335, ans=0.125 2023-11-23 07:56:02,997 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.370e+01 9.081e+01 9.662e+01 1.223e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-23 07:56:05,030 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.21 vs. limit=15.0 2023-11-23 07:56:07,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2297373.3333333335, ans=0.0 2023-11-23 07:56:11,613 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 7950, loss[loss=0.08327, simple_loss=0.1197, pruned_loss=0.01613, audio_tagging_loss=0.0073, over 15155.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09315, pruned_loss=0.01414, audio_tagging_loss=0.009248, over 3045356.89 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 07:56:26,813 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 07:56:40,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2297573.3333333335, ans=0.125 2023-11-23 07:56:46,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2297573.3333333335, ans=0.125 2023-11-23 07:56:52,887 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344650 2023-11-23 07:57:14,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2297773.3333333335, ans=0.125 2023-11-23 07:57:15,728 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8000, loss[loss=0.04381, simple_loss=0.05063, pruned_loss=0.007349, audio_tagging_loss=0.01114, over 14430.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09274, pruned_loss=0.01399, audio_tagging_loss=0.009274, over 3043891.80 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:57:19,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2297773.3333333335, ans=0.125 2023-11-23 07:57:32,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2297840.0, ans=0.125 2023-11-23 07:57:34,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2297840.0, ans=0.2 2023-11-23 07:57:34,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2297840.0, ans=0.0 2023-11-23 07:57:34,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2297840.0, ans=0.125 2023-11-23 07:57:37,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2297840.0, ans=0.125 2023-11-23 07:57:38,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2297840.0, ans=0.0 2023-11-23 07:57:42,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2297906.6666666665, ans=0.0 2023-11-23 07:57:42,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2297906.6666666665, ans=15.0 2023-11-23 07:57:46,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2297906.6666666665, ans=0.1 2023-11-23 07:57:56,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344700 2023-11-23 07:58:10,964 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.163e+01 8.824e+01 9.486e+01 1.093e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-23 07:58:17,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2298040.0, ans=0.125 2023-11-23 07:58:21,511 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8050, loss[loss=0.06634, simple_loss=0.0853, pruned_loss=0.01375, audio_tagging_loss=0.009937, over 14884.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09263, pruned_loss=0.01406, audio_tagging_loss=0.009368, over 3047836.40 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:58:29,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2298106.6666666665, ans=0.04949747468305833 2023-11-23 07:58:41,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2298173.3333333335, ans=0.125 2023-11-23 07:58:52,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2298240.0, ans=0.125 2023-11-23 07:59:02,662 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344750 2023-11-23 07:59:03,076 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.26 vs. limit=15.0 2023-11-23 07:59:16,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2298373.3333333335, ans=0.125 2023-11-23 07:59:25,551 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8100, loss[loss=0.05409, simple_loss=0.07834, pruned_loss=0.0074, audio_tagging_loss=0.007521, over 14539.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09289, pruned_loss=0.01415, audio_tagging_loss=0.009272, over 3037216.55 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 07:59:36,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2298506.6666666665, ans=0.2 2023-11-23 07:59:46,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2298506.6666666665, ans=0.0 2023-11-23 08:00:07,390 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344800 2023-11-23 08:00:22,318 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.280e+01 8.423e+01 8.748e+01 9.445e+01 1.319e+02, threshold=1.750e+02, percent-clipped=0.0 2023-11-23 08:00:27,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2298706.6666666665, ans=0.125 2023-11-23 08:00:27,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2298706.6666666665, ans=0.125 2023-11-23 08:00:29,757 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8150, loss[loss=0.053, simple_loss=0.06193, pruned_loss=0.009561, audio_tagging_loss=0.01248, over 14824.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09198, pruned_loss=0.01399, audio_tagging_loss=0.009258, over 3034455.97 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 08:00:37,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2298773.3333333335, ans=0.125 2023-11-23 08:01:01,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2298906.6666666665, ans=0.125 2023-11-23 08:01:02,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2298906.6666666665, ans=0.2 2023-11-23 08:01:07,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2298973.3333333335, ans=0.1 2023-11-23 08:01:11,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344850 2023-11-23 08:01:30,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2299040.0, ans=0.2 2023-11-23 08:01:34,472 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8200, loss[loss=0.06523, simple_loss=0.08703, pruned_loss=0.01287, audio_tagging_loss=0.008845, over 14788.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09223, pruned_loss=0.01397, audio_tagging_loss=0.009064, over 3039307.95 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 08:01:35,190 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:01:42,964 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.55 vs. limit=15.0 2023-11-23 08:01:44,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2299106.6666666665, ans=0.07 2023-11-23 08:01:44,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2299106.6666666665, ans=0.125 2023-11-23 08:01:52,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2299173.3333333335, ans=0.125 2023-11-23 08:01:58,913 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.93 vs. limit=22.5 2023-11-23 08:02:08,901 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.65 vs. limit=15.0 2023-11-23 08:02:09,102 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0 2023-11-23 08:02:14,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344900 2023-11-23 08:02:31,498 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.892e+01 8.311e+01 8.960e+01 9.864e+01 1.172e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-23 08:02:39,004 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8250, loss[loss=0.07754, simple_loss=0.109, pruned_loss=0.01739, audio_tagging_loss=0.005635, over 16780.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09197, pruned_loss=0.01396, audio_tagging_loss=0.008979, over 3041912.61 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 08:02:54,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2299506.6666666665, ans=0.125 2023-11-23 08:03:07,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=15.0 2023-11-23 08:03:21,215 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 344950 2023-11-23 08:03:23,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2299640.0, ans=0.125 2023-11-23 08:03:42,987 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8300, loss[loss=0.05385, simple_loss=0.07047, pruned_loss=0.01035, audio_tagging_loss=0.008269, over 16614.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09012, pruned_loss=0.0137, audio_tagging_loss=0.00904, over 3054078.35 frames. ], batch size: 64, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 08:03:44,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2299773.3333333335, ans=0.0 2023-11-23 08:03:51,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2299773.3333333335, ans=0.5 2023-11-23 08:04:09,444 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:04:18,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2299906.6666666665, ans=0.125 2023-11-23 08:04:22,727 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:04:23,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345000 2023-11-23 08:04:24,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2299973.3333333335, ans=0.0 2023-11-23 08:04:38,781 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.704e+01 8.216e+01 8.783e+01 9.423e+01 1.176e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-23 08:04:46,574 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8350, loss[loss=0.07694, simple_loss=0.1074, pruned_loss=0.01496, audio_tagging_loss=0.008288, over 16357.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09116, pruned_loss=0.01378, audio_tagging_loss=0.008933, over 3052076.75 frames. ], batch size: 62, lr: 2.34e-03, grad_scale: 16.0 2023-11-23 08:04:54,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2300106.6666666665, ans=0.2 2023-11-23 08:05:11,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.10 vs. limit=6.0 2023-11-23 08:05:21,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.44 vs. limit=15.0 2023-11-23 08:05:27,865 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345050 2023-11-23 08:05:33,838 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.02 vs. limit=6.0 2023-11-23 08:05:51,648 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8400, loss[loss=0.06148, simple_loss=0.07577, pruned_loss=0.01554, audio_tagging_loss=0.008055, over 14883.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.0913, pruned_loss=0.01378, audio_tagging_loss=0.009015, over 3048427.20 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:05:52,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2300440.0, ans=0.125 2023-11-23 08:05:58,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2300440.0, ans=0.125 2023-11-23 08:06:02,179 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.31 vs. limit=15.0 2023-11-23 08:06:03,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2300506.6666666665, ans=0.0 2023-11-23 08:06:05,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2300506.6666666665, ans=0.125 2023-11-23 08:06:08,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2300506.6666666665, ans=0.0 2023-11-23 08:06:15,588 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:06:18,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2300573.3333333335, ans=0.1 2023-11-23 08:06:26,513 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:06:33,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345100 2023-11-23 08:06:45,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2300706.6666666665, ans=0.5 2023-11-23 08:06:47,800 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.105e+01 8.014e+01 8.846e+01 9.500e+01 1.669e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 08:06:54,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.62 vs. limit=15.0 2023-11-23 08:06:55,303 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8450, loss[loss=0.06975, simple_loss=0.08888, pruned_loss=0.01453, audio_tagging_loss=0.01078, over 15283.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09137, pruned_loss=0.01399, audio_tagging_loss=0.009015, over 3045291.30 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:06:57,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2300773.3333333335, ans=0.0 2023-11-23 08:07:05,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2300773.3333333335, ans=0.125 2023-11-23 08:07:13,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2300840.0, ans=0.125 2023-11-23 08:07:29,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.88 vs. limit=22.5 2023-11-23 08:07:36,238 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345150 2023-11-23 08:07:44,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2301040.0, ans=0.125 2023-11-23 08:07:58,692 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8500, loss[loss=0.06178, simple_loss=0.07699, pruned_loss=0.01277, audio_tagging_loss=0.01051, over 16769.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09203, pruned_loss=0.01418, audio_tagging_loss=0.009028, over 3054054.97 frames. ], batch size: 64, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:08:14,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2301173.3333333335, ans=0.0 2023-11-23 08:08:18,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2301173.3333333335, ans=0.1 2023-11-23 08:08:26,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2301240.0, ans=0.1 2023-11-23 08:08:34,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2301240.0, ans=0.125 2023-11-23 08:08:39,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345200 2023-11-23 08:08:42,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2301306.6666666665, ans=0.125 2023-11-23 08:08:50,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2301373.3333333335, ans=0.0 2023-11-23 08:08:55,833 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.259e+01 8.832e+01 9.401e+01 1.675e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 08:09:03,817 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8550, loss[loss=0.07594, simple_loss=0.1007, pruned_loss=0.0183, audio_tagging_loss=0.007314, over 15623.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.092, pruned_loss=0.01406, audio_tagging_loss=0.009109, over 3048289.22 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:09:09,048 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:09:12,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2301440.0, ans=0.125 2023-11-23 08:09:26,607 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.79 vs. limit=22.5 2023-11-23 08:09:36,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2301573.3333333335, ans=0.125 2023-11-23 08:09:44,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345250 2023-11-23 08:09:54,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2301706.6666666665, ans=0.125 2023-11-23 08:09:58,393 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0 2023-11-23 08:10:00,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2301706.6666666665, ans=0.2 2023-11-23 08:10:07,651 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8600, loss[loss=0.05321, simple_loss=0.07305, pruned_loss=0.007446, audio_tagging_loss=0.009241, over 15439.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.0915, pruned_loss=0.0139, audio_tagging_loss=0.009138, over 3043587.27 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:10:10,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2301773.3333333335, ans=0.125 2023-11-23 08:10:22,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2301840.0, ans=0.125 2023-11-23 08:10:31,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2301840.0, ans=22.5 2023-11-23 08:10:47,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.59 vs. limit=15.0 2023-11-23 08:10:49,391 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345300 2023-11-23 08:11:03,887 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.320e+01 8.873e+01 9.523e+01 1.247e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-23 08:11:08,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2302040.0, ans=0.125 2023-11-23 08:11:11,321 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8650, loss[loss=0.08111, simple_loss=0.1162, pruned_loss=0.01518, audio_tagging_loss=0.007824, over 15252.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09218, pruned_loss=0.01379, audio_tagging_loss=0.009271, over 3048708.05 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:11:42,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2302240.0, ans=0.0 2023-11-23 08:11:46,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.68 vs. limit=15.0 2023-11-23 08:11:47,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2302240.0, ans=0.04949747468305833 2023-11-23 08:11:50,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2302306.6666666665, ans=0.09899494936611666 2023-11-23 08:11:52,606 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345350 2023-11-23 08:12:16,460 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8700, loss[loss=0.06067, simple_loss=0.08468, pruned_loss=0.007571, audio_tagging_loss=0.01075, over 15525.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09274, pruned_loss=0.0139, audio_tagging_loss=0.00928, over 3043526.45 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:12:26,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.73 vs. limit=15.0 2023-11-23 08:12:39,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2302506.6666666665, ans=0.0 2023-11-23 08:12:40,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2302573.3333333335, ans=0.125 2023-11-23 08:12:40,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2302573.3333333335, ans=0.1 2023-11-23 08:12:51,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2302573.3333333335, ans=0.0 2023-11-23 08:12:57,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345400 2023-11-23 08:13:03,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2302640.0, ans=0.125 2023-11-23 08:13:08,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2302706.6666666665, ans=0.0 2023-11-23 08:13:13,357 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.372e+01 9.197e+01 9.763e+01 1.279e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-23 08:13:20,775 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8750, loss[loss=0.08099, simple_loss=0.1122, pruned_loss=0.01804, audio_tagging_loss=0.006862, over 14773.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09334, pruned_loss=0.01411, audio_tagging_loss=0.009289, over 3051105.09 frames. ], batch size: 53, lr: 2.34e-03, grad_scale: 32.0 2023-11-23 08:13:24,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2302773.3333333335, ans=0.1 2023-11-23 08:13:38,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2302840.0, ans=0.1 2023-11-23 08:13:48,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2302906.6666666665, ans=0.0 2023-11-23 08:14:01,532 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345450 2023-11-23 08:14:12,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2303040.0, ans=0.2 2023-11-23 08:14:24,247 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8800, loss[loss=0.09119, simple_loss=0.1206, pruned_loss=0.02157, audio_tagging_loss=0.009327, over 15422.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09428, pruned_loss=0.01438, audio_tagging_loss=0.009317, over 3053902.72 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:14:57,689 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:15:03,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2303306.6666666665, ans=0.1 2023-11-23 08:15:04,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345500 2023-11-23 08:15:20,171 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.339e+01 9.015e+01 9.606e+01 1.190e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-23 08:15:20,811 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.86 vs. limit=15.0 2023-11-23 08:15:28,090 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8850, loss[loss=0.06969, simple_loss=0.0933, pruned_loss=0.01353, audio_tagging_loss=0.009512, over 15527.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09331, pruned_loss=0.01414, audio_tagging_loss=0.009376, over 3050881.81 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:15:39,104 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:15:54,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2303573.3333333335, ans=0.0 2023-11-23 08:16:06,701 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.68 vs. limit=15.0 2023-11-23 08:16:08,817 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345550 2023-11-23 08:16:12,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2303640.0, ans=0.125 2023-11-23 08:16:31,263 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8900, loss[loss=0.07255, simple_loss=0.09147, pruned_loss=0.01883, audio_tagging_loss=0.007986, over 15202.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09362, pruned_loss=0.01424, audio_tagging_loss=0.009227, over 3048626.52 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:16:35,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2303773.3333333335, ans=0.1 2023-11-23 08:16:36,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2303773.3333333335, ans=0.125 2023-11-23 08:16:50,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2303840.0, ans=0.125 2023-11-23 08:16:53,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2303840.0, ans=0.1 2023-11-23 08:17:04,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2303906.6666666665, ans=0.0 2023-11-23 08:17:11,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345600 2023-11-23 08:17:27,492 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.364e+01 9.019e+01 9.831e+01 1.164e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 08:17:32,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2304040.0, ans=0.125 2023-11-23 08:17:34,814 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 8950, loss[loss=0.06855, simple_loss=0.08553, pruned_loss=0.01402, audio_tagging_loss=0.01176, over 15454.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09447, pruned_loss=0.0144, audio_tagging_loss=0.009065, over 3052840.46 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:17:50,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2304173.3333333335, ans=0.125 2023-11-23 08:17:51,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2304173.3333333335, ans=0.1 2023-11-23 08:17:57,621 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:18:00,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2304240.0, ans=0.2 2023-11-23 08:18:16,377 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345650 2023-11-23 08:18:40,009 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9000, loss[loss=0.06758, simple_loss=0.0871, pruned_loss=0.01448, audio_tagging_loss=0.009548, over 16885.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09505, pruned_loss=0.01467, audio_tagging_loss=0.008981, over 3054412.04 frames. ], batch size: 63, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:18:40,010 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 08:19:09,206 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.8540, 1.3064, 3.6264, 3.1616, 2.9548, 3.2070, 3.0132, 3.2436], device='cuda:3') 2023-11-23 08:19:21,767 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.0276, 2.8796, 3.0903, 2.7513], device='cuda:3') 2023-11-23 08:19:23,268 INFO [train_asr.py:1253] (3/4) Epoch 29, validation: loss=0.05895, simple_loss=0.05118, pruned_loss=0.005121, audio_tagging_loss=0.02824, over 4681554.00 frames. 2023-11-23 08:19:23,269 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 08:19:27,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2304440.0, ans=0.0 2023-11-23 08:19:30,529 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.35 vs. limit=15.0 2023-11-23 08:19:42,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2304506.6666666665, ans=0.125 2023-11-23 08:20:04,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345700 2023-11-23 08:20:19,505 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.885e+01 8.225e+01 9.124e+01 9.733e+01 1.162e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-23 08:20:19,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2304706.6666666665, ans=0.125 2023-11-23 08:20:27,058 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9050, loss[loss=0.05977, simple_loss=0.07933, pruned_loss=0.008943, audio_tagging_loss=0.01116, over 16358.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09525, pruned_loss=0.0146, audio_tagging_loss=0.008914, over 3061653.44 frames. ], batch size: 63, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:21:03,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.98 vs. limit=6.0 2023-11-23 08:21:08,205 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345750 2023-11-23 08:21:08,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2304973.3333333335, ans=0.2 2023-11-23 08:21:31,443 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9100, loss[loss=0.08679, simple_loss=0.1324, pruned_loss=0.01497, audio_tagging_loss=0.005604, over 16138.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09491, pruned_loss=0.0144, audio_tagging_loss=0.008915, over 3059482.12 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:21:31,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2305106.6666666665, ans=0.125 2023-11-23 08:21:52,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2305173.3333333335, ans=0.125 2023-11-23 08:21:52,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2305173.3333333335, ans=0.0 2023-11-23 08:21:59,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2305240.0, ans=0.0 2023-11-23 08:22:12,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345800 2023-11-23 08:22:23,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.66 vs. limit=10.0 2023-11-23 08:22:28,183 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.178e+01 8.924e+01 9.840e+01 1.213e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 08:22:34,169 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9150, loss[loss=0.05661, simple_loss=0.06709, pruned_loss=0.01236, audio_tagging_loss=0.01071, over 15286.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09454, pruned_loss=0.01439, audio_tagging_loss=0.008993, over 3056836.11 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:23:14,442 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:23:15,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345850 2023-11-23 08:23:15,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2305640.0, ans=0.0 2023-11-23 08:23:37,719 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9200, loss[loss=0.06518, simple_loss=0.08259, pruned_loss=0.0157, audio_tagging_loss=0.00818, over 14701.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09401, pruned_loss=0.0144, audio_tagging_loss=0.008928, over 3051694.87 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:24:00,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2305840.0, ans=0.0 2023-11-23 08:24:07,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2305906.6666666665, ans=0.1 2023-11-23 08:24:14,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2305973.3333333335, ans=0.015 2023-11-23 08:24:18,697 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345900 2023-11-23 08:24:18,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_na.min_abs, batch_count=2305973.3333333335, ans=0.02 2023-11-23 08:24:26,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2305973.3333333335, ans=0.0 2023-11-23 08:24:35,650 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.303e+01 8.816e+01 9.419e+01 1.160e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-23 08:24:39,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2306040.0, ans=0.0 2023-11-23 08:24:40,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2306040.0, ans=0.125 2023-11-23 08:24:40,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2306040.0, ans=0.125 2023-11-23 08:24:42,464 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9250, loss[loss=0.05251, simple_loss=0.06422, pruned_loss=0.01148, audio_tagging_loss=0.008918, over 14507.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.0937, pruned_loss=0.01435, audio_tagging_loss=0.008986, over 3055474.99 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:24:59,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2306173.3333333335, ans=0.125 2023-11-23 08:25:11,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2306240.0, ans=0.0 2023-11-23 08:25:22,701 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 345950 2023-11-23 08:25:34,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2306373.3333333335, ans=0.125 2023-11-23 08:25:41,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2306373.3333333335, ans=0.125 2023-11-23 08:25:44,889 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9300, loss[loss=0.0513, simple_loss=0.05862, pruned_loss=0.009507, audio_tagging_loss=0.01249, over 14460.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09363, pruned_loss=0.01442, audio_tagging_loss=0.008972, over 3057100.46 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:25:54,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2306440.0, ans=0.125 2023-11-23 08:25:58,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2306506.6666666665, ans=0.125 2023-11-23 08:26:12,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2306573.3333333335, ans=0.2 2023-11-23 08:26:25,829 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346000 2023-11-23 08:26:27,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2306640.0, ans=0.125 2023-11-23 08:26:32,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2306640.0, ans=0.2 2023-11-23 08:26:41,832 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.507e+01 8.976e+01 9.625e+01 1.274e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-23 08:26:48,013 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9350, loss[loss=0.0917, simple_loss=0.1299, pruned_loss=0.01988, audio_tagging_loss=0.006878, over 16317.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09318, pruned_loss=0.01449, audio_tagging_loss=0.008979, over 3060028.46 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:27:08,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2306840.0, ans=0.125 2023-11-23 08:27:18,686 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.73 vs. limit=22.5 2023-11-23 08:27:29,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346050 2023-11-23 08:27:36,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2306973.3333333335, ans=0.125 2023-11-23 08:27:53,085 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9400, loss[loss=0.05748, simple_loss=0.07289, pruned_loss=0.01179, audio_tagging_loss=0.009238, over 14437.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09248, pruned_loss=0.01428, audio_tagging_loss=0.009065, over 3064503.98 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:27:54,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2307106.6666666665, ans=0.1 2023-11-23 08:27:55,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2307106.6666666665, ans=0.125 2023-11-23 08:28:11,644 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:28:15,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2307173.3333333335, ans=0.0 2023-11-23 08:28:28,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2307306.6666666665, ans=0.125 2023-11-23 08:28:33,043 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346100 2023-11-23 08:28:41,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2307306.6666666665, ans=0.1 2023-11-23 08:28:44,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2307373.3333333335, ans=0.0 2023-11-23 08:28:45,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2307373.3333333335, ans=0.04949747468305833 2023-11-23 08:28:51,283 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.336e+01 8.834e+01 9.625e+01 1.227e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 08:28:53,824 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:28:56,224 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9450, loss[loss=0.06781, simple_loss=0.09867, pruned_loss=0.01083, audio_tagging_loss=0.007642, over 16289.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.0927, pruned_loss=0.01409, audio_tagging_loss=0.009152, over 3063781.83 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:29:14,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2307506.6666666665, ans=0.125 2023-11-23 08:29:36,717 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346150 2023-11-23 08:29:43,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2307640.0, ans=0.125 2023-11-23 08:29:55,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2307706.6666666665, ans=0.125 2023-11-23 08:29:58,774 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9500, loss[loss=0.06668, simple_loss=0.08403, pruned_loss=0.0133, audio_tagging_loss=0.01137, over 15251.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09316, pruned_loss=0.01419, audio_tagging_loss=0.009165, over 3062792.68 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:30:03,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2307773.3333333335, ans=0.0 2023-11-23 08:30:04,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.29 vs. limit=6.0 2023-11-23 08:30:21,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2307840.0, ans=0.0 2023-11-23 08:30:23,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2307906.6666666665, ans=0.1 2023-11-23 08:30:39,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346200 2023-11-23 08:30:56,977 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.410e+01 8.909e+01 9.826e+01 1.512e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 08:31:03,147 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9550, loss[loss=0.06266, simple_loss=0.08731, pruned_loss=0.01027, audio_tagging_loss=0.008743, over 14799.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09368, pruned_loss=0.01424, audio_tagging_loss=0.009175, over 3056511.80 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:31:07,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.36 vs. limit=22.5 2023-11-23 08:31:43,413 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346250 2023-11-23 08:32:05,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2308373.3333333335, ans=0.125 2023-11-23 08:32:07,632 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9600, loss[loss=0.05325, simple_loss=0.07424, pruned_loss=0.007359, audio_tagging_loss=0.008772, over 15113.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09361, pruned_loss=0.01423, audio_tagging_loss=0.009264, over 3051979.93 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:32:12,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2308440.0, ans=0.0 2023-11-23 08:32:20,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2308506.6666666665, ans=0.1 2023-11-23 08:32:24,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2308506.6666666665, ans=0.125 2023-11-23 08:32:40,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.13 vs. limit=15.0 2023-11-23 08:32:43,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2308573.3333333335, ans=0.1 2023-11-23 08:32:49,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346300 2023-11-23 08:32:52,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2308640.0, ans=0.125 2023-11-23 08:32:54,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2308640.0, ans=0.125 2023-11-23 08:33:06,938 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.195e+01 8.302e+01 9.179e+01 9.827e+01 1.321e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-23 08:33:11,996 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9650, loss[loss=0.0785, simple_loss=0.1006, pruned_loss=0.01861, audio_tagging_loss=0.009594, over 15555.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09341, pruned_loss=0.01415, audio_tagging_loss=0.009209, over 3053411.72 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:33:17,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2308773.3333333335, ans=0.2 2023-11-23 08:33:20,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2308773.3333333335, ans=0.0 2023-11-23 08:33:40,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2308906.6666666665, ans=0.1 2023-11-23 08:33:53,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346350 2023-11-23 08:34:02,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2309040.0, ans=0.0 2023-11-23 08:34:06,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2309040.0, ans=0.1 2023-11-23 08:34:13,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2309040.0, ans=0.1 2023-11-23 08:34:15,625 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9700, loss[loss=0.08826, simple_loss=0.1231, pruned_loss=0.01858, audio_tagging_loss=0.008119, over 16259.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09259, pruned_loss=0.01407, audio_tagging_loss=0.009211, over 3057546.31 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:34:38,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2309173.3333333335, ans=0.125 2023-11-23 08:34:41,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2309173.3333333335, ans=0.2 2023-11-23 08:34:42,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2309240.0, ans=0.0 2023-11-23 08:34:52,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2309240.0, ans=0.0 2023-11-23 08:34:58,299 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346400 2023-11-23 08:35:06,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2309306.6666666665, ans=0.09899494936611666 2023-11-23 08:35:09,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2309373.3333333335, ans=0.125 2023-11-23 08:35:12,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2309373.3333333335, ans=0.0 2023-11-23 08:35:19,617 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.905e+01 8.371e+01 8.992e+01 9.698e+01 1.313e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-23 08:35:23,349 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9750, loss[loss=0.05591, simple_loss=0.07929, pruned_loss=0.009932, audio_tagging_loss=0.006331, over 15980.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09257, pruned_loss=0.0141, audio_tagging_loss=0.009106, over 3056956.08 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:35:27,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2309440.0, ans=0.0 2023-11-23 08:35:34,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2309506.6666666665, ans=0.125 2023-11-23 08:35:47,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2309573.3333333335, ans=0.1 2023-11-23 08:35:55,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2309573.3333333335, ans=0.04949747468305833 2023-11-23 08:35:59,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.85 vs. limit=15.0 2023-11-23 08:36:04,832 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346450 2023-11-23 08:36:27,509 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9800, loss[loss=0.06, simple_loss=0.07842, pruned_loss=0.01135, audio_tagging_loss=0.009441, over 14998.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09255, pruned_loss=0.0141, audio_tagging_loss=0.008965, over 3055716.71 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:36:35,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2309773.3333333335, ans=0.125 2023-11-23 08:36:36,762 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.95 vs. limit=22.5 2023-11-23 08:37:01,605 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.88 vs. limit=6.0 2023-11-23 08:37:02,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2309906.6666666665, ans=0.05 2023-11-23 08:37:03,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309906.6666666665, ans=0.1 2023-11-23 08:37:09,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346500 2023-11-23 08:37:12,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2309973.3333333335, ans=10.0 2023-11-23 08:37:12,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2309973.3333333335, ans=0.125 2023-11-23 08:37:21,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2310040.0, ans=0.0 2023-11-23 08:37:23,988 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:37:24,695 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.53 vs. limit=15.0 2023-11-23 08:37:27,540 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.413e+01 9.129e+01 9.676e+01 1.290e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-23 08:37:29,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.38 vs. limit=22.5 2023-11-23 08:37:31,277 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9850, loss[loss=0.04895, simple_loss=0.064, pruned_loss=0.005485, audio_tagging_loss=0.01147, over 15935.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.0933, pruned_loss=0.01427, audio_tagging_loss=0.008953, over 3057310.66 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:38:12,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346550 2023-11-23 08:38:20,902 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0 2023-11-23 08:38:36,345 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9900, loss[loss=0.08822, simple_loss=0.1186, pruned_loss=0.01974, audio_tagging_loss=0.009181, over 15124.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09319, pruned_loss=0.01426, audio_tagging_loss=0.009005, over 3055298.07 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:38:39,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2310440.0, ans=0.125 2023-11-23 08:39:00,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2310573.3333333335, ans=0.1 2023-11-23 08:39:17,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346600 2023-11-23 08:39:29,531 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0 2023-11-23 08:39:36,801 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.313e+01 8.245e+01 9.085e+01 9.612e+01 1.420e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-23 08:39:40,501 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 9950, loss[loss=0.08274, simple_loss=0.1169, pruned_loss=0.01704, audio_tagging_loss=0.007256, over 15276.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09208, pruned_loss=0.01416, audio_tagging_loss=0.008986, over 3055190.02 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:39:47,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2310773.3333333335, ans=0.0 2023-11-23 08:39:48,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2310773.3333333335, ans=0.2 2023-11-23 08:40:22,045 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346650 2023-11-23 08:40:27,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2310973.3333333335, ans=0.0 2023-11-23 08:40:43,977 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10000, loss[loss=0.09452, simple_loss=0.1235, pruned_loss=0.02409, audio_tagging_loss=0.008679, over 15124.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09123, pruned_loss=0.01401, audio_tagging_loss=0.009033, over 3052734.57 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:41:14,449 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.62 vs. limit=15.0 2023-11-23 08:41:19,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2311240.0, ans=0.125 2023-11-23 08:41:23,908 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.95 vs. limit=15.0 2023-11-23 08:41:24,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346700 2023-11-23 08:41:35,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2311373.3333333335, ans=0.125 2023-11-23 08:41:44,041 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.283e+01 8.893e+01 9.595e+01 1.369e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 08:41:47,714 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10050, loss[loss=0.07544, simple_loss=0.0982, pruned_loss=0.01634, audio_tagging_loss=0.009989, over 14981.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09187, pruned_loss=0.01413, audio_tagging_loss=0.008923, over 3044671.62 frames. ], batch size: 54, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:41:54,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2311440.0, ans=0.125 2023-11-23 08:41:54,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2311440.0, ans=0.1 2023-11-23 08:42:09,471 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:42:16,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2311573.3333333335, ans=0.1 2023-11-23 08:42:28,468 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346750 2023-11-23 08:42:42,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2311706.6666666665, ans=0.125 2023-11-23 08:42:51,642 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10100, loss[loss=0.04862, simple_loss=0.05633, pruned_loss=0.007037, audio_tagging_loss=0.01342, over 13226.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09207, pruned_loss=0.01407, audio_tagging_loss=0.008998, over 3043464.95 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:43:09,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2311840.0, ans=0.125 2023-11-23 08:43:17,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2311906.6666666665, ans=22.5 2023-11-23 08:43:25,470 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.22 vs. limit=12.0 2023-11-23 08:43:28,155 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.02 vs. limit=22.5 2023-11-23 08:43:32,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346800 2023-11-23 08:43:43,430 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:43:53,030 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.400e+01 8.955e+01 9.679e+01 1.203e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 08:43:55,576 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10150, loss[loss=0.09975, simple_loss=0.1348, pruned_loss=0.02527, audio_tagging_loss=0.007075, over 16040.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09234, pruned_loss=0.01406, audio_tagging_loss=0.009135, over 3046347.52 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:43:57,385 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.76 vs. limit=12.0 2023-11-23 08:44:06,186 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.20 vs. limit=22.5 2023-11-23 08:44:08,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.97 vs. limit=15.0 2023-11-23 08:44:20,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2312240.0, ans=0.125 2023-11-23 08:44:25,155 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:44:36,942 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346850 2023-11-23 08:44:39,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2312306.6666666665, ans=0.125 2023-11-23 08:44:54,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.66 vs. limit=15.0 2023-11-23 08:44:55,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2312373.3333333335, ans=0.09899494936611666 2023-11-23 08:44:56,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2312373.3333333335, ans=0.125 2023-11-23 08:45:00,221 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10200, loss[loss=0.09201, simple_loss=0.1273, pruned_loss=0.01844, audio_tagging_loss=0.009908, over 15488.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09286, pruned_loss=0.0143, audio_tagging_loss=0.009165, over 3051572.96 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:45:04,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2312440.0, ans=0.0 2023-11-23 08:45:14,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.57 vs. limit=15.0 2023-11-23 08:45:23,109 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:45:28,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2312573.3333333335, ans=0.0 2023-11-23 08:45:28,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2312573.3333333335, ans=0.1 2023-11-23 08:45:40,930 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346900 2023-11-23 08:45:47,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2312640.0, ans=0.125 2023-11-23 08:45:48,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.62 vs. limit=15.0 2023-11-23 08:46:01,784 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.340e+01 8.896e+01 9.645e+01 1.207e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 08:46:03,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2312773.3333333335, ans=0.1 2023-11-23 08:46:04,278 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10250, loss[loss=0.06243, simple_loss=0.08287, pruned_loss=0.01175, audio_tagging_loss=0.009249, over 14351.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09325, pruned_loss=0.01419, audio_tagging_loss=0.009207, over 3051448.09 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:46:07,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.55 vs. limit=15.0 2023-11-23 08:46:19,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2312840.0, ans=0.125 2023-11-23 08:46:22,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2312840.0, ans=0.0 2023-11-23 08:46:24,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2312840.0, ans=0.125 2023-11-23 08:46:26,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2312840.0, ans=0.0 2023-11-23 08:46:28,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2312906.6666666665, ans=0.125 2023-11-23 08:46:37,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2312906.6666666665, ans=0.125 2023-11-23 08:46:45,692 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 346950 2023-11-23 08:46:51,066 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.11 vs. limit=15.0 2023-11-23 08:47:08,021 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10300, loss[loss=0.05026, simple_loss=0.05896, pruned_loss=0.007693, audio_tagging_loss=0.01309, over 14093.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09288, pruned_loss=0.01414, audio_tagging_loss=0.009251, over 3047970.06 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:47:14,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.16 vs. limit=15.0 2023-11-23 08:47:19,134 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.39 vs. limit=22.5 2023-11-23 08:47:29,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2313173.3333333335, ans=0.0 2023-11-23 08:47:44,146 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0 2023-11-23 08:47:48,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347000 2023-11-23 08:47:57,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2313306.6666666665, ans=0.0 2023-11-23 08:48:00,045 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.18 vs. limit=15.0 2023-11-23 08:48:03,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2313373.3333333335, ans=0.125 2023-11-23 08:48:07,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2313373.3333333335, ans=0.0 2023-11-23 08:48:10,246 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.826e+01 8.491e+01 9.106e+01 9.880e+01 1.580e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-23 08:48:12,729 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10350, loss[loss=0.07648, simple_loss=0.09441, pruned_loss=0.0188, audio_tagging_loss=0.01047, over 15250.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09329, pruned_loss=0.01428, audio_tagging_loss=0.009318, over 3048786.66 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:48:38,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2313573.3333333335, ans=0.0 2023-11-23 08:48:48,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2313573.3333333335, ans=0.0 2023-11-23 08:48:52,314 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.08 vs. limit=15.0 2023-11-23 08:48:52,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347050 2023-11-23 08:49:16,476 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10400, loss[loss=0.04314, simple_loss=0.05702, pruned_loss=0.006539, audio_tagging_loss=0.008088, over 15470.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09347, pruned_loss=0.01427, audio_tagging_loss=0.009332, over 3046527.39 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:49:19,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2313773.3333333335, ans=0.2 2023-11-23 08:49:39,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2313840.0, ans=0.0 2023-11-23 08:49:51,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2313906.6666666665, ans=0.0 2023-11-23 08:49:58,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347100 2023-11-23 08:50:18,925 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.096e+01 8.361e+01 8.771e+01 9.632e+01 1.204e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-23 08:50:20,795 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10450, loss[loss=0.08052, simple_loss=0.1095, pruned_loss=0.015, audio_tagging_loss=0.0108, over 16005.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09362, pruned_loss=0.01428, audio_tagging_loss=0.009276, over 3046895.77 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:50:24,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2314106.6666666665, ans=0.1 2023-11-23 08:50:30,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2314106.6666666665, ans=0.0 2023-11-23 08:50:40,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2314173.3333333335, ans=0.125 2023-11-23 08:50:46,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2314240.0, ans=0.125 2023-11-23 08:51:02,139 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347150 2023-11-23 08:51:04,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2314306.6666666665, ans=0.1 2023-11-23 08:51:08,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.65 vs. limit=15.0 2023-11-23 08:51:23,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2314373.3333333335, ans=0.1 2023-11-23 08:51:26,485 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10500, loss[loss=0.05794, simple_loss=0.07142, pruned_loss=0.01121, audio_tagging_loss=0.01102, over 16400.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09296, pruned_loss=0.01418, audio_tagging_loss=0.009152, over 3042772.43 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:51:37,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2314506.6666666665, ans=0.0 2023-11-23 08:51:41,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2314506.6666666665, ans=0.2 2023-11-23 08:52:06,415 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347200 2023-11-23 08:52:16,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2314640.0, ans=0.025 2023-11-23 08:52:29,019 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.754e+01 8.399e+01 8.762e+01 9.493e+01 1.186e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-23 08:52:30,288 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10550, loss[loss=0.0716, simple_loss=0.09865, pruned_loss=0.01309, audio_tagging_loss=0.009186, over 16620.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09282, pruned_loss=0.01415, audio_tagging_loss=0.009162, over 3044359.35 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:52:36,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2314773.3333333335, ans=0.125 2023-11-23 08:53:12,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347250 2023-11-23 08:53:15,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2314973.3333333335, ans=0.95 2023-11-23 08:53:17,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2314973.3333333335, ans=0.125 2023-11-23 08:53:21,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2315040.0, ans=0.0 2023-11-23 08:53:29,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.40 vs. limit=15.0 2023-11-23 08:53:30,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2315040.0, ans=0.0 2023-11-23 08:53:31,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2315040.0, ans=0.1 2023-11-23 08:53:33,971 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10600, loss[loss=0.06687, simple_loss=0.08476, pruned_loss=0.01388, audio_tagging_loss=0.01061, over 15348.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09289, pruned_loss=0.01409, audio_tagging_loss=0.009102, over 3037911.64 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:53:36,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2315106.6666666665, ans=0.125 2023-11-23 08:53:40,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2315106.6666666665, ans=0.125 2023-11-23 08:54:05,178 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 08:54:12,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2315306.6666666665, ans=0.0 2023-11-23 08:54:15,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2315306.6666666665, ans=0.1 2023-11-23 08:54:15,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347300 2023-11-23 08:54:19,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2315306.6666666665, ans=0.0 2023-11-23 08:54:20,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2315306.6666666665, ans=10.0 2023-11-23 08:54:25,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2315373.3333333335, ans=0.1 2023-11-23 08:54:37,206 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.247e+01 8.943e+01 9.697e+01 1.268e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-23 08:54:39,151 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10650, loss[loss=0.05808, simple_loss=0.07888, pruned_loss=0.01086, audio_tagging_loss=0.007784, over 15267.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09212, pruned_loss=0.01394, audio_tagging_loss=0.009025, over 3036249.08 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:54:54,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2315506.6666666665, ans=0.0 2023-11-23 08:54:56,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2315506.6666666665, ans=0.025 2023-11-23 08:55:19,790 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347350 2023-11-23 08:55:22,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2315640.0, ans=0.0 2023-11-23 08:55:41,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2315706.6666666665, ans=0.2 2023-11-23 08:55:44,091 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10700, loss[loss=0.05192, simple_loss=0.06578, pruned_loss=0.007927, audio_tagging_loss=0.0111, over 16364.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.0918, pruned_loss=0.01396, audio_tagging_loss=0.009085, over 3033569.91 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:56:03,449 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.70 vs. limit=15.0 2023-11-23 08:56:06,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2315840.0, ans=0.0 2023-11-23 08:56:12,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0 2023-11-23 08:56:19,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2315906.6666666665, ans=0.125 2023-11-23 08:56:25,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347400 2023-11-23 08:56:27,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.86 vs. limit=15.0 2023-11-23 08:56:41,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2316040.0, ans=0.1 2023-11-23 08:56:46,533 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.151e+01 8.772e+01 9.535e+01 1.207e+02, threshold=1.754e+02, percent-clipped=0.0 2023-11-23 08:56:47,771 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10750, loss[loss=0.06141, simple_loss=0.08282, pruned_loss=0.01002, audio_tagging_loss=0.009986, over 15535.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.0912, pruned_loss=0.01383, audio_tagging_loss=0.009057, over 3033167.48 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 08:57:08,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2316173.3333333335, ans=0.125 2023-11-23 08:57:11,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2316173.3333333335, ans=0.125 2023-11-23 08:57:13,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2316240.0, ans=0.07 2023-11-23 08:57:15,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2316240.0, ans=0.125 2023-11-23 08:57:17,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2316240.0, ans=0.0 2023-11-23 08:57:22,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2316240.0, ans=0.125 2023-11-23 08:57:28,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347450 2023-11-23 08:57:36,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2316306.6666666665, ans=0.015 2023-11-23 08:57:42,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2316373.3333333335, ans=0.0 2023-11-23 08:57:50,639 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10800, loss[loss=0.08194, simple_loss=0.1103, pruned_loss=0.01705, audio_tagging_loss=0.009745, over 14404.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09186, pruned_loss=0.01396, audio_tagging_loss=0.009001, over 3040040.30 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:58:04,359 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.12 vs. limit=10.0 2023-11-23 08:58:05,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2316506.6666666665, ans=0.125 2023-11-23 08:58:06,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2316506.6666666665, ans=0.125 2023-11-23 08:58:11,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2316506.6666666665, ans=0.125 2023-11-23 08:58:31,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347500 2023-11-23 08:58:32,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2316640.0, ans=0.0 2023-11-23 08:58:55,091 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.196e+01 8.848e+01 9.514e+01 1.233e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-23 08:58:56,335 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10850, loss[loss=0.05978, simple_loss=0.08013, pruned_loss=0.009525, audio_tagging_loss=0.01019, over 15554.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09149, pruned_loss=0.01388, audio_tagging_loss=0.009104, over 3033049.92 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 08:59:23,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2316906.6666666665, ans=0.1 2023-11-23 08:59:24,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2316906.6666666665, ans=0.2 2023-11-23 08:59:37,098 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0 2023-11-23 08:59:38,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347550 2023-11-23 08:59:41,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2316973.3333333335, ans=0.125 2023-11-23 08:59:57,034 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 08:59:58,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2317040.0, ans=0.0 2023-11-23 08:59:58,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.75 vs. limit=15.0 2023-11-23 08:59:59,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2317106.6666666665, ans=0.0 2023-11-23 09:00:00,710 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10900, loss[loss=0.07336, simple_loss=0.08562, pruned_loss=0.0218, audio_tagging_loss=0.008744, over 14073.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09152, pruned_loss=0.01385, audio_tagging_loss=0.00916, over 3030248.83 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 09:00:14,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2317173.3333333335, ans=0.2 2023-11-23 09:00:21,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2317173.3333333335, ans=0.0 2023-11-23 09:00:43,178 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347600 2023-11-23 09:01:04,204 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.420e+01 8.915e+01 9.673e+01 1.270e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 09:01:05,504 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 10950, loss[loss=0.0633, simple_loss=0.08883, pruned_loss=0.009921, audio_tagging_loss=0.008963, over 13971.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09125, pruned_loss=0.01382, audio_tagging_loss=0.009287, over 3033823.67 frames. ], batch size: 52, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 09:01:24,672 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.98 vs. limit=15.0 2023-11-23 09:01:27,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2317506.6666666665, ans=0.1 2023-11-23 09:01:43,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2317640.0, ans=0.125 2023-11-23 09:01:47,396 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347650 2023-11-23 09:01:57,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2317706.6666666665, ans=0.2 2023-11-23 09:02:11,978 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11000, loss[loss=0.07223, simple_loss=0.08779, pruned_loss=0.01665, audio_tagging_loss=0.01169, over 15304.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09144, pruned_loss=0.01388, audio_tagging_loss=0.009219, over 3034460.78 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:02:21,751 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 09:02:49,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2317973.3333333335, ans=0.125 2023-11-23 09:02:53,192 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347700 2023-11-23 09:02:54,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2317973.3333333335, ans=0.0 2023-11-23 09:03:02,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2318040.0, ans=0.125 2023-11-23 09:03:02,974 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.71 vs. limit=10.0 2023-11-23 09:03:16,062 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.585e+01 9.370e+01 1.001e+02 1.222e+02, threshold=1.874e+02, percent-clipped=0.0 2023-11-23 09:03:16,127 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11050, loss[loss=0.06969, simple_loss=0.09093, pruned_loss=0.01359, audio_tagging_loss=0.01063, over 15246.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09189, pruned_loss=0.01403, audio_tagging_loss=0.009334, over 3040448.64 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:03:25,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2318106.6666666665, ans=0.125 2023-11-23 09:03:37,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2318173.3333333335, ans=0.125 2023-11-23 09:03:57,651 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347750 2023-11-23 09:04:12,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2318373.3333333335, ans=0.05 2023-11-23 09:04:19,457 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11100, loss[loss=0.08288, simple_loss=0.1102, pruned_loss=0.01867, audio_tagging_loss=0.009102, over 15035.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.092, pruned_loss=0.01401, audio_tagging_loss=0.009425, over 3037759.35 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:04:19,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2318440.0, ans=0.0 2023-11-23 09:04:30,678 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.20 vs. limit=12.0 2023-11-23 09:04:32,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0 2023-11-23 09:05:01,274 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347800 2023-11-23 09:05:01,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2318640.0, ans=0.0 2023-11-23 09:05:01,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.51 vs. limit=15.0 2023-11-23 09:05:09,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2318640.0, ans=0.1 2023-11-23 09:05:15,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2318706.6666666665, ans=0.0 2023-11-23 09:05:24,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2318773.3333333335, ans=0.125 2023-11-23 09:05:25,367 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.253e+01 8.535e+01 9.196e+01 1.008e+02 1.260e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-23 09:05:25,412 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11150, loss[loss=0.08603, simple_loss=0.1156, pruned_loss=0.01967, audio_tagging_loss=0.008537, over 15223.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.0924, pruned_loss=0.01404, audio_tagging_loss=0.009423, over 3038186.95 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:05:40,975 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:05:48,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2318840.0, ans=0.0 2023-11-23 09:05:49,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2318906.6666666665, ans=0.0 2023-11-23 09:05:59,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2318906.6666666665, ans=0.125 2023-11-23 09:06:05,629 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347850 2023-11-23 09:06:11,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2318973.3333333335, ans=0.035 2023-11-23 09:06:11,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2318973.3333333335, ans=0.09899494936611666 2023-11-23 09:06:20,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2319040.0, ans=0.125 2023-11-23 09:06:29,290 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11200, loss[loss=0.07662, simple_loss=0.1101, pruned_loss=0.01391, audio_tagging_loss=0.007642, over 16339.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09235, pruned_loss=0.0141, audio_tagging_loss=0.009463, over 3044752.78 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 09:06:34,836 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.66 vs. limit=15.0 2023-11-23 09:06:41,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2319173.3333333335, ans=0.125 2023-11-23 09:06:47,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2319173.3333333335, ans=0.125 2023-11-23 09:06:47,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2319173.3333333335, ans=0.1 2023-11-23 09:07:09,979 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347900 2023-11-23 09:07:24,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2319373.3333333335, ans=0.0 2023-11-23 09:07:31,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2319440.0, ans=0.125 2023-11-23 09:07:32,466 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.786e+01 8.283e+01 8.930e+01 9.840e+01 1.502e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-23 09:07:32,520 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11250, loss[loss=0.07874, simple_loss=0.1132, pruned_loss=0.0154, audio_tagging_loss=0.006735, over 14772.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09284, pruned_loss=0.01412, audio_tagging_loss=0.009482, over 3043965.93 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 32.0 2023-11-23 09:07:32,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2319440.0, ans=0.125 2023-11-23 09:07:55,845 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:08:07,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2319573.3333333335, ans=0.1 2023-11-23 09:08:13,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 347950 2023-11-23 09:08:13,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2319640.0, ans=0.0 2023-11-23 09:08:24,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0 2023-11-23 09:08:29,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2319706.6666666665, ans=0.0 2023-11-23 09:08:35,815 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.36 vs. limit=22.5 2023-11-23 09:08:36,532 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11300, loss[loss=0.06109, simple_loss=0.08593, pruned_loss=0.009226, audio_tagging_loss=0.008903, over 14864.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09224, pruned_loss=0.01394, audio_tagging_loss=0.009314, over 3031363.88 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:08:38,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2319773.3333333335, ans=0.04949747468305833 2023-11-23 09:08:38,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2319773.3333333335, ans=0.125 2023-11-23 09:08:46,982 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.94 vs. limit=10.0 2023-11-23 09:08:54,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.73 vs. limit=15.0 2023-11-23 09:09:00,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2319906.6666666665, ans=0.5 2023-11-23 09:09:10,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2319906.6666666665, ans=0.1 2023-11-23 09:09:16,308 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348000 2023-11-23 09:09:23,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2319973.3333333335, ans=0.0 2023-11-23 09:09:32,931 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.98 vs. limit=15.0 2023-11-23 09:09:42,691 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11350, loss[loss=0.0837, simple_loss=0.1084, pruned_loss=0.01909, audio_tagging_loss=0.01039, over 14427.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09248, pruned_loss=0.01402, audio_tagging_loss=0.009174, over 3034390.51 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:09:43,917 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.379e+01 8.172e+01 9.052e+01 9.730e+01 1.154e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 09:09:47,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2320106.6666666665, ans=0.125 2023-11-23 09:10:01,553 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.90 vs. limit=6.0 2023-11-23 09:10:23,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348050 2023-11-23 09:10:23,680 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.34 vs. limit=15.0 2023-11-23 09:10:37,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2320373.3333333335, ans=0.125 2023-11-23 09:10:45,533 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11400, loss[loss=0.05865, simple_loss=0.07422, pruned_loss=0.01158, audio_tagging_loss=0.009966, over 16194.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09283, pruned_loss=0.01391, audio_tagging_loss=0.009111, over 3042458.63 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:11:07,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2320506.6666666665, ans=0.125 2023-11-23 09:11:11,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2320573.3333333335, ans=0.125 2023-11-23 09:11:25,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2320640.0, ans=0.0 2023-11-23 09:11:26,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348100 2023-11-23 09:11:28,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2320640.0, ans=0.0 2023-11-23 09:11:31,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2320640.0, ans=0.1 2023-11-23 09:11:42,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2320706.6666666665, ans=0.0 2023-11-23 09:11:49,204 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11450, loss[loss=0.07447, simple_loss=0.1017, pruned_loss=0.01567, audio_tagging_loss=0.007969, over 14408.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09342, pruned_loss=0.01414, audio_tagging_loss=0.009043, over 3038542.04 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 8.0 2023-11-23 09:11:52,265 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.268e+01 8.173e+01 8.697e+01 9.540e+01 1.271e+02, threshold=1.739e+02, percent-clipped=0.0 2023-11-23 09:11:57,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2320773.3333333335, ans=0.125 2023-11-23 09:12:11,049 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.28 vs. limit=22.5 2023-11-23 09:12:20,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2320906.6666666665, ans=0.125 2023-11-23 09:12:29,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348150 2023-11-23 09:12:40,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2321040.0, ans=0.0 2023-11-23 09:12:52,910 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11500, loss[loss=0.08358, simple_loss=0.1103, pruned_loss=0.02045, audio_tagging_loss=0.007993, over 15525.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09305, pruned_loss=0.0142, audio_tagging_loss=0.009069, over 3038920.72 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 8.0 2023-11-23 09:12:58,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2321106.6666666665, ans=0.1 2023-11-23 09:12:59,788 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.01 vs. limit=10.0 2023-11-23 09:13:13,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2321173.3333333335, ans=0.0 2023-11-23 09:13:14,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2321173.3333333335, ans=0.125 2023-11-23 09:13:29,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2321240.0, ans=0.0 2023-11-23 09:13:32,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2321306.6666666665, ans=0.1 2023-11-23 09:13:33,957 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348200 2023-11-23 09:13:34,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2321306.6666666665, ans=0.1 2023-11-23 09:13:46,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2321373.3333333335, ans=0.1 2023-11-23 09:13:47,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2321373.3333333335, ans=0.125 2023-11-23 09:13:47,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2321373.3333333335, ans=0.0 2023-11-23 09:13:56,853 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11550, loss[loss=0.06011, simple_loss=0.07671, pruned_loss=0.01261, audio_tagging_loss=0.009142, over 14641.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09353, pruned_loss=0.01445, audio_tagging_loss=0.009004, over 3042353.76 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 8.0 2023-11-23 09:13:59,201 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.220e+01 8.786e+01 9.512e+01 1.197e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-23 09:14:14,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2321506.6666666665, ans=0.0 2023-11-23 09:14:16,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2321506.6666666665, ans=0.125 2023-11-23 09:14:24,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2321573.3333333335, ans=0.0 2023-11-23 09:14:35,099 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 09:14:37,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348250 2023-11-23 09:14:41,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2321640.0, ans=0.0 2023-11-23 09:14:56,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2321706.6666666665, ans=0.125 2023-11-23 09:15:00,559 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11600, loss[loss=0.05804, simple_loss=0.0775, pruned_loss=0.01006, audio_tagging_loss=0.009232, over 15694.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09351, pruned_loss=0.01442, audio_tagging_loss=0.008998, over 3046341.32 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:15:03,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2321773.3333333335, ans=0.1 2023-11-23 09:15:05,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2321773.3333333335, ans=0.05 2023-11-23 09:15:10,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2321773.3333333335, ans=0.125 2023-11-23 09:15:13,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2321840.0, ans=0.125 2023-11-23 09:15:20,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2321840.0, ans=0.0 2023-11-23 09:15:22,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2321840.0, ans=0.1 2023-11-23 09:15:30,253 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.12 vs. limit=22.5 2023-11-23 09:15:41,406 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348300 2023-11-23 09:15:46,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2321973.3333333335, ans=0.1 2023-11-23 09:15:49,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.67 vs. limit=15.0 2023-11-23 09:16:04,501 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11650, loss[loss=0.07683, simple_loss=0.09418, pruned_loss=0.0174, audio_tagging_loss=0.01234, over 15283.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09256, pruned_loss=0.01405, audio_tagging_loss=0.009085, over 3044396.09 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:16:05,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2322106.6666666665, ans=0.125 2023-11-23 09:16:06,809 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.255e+01 8.853e+01 9.730e+01 1.242e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-23 09:16:34,686 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:16:38,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2322240.0, ans=0.0 2023-11-23 09:16:46,150 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348350 2023-11-23 09:16:46,285 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:16:47,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2322306.6666666665, ans=0.125 2023-11-23 09:16:58,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2322373.3333333335, ans=0.125 2023-11-23 09:17:08,256 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11700, loss[loss=0.06713, simple_loss=0.08966, pruned_loss=0.01351, audio_tagging_loss=0.008784, over 14379.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09256, pruned_loss=0.01412, audio_tagging_loss=0.009146, over 3047542.29 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0 2023-11-23 09:17:38,235 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.85 vs. limit=15.0 2023-11-23 09:17:50,044 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348400 2023-11-23 09:17:56,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2322640.0, ans=0.0 2023-11-23 09:18:07,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2322706.6666666665, ans=0.1 2023-11-23 09:18:08,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2322706.6666666665, ans=0.0 2023-11-23 09:18:13,158 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11750, loss[loss=0.05058, simple_loss=0.05638, pruned_loss=0.008592, audio_tagging_loss=0.01379, over 15639.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09183, pruned_loss=0.01399, audio_tagging_loss=0.009178, over 3046204.96 frames. ], batch size: 60, lr: 2.32e-03, grad_scale: 16.0 2023-11-23 09:18:16,775 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.187e+01 8.216e+01 8.845e+01 9.440e+01 1.082e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 09:18:28,062 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.29 vs. limit=15.0 2023-11-23 09:18:40,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.85 vs. limit=15.0 2023-11-23 09:18:55,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348450 2023-11-23 09:19:14,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2323040.0, ans=0.0 2023-11-23 09:19:18,721 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11800, loss[loss=0.06929, simple_loss=0.09602, pruned_loss=0.01203, audio_tagging_loss=0.009252, over 15573.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09207, pruned_loss=0.01406, audio_tagging_loss=0.009207, over 3039386.43 frames. ], batch size: 57, lr: 2.32e-03, grad_scale: 16.0 2023-11-23 09:19:26,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2323106.6666666665, ans=0.125 2023-11-23 09:19:32,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2323173.3333333335, ans=0.125 2023-11-23 09:19:32,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2323173.3333333335, ans=0.2 2023-11-23 09:19:34,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2323173.3333333335, ans=0.0 2023-11-23 09:19:59,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348500 2023-11-23 09:20:22,071 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11850, loss[loss=0.07629, simple_loss=0.1046, pruned_loss=0.01376, audio_tagging_loss=0.01023, over 15095.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09229, pruned_loss=0.01397, audio_tagging_loss=0.009231, over 3032311.89 frames. ], batch size: 55, lr: 2.32e-03, grad_scale: 16.0 2023-11-23 09:20:24,372 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.461e+01 9.109e+01 9.788e+01 1.263e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-23 09:20:26,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2323440.0, ans=0.0 2023-11-23 09:20:26,388 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.29 vs. limit=15.0 2023-11-23 09:20:29,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2323440.0, ans=0.0 2023-11-23 09:20:33,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2323506.6666666665, ans=0.1 2023-11-23 09:20:46,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2323506.6666666665, ans=0.1 2023-11-23 09:20:49,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2323573.3333333335, ans=0.125 2023-11-23 09:20:51,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2323573.3333333335, ans=0.0 2023-11-23 09:21:03,621 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348550 2023-11-23 09:21:13,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2323706.6666666665, ans=0.125 2023-11-23 09:21:25,835 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11900, loss[loss=0.08467, simple_loss=0.1154, pruned_loss=0.01865, audio_tagging_loss=0.00834, over 14622.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09301, pruned_loss=0.01401, audio_tagging_loss=0.009237, over 3035145.26 frames. ], batch size: 54, lr: 2.32e-03, grad_scale: 16.0 2023-11-23 09:21:29,041 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0 2023-11-23 09:21:33,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2323773.3333333335, ans=0.1 2023-11-23 09:21:40,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2323840.0, ans=0.125 2023-11-23 09:21:41,718 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0 2023-11-23 09:21:47,793 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.63 vs. limit=22.5 2023-11-23 09:22:00,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2323906.6666666665, ans=0.125 2023-11-23 09:22:06,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348600 2023-11-23 09:22:23,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.43 vs. limit=22.5 2023-11-23 09:22:29,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2324040.0, ans=0.1 2023-11-23 09:22:31,775 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 11950, loss[loss=0.06721, simple_loss=0.09826, pruned_loss=0.009647, audio_tagging_loss=0.008433, over 15619.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09334, pruned_loss=0.01406, audio_tagging_loss=0.009247, over 3039940.55 frames. ], batch size: 57, lr: 2.32e-03, grad_scale: 16.0 2023-11-23 09:22:32,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2324106.6666666665, ans=0.0 2023-11-23 09:22:34,224 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.393e+01 8.390e+01 8.982e+01 9.534e+01 1.652e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 09:23:08,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2324306.6666666665, ans=0.125 2023-11-23 09:23:11,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348650 2023-11-23 09:23:17,351 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0 2023-11-23 09:23:23,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2324373.3333333335, ans=0.125 2023-11-23 09:23:33,670 INFO [train_asr.py:1221] (3/4) Epoch 29, batch 12000, loss[loss=0.0781, simple_loss=0.1108, pruned_loss=0.01282, audio_tagging_loss=0.009896, over 15126.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09296, pruned_loss=0.01405, audio_tagging_loss=0.009381, over 3041203.05 frames. ], batch size: 56, lr: 2.32e-03, grad_scale: 32.0 2023-11-23 09:23:33,671 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 09:23:57,160 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.7565, 4.0107, 4.0581, 3.9190], device='cuda:3') 2023-11-23 09:24:07,010 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.9772, 5.8572, 5.6391, 5.5696], device='cuda:3') 2023-11-23 09:24:15,955 INFO [train_asr.py:1253] (3/4) Epoch 29, validation: loss=0.05844, simple_loss=0.05118, pruned_loss=0.005114, audio_tagging_loss=0.02774, over 4681554.00 frames. 2023-11-23 09:24:15,956 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 09:24:26,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2324506.6666666665, ans=0.0 2023-11-23 09:24:38,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2324573.3333333335, ans=0.0 2023-11-23 09:24:39,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2324573.3333333335, ans=0.125 2023-11-23 09:24:41,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2324573.3333333335, ans=0.125 2023-11-23 09:25:22,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2324600.0, ans=0.1 2023-11-23 09:25:23,507 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 0, loss[loss=0.06944, simple_loss=0.05853, pruned_loss=0.008359, audio_tagging_loss=0.03182, over 15169.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.05853, pruned_loss=0.008359, audio_tagging_loss=0.03182, over 15169.00 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:25:23,507 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 09:25:45,086 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.0997, 4.9675, 3.7074, 4.2679], device='cuda:3') 2023-11-23 09:25:58,461 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9350, 3.7373, 4.9007, 4.3738], device='cuda:3') 2023-11-23 09:26:02,080 INFO [train_asr.py:1253] (3/4) Epoch 30, validation: loss=0.05824, simple_loss=0.05113, pruned_loss=0.005061, audio_tagging_loss=0.02761, over 4681554.00 frames. 2023-11-23 09:26:02,081 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 09:26:09,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2324600.0, ans=0.2 2023-11-23 09:26:11,951 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348700 2023-11-23 09:26:13,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2324666.6666666665, ans=0.125 2023-11-23 09:26:35,754 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.954e+01 9.660e+01 1.053e+02 1.291e+02, threshold=1.932e+02, percent-clipped=0.0 2023-11-23 09:26:53,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=15.0 2023-11-23 09:27:02,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2324866.6666666665, ans=0.2 2023-11-23 09:27:05,460 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 50, loss[loss=0.08615, simple_loss=0.1135, pruned_loss=0.01556, audio_tagging_loss=0.01382, over 14717.00 frames. ], tot_loss[loss=0.07894, simple_loss=0.09422, pruned_loss=0.0143, audio_tagging_loss=0.01753, over 689181.33 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:27:15,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348750 2023-11-23 09:27:16,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2325000.0, ans=0.1 2023-11-23 09:27:20,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2325000.0, ans=0.125 2023-11-23 09:27:28,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2325000.0, ans=0.125 2023-11-23 09:27:35,592 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.19 vs. limit=12.0 2023-11-23 09:27:40,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2325066.6666666665, ans=0.125 2023-11-23 09:28:06,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2325200.0, ans=0.0 2023-11-23 09:28:08,365 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 100, loss[loss=0.08027, simple_loss=0.1008, pruned_loss=0.01763, audio_tagging_loss=0.01221, over 15189.00 frames. ], tot_loss[loss=0.07814, simple_loss=0.09424, pruned_loss=0.01428, audio_tagging_loss=0.01674, over 1206694.79 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:28:12,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.44 vs. limit=6.0 2023-11-23 09:28:18,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348800 2023-11-23 09:28:42,127 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.31 vs. limit=15.0 2023-11-23 09:28:42,920 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:28:46,352 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.439e+01 8.985e+01 9.565e+01 1.028e+02 2.272e+02, threshold=1.913e+02, percent-clipped=1.0 2023-11-23 09:28:46,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2325466.6666666665, ans=0.125 2023-11-23 09:29:13,424 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 150, loss[loss=0.06641, simple_loss=0.08468, pruned_loss=0.01119, audio_tagging_loss=0.01288, over 14378.00 frames. ], tot_loss[loss=0.07761, simple_loss=0.0956, pruned_loss=0.01479, audio_tagging_loss=0.01501, over 1613673.87 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:29:24,409 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348850 2023-11-23 09:29:24,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2325600.0, ans=0.125 2023-11-23 09:29:27,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2325666.6666666665, ans=0.125 2023-11-23 09:29:34,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2325666.6666666665, ans=0.2 2023-11-23 09:29:43,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2325733.3333333335, ans=0.125 2023-11-23 09:29:46,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2325733.3333333335, ans=0.125 2023-11-23 09:29:51,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2325800.0, ans=0.125 2023-11-23 09:29:52,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2325800.0, ans=0.125 2023-11-23 09:30:00,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2325800.0, ans=0.125 2023-11-23 09:30:08,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2325866.6666666665, ans=0.035 2023-11-23 09:30:18,421 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 200, loss[loss=0.07501, simple_loss=0.1049, pruned_loss=0.01408, audio_tagging_loss=0.008487, over 16630.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.09635, pruned_loss=0.01477, audio_tagging_loss=0.01331, over 1935911.12 frames. ], batch size: 62, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:30:28,229 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348900 2023-11-23 09:30:36,148 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.24 vs. limit=15.0 2023-11-23 09:30:55,306 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.392e+01 9.007e+01 9.652e+01 1.196e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 09:31:03,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2326133.3333333335, ans=0.125 2023-11-23 09:31:21,702 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 250, loss[loss=0.08955, simple_loss=0.1279, pruned_loss=0.02028, audio_tagging_loss=0.005322, over 16222.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09569, pruned_loss=0.01458, audio_tagging_loss=0.01193, over 2180378.16 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:31:31,591 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 348950 2023-11-23 09:31:33,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2326333.3333333335, ans=0.125 2023-11-23 09:31:54,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.45 vs. limit=15.0 2023-11-23 09:32:16,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2326533.3333333335, ans=0.0 2023-11-23 09:32:18,941 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:32:23,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2326533.3333333335, ans=0.0 2023-11-23 09:32:26,057 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 300, loss[loss=0.06741, simple_loss=0.09034, pruned_loss=0.01455, audio_tagging_loss=0.007686, over 15010.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09523, pruned_loss=0.0144, audio_tagging_loss=0.01104, over 2370326.96 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:32:36,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349000 2023-11-23 09:33:02,992 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.407e+01 8.907e+01 9.577e+01 1.241e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-23 09:33:08,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2326800.0, ans=0.1 2023-11-23 09:33:26,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2326866.6666666665, ans=0.0 2023-11-23 09:33:30,927 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 350, loss[loss=0.05431, simple_loss=0.07118, pruned_loss=0.006981, audio_tagging_loss=0.01174, over 15286.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09494, pruned_loss=0.01432, audio_tagging_loss=0.01045, over 2524129.85 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 09:33:40,827 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349050 2023-11-23 09:33:48,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2327000.0, ans=0.125 2023-11-23 09:33:53,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2327000.0, ans=0.125 2023-11-23 09:33:58,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2327066.6666666665, ans=0.125 2023-11-23 09:34:04,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2327066.6666666665, ans=0.0 2023-11-23 09:34:16,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2327133.3333333335, ans=0.2 2023-11-23 09:34:34,498 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 400, loss[loss=0.0646, simple_loss=0.08413, pruned_loss=0.01243, audio_tagging_loss=0.01011, over 15801.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.0939, pruned_loss=0.0142, audio_tagging_loss=0.0101, over 2640884.51 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:34:43,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2327266.6666666665, ans=0.2 2023-11-23 09:34:44,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349100 2023-11-23 09:34:53,753 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.90 vs. limit=10.0 2023-11-23 09:35:02,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2327400.0, ans=0.2 2023-11-23 09:35:04,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.61 vs. limit=15.0 2023-11-23 09:35:13,632 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.415e+01 9.145e+01 9.820e+01 1.379e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-23 09:35:24,039 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.43 vs. limit=22.5 2023-11-23 09:35:34,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2327533.3333333335, ans=0.125 2023-11-23 09:35:39,520 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 450, loss[loss=0.07711, simple_loss=0.1059, pruned_loss=0.01803, audio_tagging_loss=0.006151, over 14901.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09343, pruned_loss=0.01403, audio_tagging_loss=0.009739, over 2731323.53 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:35:40,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.16 vs. limit=15.0 2023-11-23 09:35:50,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349150 2023-11-23 09:35:52,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2327666.6666666665, ans=0.0 2023-11-23 09:36:11,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2327733.3333333335, ans=0.125 2023-11-23 09:36:12,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.19 vs. limit=15.0 2023-11-23 09:36:43,354 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 500, loss[loss=0.07579, simple_loss=0.09715, pruned_loss=0.02, audio_tagging_loss=0.007212, over 16150.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09398, pruned_loss=0.01432, audio_tagging_loss=0.009507, over 2798465.76 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:36:46,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2327933.3333333335, ans=0.125 2023-11-23 09:36:46,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2327933.3333333335, ans=0.1 2023-11-23 09:36:53,751 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349200 2023-11-23 09:37:22,399 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 8.359e+01 8.809e+01 9.526e+01 1.262e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-23 09:37:35,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2328200.0, ans=0.125 2023-11-23 09:37:36,279 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.71 vs. limit=15.0 2023-11-23 09:37:37,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2328200.0, ans=0.125 2023-11-23 09:37:39,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=23.10 vs. limit=22.5 2023-11-23 09:37:44,201 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.57 vs. limit=15.0 2023-11-23 09:37:46,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2328200.0, ans=0.0 2023-11-23 09:37:48,456 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 550, loss[loss=0.05747, simple_loss=0.07956, pruned_loss=0.008173, audio_tagging_loss=0.009515, over 16519.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09441, pruned_loss=0.01423, audio_tagging_loss=0.00939, over 2850499.84 frames. ], batch size: 62, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:37:58,388 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349250 2023-11-23 09:38:12,433 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.13 vs. limit=22.5 2023-11-23 09:38:22,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2328400.0, ans=0.125 2023-11-23 09:38:27,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2328466.6666666665, ans=0.2 2023-11-23 09:38:42,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.50 vs. limit=15.0 2023-11-23 09:38:52,685 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 600, loss[loss=0.04441, simple_loss=0.05412, pruned_loss=0.007145, audio_tagging_loss=0.0102, over 15541.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09367, pruned_loss=0.01411, audio_tagging_loss=0.009383, over 2887260.25 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:39:03,366 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349300 2023-11-23 09:39:09,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2328666.6666666665, ans=0.125 2023-11-23 09:39:13,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2328666.6666666665, ans=0.125 2023-11-23 09:39:16,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2328666.6666666665, ans=0.125 2023-11-23 09:39:31,023 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.142e+01 8.700e+01 9.475e+01 1.259e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-23 09:39:46,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.88 vs. limit=15.0 2023-11-23 09:39:51,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.42 vs. limit=6.0 2023-11-23 09:39:52,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2328866.6666666665, ans=0.125 2023-11-23 09:39:57,388 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 650, loss[loss=0.07704, simple_loss=0.1053, pruned_loss=0.01648, audio_tagging_loss=0.007897, over 15340.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09268, pruned_loss=0.01406, audio_tagging_loss=0.009383, over 2917275.48 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:39:58,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2328933.3333333335, ans=0.125 2023-11-23 09:40:07,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349350 2023-11-23 09:40:11,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2329000.0, ans=0.0 2023-11-23 09:40:13,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2329000.0, ans=0.2 2023-11-23 09:40:17,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.78 vs. limit=22.5 2023-11-23 09:40:19,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2329000.0, ans=0.035 2023-11-23 09:40:24,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2329066.6666666665, ans=0.1 2023-11-23 09:40:35,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2329133.3333333335, ans=0.125 2023-11-23 09:40:43,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2329133.3333333335, ans=0.125 2023-11-23 09:40:49,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2329200.0, ans=0.125 2023-11-23 09:40:50,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2329200.0, ans=0.125 2023-11-23 09:41:02,197 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 700, loss[loss=0.06297, simple_loss=0.09205, pruned_loss=0.007532, audio_tagging_loss=0.009417, over 14025.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09168, pruned_loss=0.01373, audio_tagging_loss=0.009404, over 2942665.52 frames. ], batch size: 51, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:41:02,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2329266.6666666665, ans=0.0 2023-11-23 09:41:07,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2329266.6666666665, ans=0.125 2023-11-23 09:41:12,063 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349400 2023-11-23 09:41:23,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2329333.3333333335, ans=0.125 2023-11-23 09:41:28,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2329400.0, ans=0.0 2023-11-23 09:41:41,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.086e+01 8.675e+01 9.792e+01 1.198e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-23 09:41:41,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2329466.6666666665, ans=0.125 2023-11-23 09:41:41,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2329466.6666666665, ans=0.1 2023-11-23 09:41:52,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2329533.3333333335, ans=0.05 2023-11-23 09:41:54,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2329533.3333333335, ans=0.0 2023-11-23 09:41:58,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2329533.3333333335, ans=0.04949747468305833 2023-11-23 09:42:06,223 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 750, loss[loss=0.06987, simple_loss=0.09897, pruned_loss=0.01324, audio_tagging_loss=0.007147, over 15529.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.0918, pruned_loss=0.01382, audio_tagging_loss=0.009331, over 2967317.33 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:42:17,154 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349450 2023-11-23 09:42:30,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.90 vs. limit=6.0 2023-11-23 09:43:00,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2329866.6666666665, ans=0.1 2023-11-23 09:43:12,049 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 800, loss[loss=0.09079, simple_loss=0.1289, pruned_loss=0.01813, audio_tagging_loss=0.008232, over 14996.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09169, pruned_loss=0.01372, audio_tagging_loss=0.009426, over 2980334.34 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:43:21,736 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349500 2023-11-23 09:43:26,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2330000.0, ans=0.125 2023-11-23 09:43:50,463 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.139e+01 8.858e+01 9.489e+01 1.259e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 09:43:50,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2330133.3333333335, ans=0.1 2023-11-23 09:44:01,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2330133.3333333335, ans=0.95 2023-11-23 09:44:14,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2330266.6666666665, ans=0.125 2023-11-23 09:44:15,728 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 850, loss[loss=0.07224, simple_loss=0.09222, pruned_loss=0.01389, audio_tagging_loss=0.01224, over 14890.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09167, pruned_loss=0.01369, audio_tagging_loss=0.009572, over 2999505.17 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:44:26,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349550 2023-11-23 09:44:49,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2330400.0, ans=0.125 2023-11-23 09:44:54,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2330466.6666666665, ans=0.0 2023-11-23 09:45:05,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2330533.3333333335, ans=0.95 2023-11-23 09:45:09,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2330533.3333333335, ans=0.2 2023-11-23 09:45:10,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2330533.3333333335, ans=0.0 2023-11-23 09:45:17,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2330533.3333333335, ans=0.0 2023-11-23 09:45:19,625 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 900, loss[loss=0.07032, simple_loss=0.09496, pruned_loss=0.0137, audio_tagging_loss=0.009146, over 15903.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09142, pruned_loss=0.01357, audio_tagging_loss=0.009585, over 3012156.90 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:45:30,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349600 2023-11-23 09:45:32,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2330666.6666666665, ans=0.1 2023-11-23 09:45:51,228 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:45:58,097 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.328e+01 8.818e+01 9.334e+01 1.944e+02, threshold=1.764e+02, percent-clipped=1.0 2023-11-23 09:45:58,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2330800.0, ans=0.0 2023-11-23 09:46:14,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2330866.6666666665, ans=0.125 2023-11-23 09:46:17,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2330866.6666666665, ans=0.125 2023-11-23 09:46:23,767 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.73 vs. limit=22.5 2023-11-23 09:46:24,195 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 950, loss[loss=0.06545, simple_loss=0.08403, pruned_loss=0.01574, audio_tagging_loss=0.0077, over 13916.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09193, pruned_loss=0.01376, audio_tagging_loss=0.009471, over 3022552.69 frames. ], batch size: 52, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:46:34,459 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349650 2023-11-23 09:46:34,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2330933.3333333335, ans=0.0 2023-11-23 09:46:38,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=2331000.0, ans=0.02 2023-11-23 09:46:47,236 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.06 vs. limit=12.0 2023-11-23 09:46:54,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5 2023-11-23 09:47:14,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2331200.0, ans=0.125 2023-11-23 09:47:28,214 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1000, loss[loss=0.0781, simple_loss=0.1113, pruned_loss=0.01348, audio_tagging_loss=0.008976, over 15427.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09173, pruned_loss=0.01382, audio_tagging_loss=0.009312, over 3031542.22 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:47:38,063 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349700 2023-11-23 09:47:57,306 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 09:47:59,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.02 vs. limit=22.5 2023-11-23 09:48:07,584 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.508e+01 8.244e+01 9.173e+01 9.794e+01 1.161e+02, threshold=1.835e+02, percent-clipped=0.0 2023-11-23 09:48:26,574 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.91 vs. limit=15.0 2023-11-23 09:48:32,089 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1050, loss[loss=0.0774, simple_loss=0.1007, pruned_loss=0.01852, audio_tagging_loss=0.008525, over 14825.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09149, pruned_loss=0.01386, audio_tagging_loss=0.009207, over 3031298.29 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:48:43,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349750 2023-11-23 09:49:26,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2331866.6666666665, ans=0.0 2023-11-23 09:49:37,738 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1100, loss[loss=0.06569, simple_loss=0.08327, pruned_loss=0.01573, audio_tagging_loss=0.00833, over 14642.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09133, pruned_loss=0.01386, audio_tagging_loss=0.009126, over 3039094.60 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:49:41,468 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 09:49:41,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2331933.3333333335, ans=0.2 2023-11-23 09:49:47,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2331933.3333333335, ans=15.0 2023-11-23 09:49:48,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349800 2023-11-23 09:50:02,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2332066.6666666665, ans=0.125 2023-11-23 09:50:10,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2332066.6666666665, ans=0.0 2023-11-23 09:50:15,126 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 8.049e+01 8.629e+01 9.475e+01 1.161e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-23 09:50:25,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2332133.3333333335, ans=0.0 2023-11-23 09:50:30,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2332200.0, ans=0.125 2023-11-23 09:50:33,979 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.72 vs. limit=22.5 2023-11-23 09:50:42,009 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1150, loss[loss=0.08632, simple_loss=0.1145, pruned_loss=0.01871, audio_tagging_loss=0.01035, over 15244.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09194, pruned_loss=0.01396, audio_tagging_loss=0.008962, over 3042898.74 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:50:44,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2332266.6666666665, ans=0.025 2023-11-23 09:50:52,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349850 2023-11-23 09:50:58,734 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.31 vs. limit=15.0 2023-11-23 09:51:00,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2332333.3333333335, ans=0.0 2023-11-23 09:51:43,481 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.90 vs. limit=15.0 2023-11-23 09:51:45,118 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1200, loss[loss=0.07411, simple_loss=0.1075, pruned_loss=0.01331, audio_tagging_loss=0.007024, over 15916.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09207, pruned_loss=0.01408, audio_tagging_loss=0.008941, over 3042996.92 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:51:54,954 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349900 2023-11-23 09:52:21,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2332733.3333333335, ans=0.125 2023-11-23 09:52:24,714 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.340e+01 9.039e+01 9.535e+01 1.288e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-23 09:52:38,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2332866.6666666665, ans=0.1 2023-11-23 09:52:45,303 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:52:49,377 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1250, loss[loss=0.06619, simple_loss=0.0919, pruned_loss=0.01233, audio_tagging_loss=0.007914, over 15301.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09135, pruned_loss=0.01396, audio_tagging_loss=0.008993, over 3041008.63 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:52:50,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2332933.3333333335, ans=0.125 2023-11-23 09:52:59,695 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 349950 2023-11-23 09:53:14,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2333066.6666666665, ans=0.04949747468305833 2023-11-23 09:53:14,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2333066.6666666665, ans=0.125 2023-11-23 09:53:25,939 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.09 vs. limit=22.5 2023-11-23 09:53:32,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2333133.3333333335, ans=0.1 2023-11-23 09:53:48,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2333200.0, ans=0.2 2023-11-23 09:53:52,859 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1300, loss[loss=0.05733, simple_loss=0.07725, pruned_loss=0.009528, audio_tagging_loss=0.009175, over 14973.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09103, pruned_loss=0.01381, audio_tagging_loss=0.009009, over 3037048.13 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:54:03,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350000 2023-11-23 09:54:06,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2333333.3333333335, ans=0.0 2023-11-23 09:54:14,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2333333.3333333335, ans=0.1 2023-11-23 09:54:28,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2333400.0, ans=0.125 2023-11-23 09:54:32,747 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.017e+01 7.999e+01 8.654e+01 9.564e+01 1.101e+02, threshold=1.731e+02, percent-clipped=0.0 2023-11-23 09:54:48,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2333533.3333333335, ans=0.1 2023-11-23 09:54:56,652 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1350, loss[loss=0.05011, simple_loss=0.06144, pruned_loss=0.006733, audio_tagging_loss=0.01266, over 15373.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09095, pruned_loss=0.01385, audio_tagging_loss=0.009, over 3036748.71 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:55:06,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350050 2023-11-23 09:55:10,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2333666.6666666665, ans=0.0 2023-11-23 09:55:27,252 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 09:55:29,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2333733.3333333335, ans=0.1 2023-11-23 09:55:34,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2333800.0, ans=0.0 2023-11-23 09:55:43,886 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 09:56:00,479 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1400, loss[loss=0.06698, simple_loss=0.09194, pruned_loss=0.01262, audio_tagging_loss=0.008389, over 15790.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09026, pruned_loss=0.01362, audio_tagging_loss=0.009129, over 3044169.07 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:56:09,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.82 vs. limit=15.0 2023-11-23 09:56:11,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350100 2023-11-23 09:56:20,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2334000.0, ans=0.0 2023-11-23 09:56:31,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2334066.6666666665, ans=0.125 2023-11-23 09:56:37,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2334133.3333333335, ans=0.07 2023-11-23 09:56:39,712 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 8.084e+01 8.982e+01 9.535e+01 1.222e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 09:56:43,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2334133.3333333335, ans=0.1 2023-11-23 09:56:50,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2334200.0, ans=0.07 2023-11-23 09:56:51,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.07 vs. limit=15.0 2023-11-23 09:57:04,687 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1450, loss[loss=0.06188, simple_loss=0.07326, pruned_loss=0.01595, audio_tagging_loss=0.009301, over 15840.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09031, pruned_loss=0.01368, audio_tagging_loss=0.009217, over 3041964.76 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 09:57:14,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350150 2023-11-23 09:57:14,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=15.0 2023-11-23 09:57:15,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2334333.3333333335, ans=0.125 2023-11-23 09:57:18,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2334333.3333333335, ans=0.125 2023-11-23 09:57:21,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2334333.3333333335, ans=0.125 2023-11-23 09:57:29,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2334400.0, ans=0.0 2023-11-23 09:57:29,464 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.22 vs. limit=15.0 2023-11-23 09:57:39,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2334400.0, ans=0.0 2023-11-23 09:57:41,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=22.5 2023-11-23 09:57:44,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2334466.6666666665, ans=0.125 2023-11-23 09:57:51,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2334466.6666666665, ans=0.2 2023-11-23 09:58:06,407 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1500, loss[loss=0.06404, simple_loss=0.08604, pruned_loss=0.01213, audio_tagging_loss=0.008882, over 15919.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09122, pruned_loss=0.0139, audio_tagging_loss=0.009203, over 3036016.88 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 09:58:10,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2334600.0, ans=0.125 2023-11-23 09:58:16,313 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350200 2023-11-23 09:58:30,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2334733.3333333335, ans=0.125 2023-11-23 09:58:39,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2334733.3333333335, ans=0.0 2023-11-23 09:58:48,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.399e+01 9.056e+01 9.468e+01 1.678e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-23 09:58:55,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2334800.0, ans=0.125 2023-11-23 09:58:57,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2334866.6666666665, ans=0.0 2023-11-23 09:58:57,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2334866.6666666665, ans=0.1 2023-11-23 09:58:57,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2334866.6666666665, ans=0.1 2023-11-23 09:58:58,568 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.91 vs. limit=15.0 2023-11-23 09:59:10,063 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1550, loss[loss=0.05735, simple_loss=0.07016, pruned_loss=0.01095, audio_tagging_loss=0.01133, over 14782.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09254, pruned_loss=0.01419, audio_tagging_loss=0.00928, over 3036921.30 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 09:59:17,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2334933.3333333335, ans=0.125 2023-11-23 09:59:20,632 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350250 2023-11-23 09:59:20,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2334933.3333333335, ans=0.1 2023-11-23 09:59:25,461 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.21 vs. limit=10.0 2023-11-23 09:59:28,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2335000.0, ans=0.2 2023-11-23 09:59:31,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2335000.0, ans=0.125 2023-11-23 09:59:32,112 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.53 vs. limit=15.0 2023-11-23 09:59:55,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2335133.3333333335, ans=0.2 2023-11-23 10:00:10,996 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.48 vs. limit=15.0 2023-11-23 10:00:14,406 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1600, loss[loss=0.07611, simple_loss=0.09611, pruned_loss=0.01553, audio_tagging_loss=0.01252, over 15303.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09269, pruned_loss=0.01436, audio_tagging_loss=0.009335, over 3036973.35 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:00:24,791 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350300 2023-11-23 10:00:42,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0 2023-11-23 10:00:52,900 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.88 vs. limit=15.0 2023-11-23 10:00:55,566 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.176e+01 8.486e+01 9.012e+01 9.974e+01 1.250e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-23 10:01:06,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2335533.3333333335, ans=0.1 2023-11-23 10:01:12,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2335533.3333333335, ans=0.2 2023-11-23 10:01:14,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2335533.3333333335, ans=0.0 2023-11-23 10:01:17,418 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1650, loss[loss=0.06242, simple_loss=0.0878, pruned_loss=0.01086, audio_tagging_loss=0.007665, over 15893.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09249, pruned_loss=0.01415, audio_tagging_loss=0.009473, over 3045530.13 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:01:27,201 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350350 2023-11-23 10:01:39,928 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0 2023-11-23 10:02:05,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.69 vs. limit=22.5 2023-11-23 10:02:16,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2335866.6666666665, ans=0.0 2023-11-23 10:02:20,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2335933.3333333335, ans=0.125 2023-11-23 10:02:21,315 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1700, loss[loss=0.06887, simple_loss=0.09229, pruned_loss=0.0118, audio_tagging_loss=0.01092, over 15722.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09329, pruned_loss=0.01424, audio_tagging_loss=0.009359, over 3045881.33 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 10:02:31,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350400 2023-11-23 10:02:38,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2336000.0, ans=0.0 2023-11-23 10:02:46,335 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 10:02:54,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2336066.6666666665, ans=0.07 2023-11-23 10:03:04,971 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.246e+01 8.760e+01 9.410e+01 1.135e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-23 10:03:05,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2336133.3333333335, ans=0.125 2023-11-23 10:03:06,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.02 vs. limit=15.0 2023-11-23 10:03:25,711 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1750, loss[loss=0.05807, simple_loss=0.07308, pruned_loss=0.008003, audio_tagging_loss=0.01353, over 16810.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09233, pruned_loss=0.01394, audio_tagging_loss=0.009286, over 3046305.26 frames. ], batch size: 64, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 10:03:28,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.10 vs. limit=10.0 2023-11-23 10:03:29,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2336266.6666666665, ans=0.0 2023-11-23 10:03:36,512 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350450 2023-11-23 10:04:04,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2336466.6666666665, ans=0.1 2023-11-23 10:04:22,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2336533.3333333335, ans=0.125 2023-11-23 10:04:31,144 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1800, loss[loss=0.04413, simple_loss=0.05471, pruned_loss=0.007131, audio_tagging_loss=0.009646, over 15299.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09253, pruned_loss=0.0139, audio_tagging_loss=0.009178, over 3042196.97 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 10:04:35,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2336600.0, ans=0.07 2023-11-23 10:04:35,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2336600.0, ans=0.1 2023-11-23 10:04:41,007 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350500 2023-11-23 10:04:41,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2336600.0, ans=0.2 2023-11-23 10:05:03,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2336733.3333333335, ans=0.0 2023-11-23 10:05:06,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2336733.3333333335, ans=0.125 2023-11-23 10:05:13,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2336800.0, ans=0.125 2023-11-23 10:05:15,199 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.590e+01 9.171e+01 9.752e+01 1.397e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-23 10:05:25,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2336866.6666666665, ans=0.125 2023-11-23 10:05:26,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2336866.6666666665, ans=0.0 2023-11-23 10:05:35,403 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1850, loss[loss=0.0907, simple_loss=0.1256, pruned_loss=0.02064, audio_tagging_loss=0.007255, over 16369.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.0926, pruned_loss=0.01398, audio_tagging_loss=0.009127, over 3037601.94 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 10:05:45,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2336933.3333333335, ans=0.125 2023-11-23 10:05:46,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350550 2023-11-23 10:05:51,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2337000.0, ans=0.2 2023-11-23 10:05:57,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2337000.0, ans=0.125 2023-11-23 10:05:59,953 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.70 vs. limit=15.0 2023-11-23 10:06:37,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2337200.0, ans=10.0 2023-11-23 10:06:39,834 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1900, loss[loss=0.0554, simple_loss=0.07762, pruned_loss=0.007009, audio_tagging_loss=0.009583, over 15826.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09217, pruned_loss=0.01382, audio_tagging_loss=0.00904, over 3041458.39 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 10:06:49,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2337266.6666666665, ans=0.0 2023-11-23 10:06:50,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350600 2023-11-23 10:06:51,737 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 10:07:04,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.01 vs. limit=8.0 2023-11-23 10:07:11,651 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.41 vs. limit=22.5 2023-11-23 10:07:24,842 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.717e+01 8.255e+01 9.028e+01 9.866e+01 1.250e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-23 10:07:45,202 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 1950, loss[loss=0.05766, simple_loss=0.06698, pruned_loss=0.01024, audio_tagging_loss=0.01393, over 14817.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09195, pruned_loss=0.01381, audio_tagging_loss=0.00907, over 3041534.74 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 8.0 2023-11-23 10:07:47,026 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.78 vs. limit=15.0 2023-11-23 10:07:54,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2337600.0, ans=0.125 2023-11-23 10:07:54,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2337600.0, ans=0.125 2023-11-23 10:07:55,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350650 2023-11-23 10:08:17,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2337733.3333333335, ans=0.125 2023-11-23 10:08:25,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2337800.0, ans=0.125 2023-11-23 10:08:28,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2337800.0, ans=0.0 2023-11-23 10:08:51,213 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2000, loss[loss=0.07209, simple_loss=0.1054, pruned_loss=0.009633, audio_tagging_loss=0.00978, over 14670.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.0908, pruned_loss=0.01364, audio_tagging_loss=0.009198, over 3039113.47 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:09:02,131 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350700 2023-11-23 10:09:06,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2338000.0, ans=0.125 2023-11-23 10:09:35,805 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.268e+01 8.767e+01 9.455e+01 1.201e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-23 10:09:38,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2338133.3333333335, ans=0.1 2023-11-23 10:09:57,056 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2050, loss[loss=0.06932, simple_loss=0.08996, pruned_loss=0.01403, audio_tagging_loss=0.01031, over 15084.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09216, pruned_loss=0.01393, audio_tagging_loss=0.009122, over 3037109.73 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:10:07,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350750 2023-11-23 10:10:15,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2338333.3333333335, ans=0.125 2023-11-23 10:11:01,824 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2100, loss[loss=0.07424, simple_loss=0.0942, pruned_loss=0.01796, audio_tagging_loss=0.009187, over 15335.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09255, pruned_loss=0.01393, audio_tagging_loss=0.009094, over 3042667.29 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:11:10,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2338600.0, ans=0.0 2023-11-23 10:11:11,738 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350800 2023-11-23 10:11:26,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2338733.3333333335, ans=0.125 2023-11-23 10:11:46,476 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.972e+01 8.232e+01 8.810e+01 9.514e+01 1.231e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-23 10:11:54,704 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0 2023-11-23 10:11:59,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2338866.6666666665, ans=0.07 2023-11-23 10:12:02,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2338866.6666666665, ans=0.0 2023-11-23 10:12:02,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.02 vs. limit=6.0 2023-11-23 10:12:06,521 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2150, loss[loss=0.09166, simple_loss=0.1244, pruned_loss=0.02052, audio_tagging_loss=0.008946, over 16946.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.0927, pruned_loss=0.01396, audio_tagging_loss=0.009144, over 3040848.12 frames. ], batch size: 62, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:12:11,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2338933.3333333335, ans=0.2 2023-11-23 10:12:15,281 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.18 vs. limit=15.0 2023-11-23 10:12:17,142 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350850 2023-11-23 10:12:23,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2339000.0, ans=0.1 2023-11-23 10:12:41,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2339066.6666666665, ans=0.1 2023-11-23 10:12:41,532 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.95 vs. limit=15.0 2023-11-23 10:12:47,108 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:13:09,902 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.52 vs. limit=10.0 2023-11-23 10:13:12,303 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2200, loss[loss=0.06418, simple_loss=0.08281, pruned_loss=0.01363, audio_tagging_loss=0.009144, over 15464.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09265, pruned_loss=0.01411, audio_tagging_loss=0.009195, over 3047308.91 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:13:13,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2339266.6666666665, ans=0.0 2023-11-23 10:13:22,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350900 2023-11-23 10:13:22,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2339266.6666666665, ans=0.0 2023-11-23 10:13:48,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2339466.6666666665, ans=0.125 2023-11-23 10:13:55,232 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.264e+01 8.958e+01 9.575e+01 1.152e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-23 10:13:57,715 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.58 vs. limit=12.0 2023-11-23 10:14:00,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2339466.6666666665, ans=0.0 2023-11-23 10:14:10,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2339533.3333333335, ans=0.2 2023-11-23 10:14:16,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2339600.0, ans=0.1 2023-11-23 10:14:17,024 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2250, loss[loss=0.05271, simple_loss=0.06381, pruned_loss=0.008342, audio_tagging_loss=0.01247, over 15065.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09372, pruned_loss=0.01439, audio_tagging_loss=0.00916, over 3044771.35 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:14:19,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2339600.0, ans=0.125 2023-11-23 10:14:27,002 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 350950 2023-11-23 10:14:42,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.34 vs. limit=12.0 2023-11-23 10:14:45,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2339733.3333333335, ans=0.125 2023-11-23 10:15:21,431 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2300, loss[loss=0.04215, simple_loss=0.04981, pruned_loss=0.00408, audio_tagging_loss=0.01316, over 15224.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09344, pruned_loss=0.01429, audio_tagging_loss=0.009277, over 3045959.06 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:15:22,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2339933.3333333335, ans=0.125 2023-11-23 10:15:31,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351000 2023-11-23 10:15:39,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2340000.0, ans=0.2 2023-11-23 10:15:41,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2340000.0, ans=0.0 2023-11-23 10:15:44,841 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 10:15:48,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2340066.6666666665, ans=15.0 2023-11-23 10:15:51,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2340066.6666666665, ans=0.125 2023-11-23 10:15:51,677 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 10:16:02,642 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 10:16:06,089 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.543e+01 9.090e+01 9.740e+01 1.795e+02, threshold=1.818e+02, percent-clipped=1.0 2023-11-23 10:16:11,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2340133.3333333335, ans=0.05 2023-11-23 10:16:18,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2340200.0, ans=0.125 2023-11-23 10:16:19,862 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:16:23,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2340200.0, ans=0.125 2023-11-23 10:16:26,864 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2350, loss[loss=0.08105, simple_loss=0.1043, pruned_loss=0.01665, audio_tagging_loss=0.01226, over 14717.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.0941, pruned_loss=0.0144, audio_tagging_loss=0.009226, over 3048550.97 frames. ], batch size: 53, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:16:31,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2340266.6666666665, ans=0.09899494936611666 2023-11-23 10:16:38,054 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351050 2023-11-23 10:16:55,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2340400.0, ans=0.07 2023-11-23 10:17:05,387 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.22 vs. limit=15.0 2023-11-23 10:17:18,566 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.01 vs. limit=22.5 2023-11-23 10:17:24,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2340533.3333333335, ans=0.1 2023-11-23 10:17:33,020 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2400, loss[loss=0.07219, simple_loss=0.08886, pruned_loss=0.01301, audio_tagging_loss=0.01475, over 15729.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09388, pruned_loss=0.01454, audio_tagging_loss=0.009401, over 3051437.39 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:17:42,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351100 2023-11-23 10:17:50,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2340666.6666666665, ans=0.0 2023-11-23 10:17:54,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.44 vs. limit=22.5 2023-11-23 10:18:01,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2340733.3333333335, ans=0.0 2023-11-23 10:18:08,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2340733.3333333335, ans=0.1 2023-11-23 10:18:15,902 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.266e+01 8.725e+01 9.443e+01 1.197e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-23 10:18:24,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2340866.6666666665, ans=0.0 2023-11-23 10:18:31,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2340866.6666666665, ans=0.0 2023-11-23 10:18:36,509 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2450, loss[loss=0.06179, simple_loss=0.07794, pruned_loss=0.01052, audio_tagging_loss=0.0123, over 14224.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09329, pruned_loss=0.0143, audio_tagging_loss=0.009451, over 3049094.13 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:18:46,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351150 2023-11-23 10:18:49,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2341000.0, ans=0.1 2023-11-23 10:19:00,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2341000.0, ans=0.0 2023-11-23 10:19:03,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2341066.6666666665, ans=0.0 2023-11-23 10:19:04,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2341066.6666666665, ans=0.125 2023-11-23 10:19:35,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2341200.0, ans=0.125 2023-11-23 10:19:41,879 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2500, loss[loss=0.07234, simple_loss=0.1007, pruned_loss=0.01525, audio_tagging_loss=0.006717, over 14220.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.0928, pruned_loss=0.01401, audio_tagging_loss=0.009397, over 3047325.39 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:19:53,981 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351200 2023-11-23 10:20:07,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2341333.3333333335, ans=0.025 2023-11-23 10:20:27,908 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.304e+01 8.850e+01 9.688e+01 1.424e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-23 10:20:29,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.53 vs. limit=15.0 2023-11-23 10:20:36,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2341533.3333333335, ans=0.05 2023-11-23 10:20:39,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2341533.3333333335, ans=0.0 2023-11-23 10:20:51,101 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2550, loss[loss=0.06599, simple_loss=0.09167, pruned_loss=0.01105, audio_tagging_loss=0.009112, over 15512.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09358, pruned_loss=0.01414, audio_tagging_loss=0.009246, over 3040026.31 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:20:52,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2341600.0, ans=0.125 2023-11-23 10:21:01,185 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351250 2023-11-23 10:21:10,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2341666.6666666665, ans=0.125 2023-11-23 10:21:10,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2341666.6666666665, ans=0.125 2023-11-23 10:21:33,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2341800.0, ans=0.125 2023-11-23 10:21:34,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.60 vs. limit=12.0 2023-11-23 10:21:42,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2341800.0, ans=0.0 2023-11-23 10:21:56,974 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2600, loss[loss=0.07293, simple_loss=0.1007, pruned_loss=0.01282, audio_tagging_loss=0.009768, over 14855.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09266, pruned_loss=0.01391, audio_tagging_loss=0.009119, over 3042586.26 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:22:06,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351300 2023-11-23 10:22:16,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2342000.0, ans=0.07 2023-11-23 10:22:37,988 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.84 vs. limit=15.0 2023-11-23 10:22:40,708 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.78 vs. limit=10.0 2023-11-23 10:22:42,340 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.380e+01 9.027e+01 9.591e+01 1.211e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 10:23:02,632 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2650, loss[loss=0.08565, simple_loss=0.1169, pruned_loss=0.01843, audio_tagging_loss=0.008748, over 15594.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09209, pruned_loss=0.01388, audio_tagging_loss=0.009069, over 3046255.96 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:23:04,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2342266.6666666665, ans=0.04949747468305833 2023-11-23 10:23:07,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.87 vs. limit=6.0 2023-11-23 10:23:14,190 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351350 2023-11-23 10:23:37,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2342400.0, ans=0.125 2023-11-23 10:24:06,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2342533.3333333335, ans=0.0 2023-11-23 10:24:09,564 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2700, loss[loss=0.0701, simple_loss=0.1027, pruned_loss=0.009454, audio_tagging_loss=0.009319, over 14949.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09115, pruned_loss=0.01381, audio_tagging_loss=0.009035, over 3046594.13 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:24:20,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351400 2023-11-23 10:24:31,766 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.07 vs. limit=22.5 2023-11-23 10:24:32,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2342666.6666666665, ans=0.125 2023-11-23 10:24:36,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=12.0 2023-11-23 10:24:43,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2342733.3333333335, ans=0.2 2023-11-23 10:24:44,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2342733.3333333335, ans=0.1 2023-11-23 10:24:55,403 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.154e+01 8.358e+01 8.964e+01 9.971e+01 1.230e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-23 10:25:15,062 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2750, loss[loss=0.05183, simple_loss=0.06183, pruned_loss=0.009267, audio_tagging_loss=0.01165, over 15134.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09232, pruned_loss=0.01414, audio_tagging_loss=0.008971, over 3050431.73 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0 2023-11-23 10:25:24,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351450 2023-11-23 10:26:12,093 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:26:19,271 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2800, loss[loss=0.06408, simple_loss=0.08481, pruned_loss=0.01465, audio_tagging_loss=0.007021, over 15575.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09144, pruned_loss=0.01397, audio_tagging_loss=0.009115, over 3049813.48 frames. ], batch size: 63, lr: 2.28e-03, grad_scale: 32.0 2023-11-23 10:26:24,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.86 vs. limit=10.0 2023-11-23 10:26:27,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2343266.6666666665, ans=0.0 2023-11-23 10:26:30,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351500 2023-11-23 10:26:35,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2343333.3333333335, ans=0.0 2023-11-23 10:26:54,772 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.56 vs. limit=15.0 2023-11-23 10:26:57,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2343466.6666666665, ans=0.1 2023-11-23 10:27:05,101 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.185e+01 8.850e+01 9.511e+01 1.118e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-23 10:27:12,002 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.64 vs. limit=15.0 2023-11-23 10:27:24,903 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2850, loss[loss=0.07846, simple_loss=0.1053, pruned_loss=0.01697, audio_tagging_loss=0.008852, over 15946.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09255, pruned_loss=0.01411, audio_tagging_loss=0.009114, over 3056130.76 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:27:35,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351550 2023-11-23 10:27:40,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2343666.6666666665, ans=0.125 2023-11-23 10:27:44,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2343666.6666666665, ans=0.125 2023-11-23 10:28:01,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2343733.3333333335, ans=0.2 2023-11-23 10:28:07,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2343800.0, ans=0.0 2023-11-23 10:28:29,925 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2900, loss[loss=0.04707, simple_loss=0.05795, pruned_loss=0.009104, audio_tagging_loss=0.008989, over 14679.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09238, pruned_loss=0.01421, audio_tagging_loss=0.009099, over 3052084.44 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:28:31,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2343933.3333333335, ans=0.1 2023-11-23 10:28:40,809 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351600 2023-11-23 10:28:42,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0 2023-11-23 10:28:51,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2344000.0, ans=0.0 2023-11-23 10:28:58,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2344066.6666666665, ans=0.125 2023-11-23 10:29:17,046 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.374e+01 8.898e+01 9.788e+01 1.211e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-23 10:29:24,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2344200.0, ans=0.1 2023-11-23 10:29:28,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2344200.0, ans=0.0 2023-11-23 10:29:34,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2344200.0, ans=0.0 2023-11-23 10:29:35,673 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.72 vs. limit=15.0 2023-11-23 10:29:36,333 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 2950, loss[loss=0.06401, simple_loss=0.08663, pruned_loss=0.0113, audio_tagging_loss=0.009403, over 14439.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09205, pruned_loss=0.01423, audio_tagging_loss=0.009156, over 3049929.87 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:29:47,274 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351650 2023-11-23 10:30:36,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2344533.3333333335, ans=0.0 2023-11-23 10:30:41,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2344600.0, ans=0.125 2023-11-23 10:30:42,059 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3000, loss[loss=0.05754, simple_loss=0.07299, pruned_loss=0.00948, audio_tagging_loss=0.01157, over 14822.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09228, pruned_loss=0.01429, audio_tagging_loss=0.009173, over 3049626.20 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:30:42,060 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 10:30:59,451 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.8770, 1.7930, 2.6276, 2.7367, 2.6672, 2.6818, 2.3530, 2.6637], device='cuda:3') 2023-11-23 10:31:20,378 INFO [train_asr.py:1253] (3/4) Epoch 30, validation: loss=0.05789, simple_loss=0.05111, pruned_loss=0.005034, audio_tagging_loss=0.0273, over 4681554.00 frames. 2023-11-23 10:31:20,379 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 10:31:27,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2344600.0, ans=0.125 2023-11-23 10:31:28,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0 2023-11-23 10:31:31,346 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351700 2023-11-23 10:31:35,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2344666.6666666665, ans=0.0 2023-11-23 10:31:42,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2344666.6666666665, ans=0.0 2023-11-23 10:31:55,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2344733.3333333335, ans=0.0 2023-11-23 10:32:06,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.89 vs. limit=10.0 2023-11-23 10:32:08,866 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.435e+01 9.100e+01 1.011e+02 1.193e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-23 10:32:12,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2344866.6666666665, ans=0.125 2023-11-23 10:32:14,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2344866.6666666665, ans=0.125 2023-11-23 10:32:25,957 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3050, loss[loss=0.06952, simple_loss=0.08738, pruned_loss=0.01566, audio_tagging_loss=0.01017, over 14756.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09294, pruned_loss=0.0144, audio_tagging_loss=0.009217, over 3049141.39 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 8.0 2023-11-23 10:32:30,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2344933.3333333335, ans=0.0 2023-11-23 10:32:37,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351750 2023-11-23 10:32:39,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2345000.0, ans=0.125 2023-11-23 10:32:41,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2345000.0, ans=0.0 2023-11-23 10:33:06,574 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:33:19,098 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0 2023-11-23 10:33:22,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2345200.0, ans=0.2 2023-11-23 10:33:28,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.29 vs. limit=22.5 2023-11-23 10:33:32,027 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3100, loss[loss=0.06206, simple_loss=0.07837, pruned_loss=0.01079, audio_tagging_loss=0.01209, over 15916.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09359, pruned_loss=0.01437, audio_tagging_loss=0.009235, over 3048038.01 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 8.0 2023-11-23 10:33:32,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2345266.6666666665, ans=0.125 2023-11-23 10:33:42,780 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351800 2023-11-23 10:34:21,262 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.167e+01 8.412e+01 9.012e+01 9.624e+01 1.358e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-23 10:34:22,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2345466.6666666665, ans=0.0 2023-11-23 10:34:25,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2345533.3333333335, ans=0.125 2023-11-23 10:34:34,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2345533.3333333335, ans=0.0 2023-11-23 10:34:38,164 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3150, loss[loss=0.0876, simple_loss=0.1208, pruned_loss=0.01777, audio_tagging_loss=0.009428, over 15231.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09413, pruned_loss=0.0145, audio_tagging_loss=0.009268, over 3044620.98 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 8.0 2023-11-23 10:34:44,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2345600.0, ans=0.2 2023-11-23 10:34:48,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351850 2023-11-23 10:35:01,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.34 vs. limit=5.0 2023-11-23 10:35:04,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2345733.3333333335, ans=0.0 2023-11-23 10:35:06,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2345733.3333333335, ans=0.125 2023-11-23 10:35:10,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2345733.3333333335, ans=0.0 2023-11-23 10:35:14,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2345733.3333333335, ans=0.125 2023-11-23 10:35:33,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2345866.6666666665, ans=0.05 2023-11-23 10:35:43,467 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3200, loss[loss=0.05758, simple_loss=0.07648, pruned_loss=0.009149, audio_tagging_loss=0.01019, over 15588.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09308, pruned_loss=0.01429, audio_tagging_loss=0.009304, over 3045567.64 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:35:54,377 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351900 2023-11-23 10:36:14,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2346066.6666666665, ans=0.125 2023-11-23 10:36:29,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2346133.3333333335, ans=0.0 2023-11-23 10:36:31,824 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.179e+01 8.758e+01 9.576e+01 1.199e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-23 10:36:49,774 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3250, loss[loss=0.07181, simple_loss=0.09608, pruned_loss=0.0156, audio_tagging_loss=0.008167, over 15187.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09361, pruned_loss=0.01432, audio_tagging_loss=0.009238, over 3048738.67 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:36:58,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.10 vs. limit=15.0 2023-11-23 10:37:00,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 351950 2023-11-23 10:37:20,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2346400.0, ans=0.1 2023-11-23 10:37:25,440 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.65 vs. limit=15.0 2023-11-23 10:37:54,533 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3300, loss[loss=0.05649, simple_loss=0.06905, pruned_loss=0.0106, audio_tagging_loss=0.01137, over 15786.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.0926, pruned_loss=0.01405, audio_tagging_loss=0.009306, over 3052096.17 frames. ], batch size: 62, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:37:54,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2346600.0, ans=0.125 2023-11-23 10:38:03,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2346600.0, ans=0.07 2023-11-23 10:38:04,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352000 2023-11-23 10:38:13,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2346666.6666666665, ans=0.125 2023-11-23 10:38:15,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.33 vs. limit=15.0 2023-11-23 10:38:15,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2346666.6666666665, ans=0.125 2023-11-23 10:38:33,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2346733.3333333335, ans=0.035 2023-11-23 10:38:46,044 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.301e+01 8.918e+01 9.607e+01 1.152e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 10:38:57,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2346866.6666666665, ans=0.09899494936611666 2023-11-23 10:39:02,161 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3350, loss[loss=0.06406, simple_loss=0.07217, pruned_loss=0.01298, audio_tagging_loss=0.01499, over 15736.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09266, pruned_loss=0.01419, audio_tagging_loss=0.00925, over 3054013.92 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:39:08,547 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.68 vs. limit=12.0 2023-11-23 10:39:12,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352050 2023-11-23 10:39:13,552 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.36 vs. limit=22.5 2023-11-23 10:39:21,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2347000.0, ans=0.125 2023-11-23 10:39:25,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2347000.0, ans=0.0 2023-11-23 10:39:32,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2347066.6666666665, ans=0.125 2023-11-23 10:39:36,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2347066.6666666665, ans=0.125 2023-11-23 10:39:41,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2347133.3333333335, ans=0.0 2023-11-23 10:39:51,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2347133.3333333335, ans=0.2 2023-11-23 10:39:53,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2347200.0, ans=0.125 2023-11-23 10:39:54,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2347200.0, ans=0.125 2023-11-23 10:40:08,341 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3400, loss[loss=0.06656, simple_loss=0.1029, pruned_loss=0.008905, audio_tagging_loss=0.00621, over 15072.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09307, pruned_loss=0.01431, audio_tagging_loss=0.009166, over 3051961.12 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:40:08,821 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 10:40:13,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2347266.6666666665, ans=0.125 2023-11-23 10:40:19,073 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352100 2023-11-23 10:40:41,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2347400.0, ans=10.0 2023-11-23 10:40:44,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2347400.0, ans=0.125 2023-11-23 10:40:44,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2347400.0, ans=0.1 2023-11-23 10:40:45,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2347466.6666666665, ans=0.2 2023-11-23 10:40:46,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2347466.6666666665, ans=0.1 2023-11-23 10:40:48,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2347466.6666666665, ans=0.125 2023-11-23 10:40:48,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2347466.6666666665, ans=0.125 2023-11-23 10:40:52,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2347466.6666666665, ans=0.125 2023-11-23 10:40:55,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2347466.6666666665, ans=0.0 2023-11-23 10:40:56,360 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.131e+01 8.875e+01 9.563e+01 1.133e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-23 10:41:12,955 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3450, loss[loss=0.06724, simple_loss=0.09298, pruned_loss=0.012, audio_tagging_loss=0.008749, over 15698.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09286, pruned_loss=0.01419, audio_tagging_loss=0.009095, over 3044465.04 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:41:15,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.66 vs. limit=15.0 2023-11-23 10:41:19,471 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 10:41:23,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352150 2023-11-23 10:41:28,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2347666.6666666665, ans=0.125 2023-11-23 10:41:57,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2347800.0, ans=0.125 2023-11-23 10:42:16,594 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3500, loss[loss=0.09178, simple_loss=0.1321, pruned_loss=0.01872, audio_tagging_loss=0.007022, over 15234.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09297, pruned_loss=0.01413, audio_tagging_loss=0.009036, over 3044563.95 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:42:24,745 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.89 vs. limit=15.0 2023-11-23 10:42:25,718 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.68 vs. limit=6.0 2023-11-23 10:42:26,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352200 2023-11-23 10:42:36,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2348000.0, ans=0.125 2023-11-23 10:42:43,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2348066.6666666665, ans=0.125 2023-11-23 10:42:52,042 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:43:04,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.164e+01 8.763e+01 9.572e+01 1.144e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-23 10:43:12,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2348200.0, ans=0.2 2023-11-23 10:43:15,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2348200.0, ans=0.125 2023-11-23 10:43:20,958 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3550, loss[loss=0.06429, simple_loss=0.08362, pruned_loss=0.01249, audio_tagging_loss=0.009992, over 16094.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09296, pruned_loss=0.01409, audio_tagging_loss=0.009063, over 3039448.18 frames. ], batch size: 60, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:43:31,882 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352250 2023-11-23 10:43:46,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2348400.0, ans=0.125 2023-11-23 10:44:00,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2348466.6666666665, ans=0.0 2023-11-23 10:44:01,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.07 vs. limit=15.0 2023-11-23 10:44:01,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2348466.6666666665, ans=0.125 2023-11-23 10:44:02,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.14 vs. limit=15.0 2023-11-23 10:44:03,540 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.68 vs. limit=15.0 2023-11-23 10:44:16,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2348533.3333333335, ans=0.125 2023-11-23 10:44:25,960 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3600, loss[loss=0.06098, simple_loss=0.075, pruned_loss=0.01119, audio_tagging_loss=0.01229, over 14931.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09212, pruned_loss=0.01397, audio_tagging_loss=0.009064, over 3039189.63 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:44:26,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2348600.0, ans=0.0 2023-11-23 10:44:35,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352300 2023-11-23 10:44:40,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2348666.6666666665, ans=0.125 2023-11-23 10:44:44,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2348666.6666666665, ans=0.1 2023-11-23 10:44:45,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2348666.6666666665, ans=0.0 2023-11-23 10:44:48,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2348666.6666666665, ans=0.125 2023-11-23 10:44:51,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2348733.3333333335, ans=0.5 2023-11-23 10:45:13,762 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.915e+01 8.138e+01 8.784e+01 9.716e+01 1.349e+02, threshold=1.757e+02, percent-clipped=0.0 2023-11-23 10:45:21,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2348866.6666666665, ans=0.125 2023-11-23 10:45:22,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2348866.6666666665, ans=0.0 2023-11-23 10:45:25,799 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.68 vs. limit=22.5 2023-11-23 10:45:29,824 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3650, loss[loss=0.07275, simple_loss=0.1019, pruned_loss=0.01445, audio_tagging_loss=0.00732, over 15865.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09206, pruned_loss=0.01389, audio_tagging_loss=0.008982, over 3042879.01 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:45:39,978 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352350 2023-11-23 10:45:41,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2349000.0, ans=0.2 2023-11-23 10:45:45,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2349000.0, ans=0.0 2023-11-23 10:45:54,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.60 vs. limit=15.0 2023-11-23 10:46:00,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.71 vs. limit=15.0 2023-11-23 10:46:02,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2349066.6666666665, ans=0.2 2023-11-23 10:46:17,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2349133.3333333335, ans=0.0 2023-11-23 10:46:31,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2349200.0, ans=0.125 2023-11-23 10:46:34,692 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3700, loss[loss=0.07048, simple_loss=0.09707, pruned_loss=0.01351, audio_tagging_loss=0.008438, over 15691.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09159, pruned_loss=0.01384, audio_tagging_loss=0.009034, over 3055113.07 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:46:42,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.50 vs. limit=15.0 2023-11-23 10:46:43,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2349266.6666666665, ans=0.07 2023-11-23 10:46:46,530 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352400 2023-11-23 10:46:46,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2349266.6666666665, ans=0.125 2023-11-23 10:47:07,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2349400.0, ans=0.2 2023-11-23 10:47:15,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2349466.6666666665, ans=0.125 2023-11-23 10:47:23,652 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.475e+01 8.984e+01 9.756e+01 1.281e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 10:47:30,330 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.95 vs. limit=15.0 2023-11-23 10:47:31,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2349533.3333333335, ans=0.125 2023-11-23 10:47:42,633 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3750, loss[loss=0.07113, simple_loss=0.08992, pruned_loss=0.01416, audio_tagging_loss=0.012, over 14754.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09217, pruned_loss=0.01401, audio_tagging_loss=0.009101, over 3052236.52 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:47:53,017 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352450 2023-11-23 10:48:02,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2349666.6666666665, ans=0.0 2023-11-23 10:48:05,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2349666.6666666665, ans=0.0 2023-11-23 10:48:13,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2349733.3333333335, ans=0.2 2023-11-23 10:48:31,274 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:48:49,435 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3800, loss[loss=0.07418, simple_loss=0.1082, pruned_loss=0.01254, audio_tagging_loss=0.007523, over 15636.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09214, pruned_loss=0.01394, audio_tagging_loss=0.009138, over 3050476.19 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:48:59,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352500 2023-11-23 10:48:59,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2349933.3333333335, ans=0.025 2023-11-23 10:48:59,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2349933.3333333335, ans=0.025 2023-11-23 10:49:16,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2350066.6666666665, ans=0.0 2023-11-23 10:49:19,012 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.29 vs. limit=22.5 2023-11-23 10:49:34,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2350133.3333333335, ans=0.125 2023-11-23 10:49:37,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2350133.3333333335, ans=0.2 2023-11-23 10:49:38,116 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.465e+01 8.919e+01 9.665e+01 1.243e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 10:49:48,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2350200.0, ans=0.0 2023-11-23 10:49:54,999 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3850, loss[loss=0.07035, simple_loss=0.1036, pruned_loss=0.009528, audio_tagging_loss=0.009011, over 15324.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.0922, pruned_loss=0.01396, audio_tagging_loss=0.009094, over 3048008.53 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 10:50:01,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2350266.6666666665, ans=0.125 2023-11-23 10:50:02,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2350266.6666666665, ans=0.2 2023-11-23 10:50:03,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2350266.6666666665, ans=0.05 2023-11-23 10:50:06,581 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352550 2023-11-23 10:51:02,848 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3900, loss[loss=0.08574, simple_loss=0.1181, pruned_loss=0.01655, audio_tagging_loss=0.01014, over 14691.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09257, pruned_loss=0.01389, audio_tagging_loss=0.009135, over 3052513.08 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 8.0 2023-11-23 10:51:14,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352600 2023-11-23 10:51:34,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.44 vs. limit=15.0 2023-11-23 10:51:45,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2350800.0, ans=0.125 2023-11-23 10:51:55,079 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.310e+01 8.820e+01 9.647e+01 1.440e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-23 10:51:58,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2350866.6666666665, ans=0.1 2023-11-23 10:52:10,434 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 3950, loss[loss=0.07, simple_loss=0.09694, pruned_loss=0.01487, audio_tagging_loss=0.006664, over 14723.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09291, pruned_loss=0.01388, audio_tagging_loss=0.009141, over 3050436.44 frames. ], batch size: 53, lr: 2.27e-03, grad_scale: 8.0 2023-11-23 10:52:14,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=17.90 vs. limit=22.5 2023-11-23 10:52:18,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2350933.3333333335, ans=0.125 2023-11-23 10:52:20,606 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352650 2023-11-23 10:52:25,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2351000.0, ans=0.125 2023-11-23 10:53:06,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2351200.0, ans=0.0 2023-11-23 10:53:16,308 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4000, loss[loss=0.07707, simple_loss=0.1037, pruned_loss=0.01807, audio_tagging_loss=0.007135, over 14946.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09395, pruned_loss=0.0142, audio_tagging_loss=0.009203, over 3056714.37 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:53:23,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2351266.6666666665, ans=0.0 2023-11-23 10:53:27,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352700 2023-11-23 10:53:29,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2351333.3333333335, ans=0.125 2023-11-23 10:53:47,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2351400.0, ans=0.0 2023-11-23 10:53:58,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2351466.6666666665, ans=0.125 2023-11-23 10:54:02,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2351466.6666666665, ans=0.125 2023-11-23 10:54:02,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2351466.6666666665, ans=0.2 2023-11-23 10:54:08,346 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.731e+01 8.440e+01 9.076e+01 9.924e+01 2.102e+02, threshold=1.815e+02, percent-clipped=1.0 2023-11-23 10:54:23,978 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4050, loss[loss=0.07076, simple_loss=0.08447, pruned_loss=0.01644, audio_tagging_loss=0.01209, over 15848.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09348, pruned_loss=0.01415, audio_tagging_loss=0.009345, over 3054268.56 frames. ], batch size: 62, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:54:27,759 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:54:30,380 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.76 vs. limit=6.0 2023-11-23 10:54:31,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2351600.0, ans=0.0 2023-11-23 10:54:34,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352750 2023-11-23 10:54:36,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=2351666.6666666665, ans=0.5 2023-11-23 10:54:44,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2351666.6666666665, ans=0.0 2023-11-23 10:55:01,791 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.21 vs. limit=10.0 2023-11-23 10:55:30,784 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4100, loss[loss=0.07945, simple_loss=0.1106, pruned_loss=0.01522, audio_tagging_loss=0.008929, over 15568.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09368, pruned_loss=0.0142, audio_tagging_loss=0.009347, over 3052685.77 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:55:41,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352800 2023-11-23 10:55:43,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2352000.0, ans=0.125 2023-11-23 10:56:00,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2352066.6666666665, ans=0.2 2023-11-23 10:56:10,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2352133.3333333335, ans=0.125 2023-11-23 10:56:22,981 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.387e+01 8.886e+01 9.725e+01 1.263e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-23 10:56:37,369 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4150, loss[loss=0.08023, simple_loss=0.1049, pruned_loss=0.01876, audio_tagging_loss=0.009027, over 15740.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09428, pruned_loss=0.01433, audio_tagging_loss=0.009167, over 3061470.97 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:56:40,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2352266.6666666665, ans=0.1 2023-11-23 10:56:45,660 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.25 vs. limit=22.5 2023-11-23 10:56:47,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352850 2023-11-23 10:56:53,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2352333.3333333335, ans=0.1 2023-11-23 10:56:56,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2352333.3333333335, ans=0.0 2023-11-23 10:56:59,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2352333.3333333335, ans=0.1 2023-11-23 10:57:17,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0 2023-11-23 10:57:25,918 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 10:57:43,201 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4200, loss[loss=0.07193, simple_loss=0.09686, pruned_loss=0.01443, audio_tagging_loss=0.009071, over 15682.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09465, pruned_loss=0.0143, audio_tagging_loss=0.009071, over 3054020.91 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:57:43,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2352600.0, ans=0.1 2023-11-23 10:57:44,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2352600.0, ans=0.1 2023-11-23 10:57:49,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2352600.0, ans=0.125 2023-11-23 10:57:53,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352900 2023-11-23 10:58:05,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2352666.6666666665, ans=0.125 2023-11-23 10:58:06,967 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0 2023-11-23 10:58:09,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2352733.3333333335, ans=0.125 2023-11-23 10:58:27,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2352800.0, ans=0.04949747468305833 2023-11-23 10:58:29,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=2352800.0, ans=0.1 2023-11-23 10:58:32,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2352800.0, ans=0.125 2023-11-23 10:58:34,392 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.262e+01 8.280e+01 9.153e+01 9.878e+01 1.174e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-23 10:58:48,833 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4250, loss[loss=0.05817, simple_loss=0.07447, pruned_loss=0.01262, audio_tagging_loss=0.008317, over 14707.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09373, pruned_loss=0.01414, audio_tagging_loss=0.009037, over 3042223.24 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 10:58:58,822 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 352950 2023-11-23 10:59:49,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2353200.0, ans=0.125 2023-11-23 10:59:54,209 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4300, loss[loss=0.04388, simple_loss=0.05735, pruned_loss=0.00702, audio_tagging_loss=0.008184, over 15910.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09453, pruned_loss=0.01415, audio_tagging_loss=0.008967, over 3050478.06 frames. ], batch size: 64, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:00:04,400 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353000 2023-11-23 11:00:08,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2353333.3333333335, ans=0.0 2023-11-23 11:00:09,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2353333.3333333335, ans=0.125 2023-11-23 11:00:17,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2353333.3333333335, ans=22.5 2023-11-23 11:00:18,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2353333.3333333335, ans=0.125 2023-11-23 11:00:23,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2353400.0, ans=0.07 2023-11-23 11:00:33,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2353466.6666666665, ans=0.125 2023-11-23 11:00:45,970 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.099e+01 8.412e+01 9.137e+01 9.657e+01 1.208e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-23 11:01:01,195 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4350, loss[loss=0.05233, simple_loss=0.06662, pruned_loss=0.009663, audio_tagging_loss=0.009353, over 15519.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09449, pruned_loss=0.01413, audio_tagging_loss=0.008902, over 3046108.87 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:01:11,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353050 2023-11-23 11:01:15,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2353666.6666666665, ans=0.125 2023-11-23 11:01:19,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2353666.6666666665, ans=0.1 2023-11-23 11:01:54,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2353866.6666666665, ans=0.5 2023-11-23 11:02:03,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2353866.6666666665, ans=0.125 2023-11-23 11:02:07,057 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4400, loss[loss=0.0903, simple_loss=0.1177, pruned_loss=0.02428, audio_tagging_loss=0.00715, over 15795.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09363, pruned_loss=0.01405, audio_tagging_loss=0.009004, over 3046027.96 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:02:10,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2353933.3333333335, ans=0.125 2023-11-23 11:02:13,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2353933.3333333335, ans=0.0 2023-11-23 11:02:17,395 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353100 2023-11-23 11:02:31,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2354066.6666666665, ans=0.0 2023-11-23 11:02:58,602 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.258e+01 8.734e+01 9.523e+01 1.170e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-23 11:03:06,936 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.39 vs. limit=15.0 2023-11-23 11:03:12,647 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4450, loss[loss=0.08843, simple_loss=0.1205, pruned_loss=0.02045, audio_tagging_loss=0.007734, over 14671.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09361, pruned_loss=0.01419, audio_tagging_loss=0.009049, over 3044216.56 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:03:23,246 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353150 2023-11-23 11:04:02,848 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 11:04:03,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.13 vs. limit=6.0 2023-11-23 11:04:18,946 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4500, loss[loss=0.06735, simple_loss=0.09412, pruned_loss=0.01193, audio_tagging_loss=0.008356, over 15444.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09386, pruned_loss=0.01425, audio_tagging_loss=0.009055, over 3047855.64 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:04:21,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2354600.0, ans=0.125 2023-11-23 11:04:29,736 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353200 2023-11-23 11:04:31,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2354666.6666666665, ans=0.125 2023-11-23 11:05:10,689 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.361e+01 8.894e+01 9.678e+01 1.285e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 11:05:17,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2354866.6666666665, ans=0.125 2023-11-23 11:05:20,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2354866.6666666665, ans=0.2 2023-11-23 11:05:25,310 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4550, loss[loss=0.07384, simple_loss=0.09947, pruned_loss=0.01225, audio_tagging_loss=0.01185, over 14326.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09387, pruned_loss=0.0144, audio_tagging_loss=0.009084, over 3044113.53 frames. ], batch size: 53, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:05:35,610 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353250 2023-11-23 11:05:37,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2355000.0, ans=0.125 2023-11-23 11:06:07,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2355133.3333333335, ans=0.125 2023-11-23 11:06:16,972 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 11:06:30,710 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4600, loss[loss=0.06405, simple_loss=0.08937, pruned_loss=0.01291, audio_tagging_loss=0.006456, over 14766.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09327, pruned_loss=0.0144, audio_tagging_loss=0.009128, over 3035388.87 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:06:40,941 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353300 2023-11-23 11:07:06,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2355400.0, ans=0.09899494936611666 2023-11-23 11:07:17,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2355466.6666666665, ans=0.5 2023-11-23 11:07:22,709 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.374e+01 9.113e+01 9.738e+01 1.636e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-23 11:07:23,614 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.61 vs. limit=10.0 2023-11-23 11:07:35,307 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4650, loss[loss=0.04683, simple_loss=0.06054, pruned_loss=0.006812, audio_tagging_loss=0.009749, over 14356.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09278, pruned_loss=0.01459, audio_tagging_loss=0.009242, over 3032675.46 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:07:47,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353350 2023-11-23 11:07:55,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2355666.6666666665, ans=0.125 2023-11-23 11:08:23,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2355800.0, ans=0.0 2023-11-23 11:08:30,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2355866.6666666665, ans=0.05 2023-11-23 11:08:42,515 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4700, loss[loss=0.06264, simple_loss=0.08124, pruned_loss=0.01235, audio_tagging_loss=0.009662, over 16881.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09253, pruned_loss=0.01459, audio_tagging_loss=0.009323, over 3038148.04 frames. ], batch size: 63, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:08:52,678 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353400 2023-11-23 11:09:04,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2356000.0, ans=0.07 2023-11-23 11:09:04,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=2356000.0, ans=0.02 2023-11-23 11:09:17,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2356066.6666666665, ans=0.125 2023-11-23 11:09:34,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2356200.0, ans=0.0 2023-11-23 11:09:35,051 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.186e+01 8.702e+01 9.591e+01 1.216e+02, threshold=1.740e+02, percent-clipped=0.0 2023-11-23 11:09:42,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.26 vs. limit=5.0 2023-11-23 11:09:47,651 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4750, loss[loss=0.07102, simple_loss=0.09678, pruned_loss=0.01471, audio_tagging_loss=0.007919, over 15110.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09227, pruned_loss=0.0145, audio_tagging_loss=0.009384, over 3040683.03 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:09:54,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2356266.6666666665, ans=0.125 2023-11-23 11:09:57,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353450 2023-11-23 11:10:26,766 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.58 vs. limit=10.0 2023-11-23 11:10:36,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2356466.6666666665, ans=0.2 2023-11-23 11:10:52,110 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4800, loss[loss=0.06681, simple_loss=0.08435, pruned_loss=0.0148, audio_tagging_loss=0.009844, over 15096.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09278, pruned_loss=0.0144, audio_tagging_loss=0.009395, over 3052487.01 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:11:04,016 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353500 2023-11-23 11:11:12,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2356666.6666666665, ans=0.0 2023-11-23 11:11:18,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2356733.3333333335, ans=0.125 2023-11-23 11:11:44,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2356866.6666666665, ans=0.125 2023-11-23 11:11:44,677 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.16 vs. limit=15.0 2023-11-23 11:11:44,951 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.167e+01 8.824e+01 9.550e+01 1.189e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-23 11:11:52,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2356866.6666666665, ans=0.0 2023-11-23 11:11:54,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.27 vs. limit=22.5 2023-11-23 11:11:55,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2356866.6666666665, ans=0.125 2023-11-23 11:11:59,424 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4850, loss[loss=0.06527, simple_loss=0.08875, pruned_loss=0.01111, audio_tagging_loss=0.009789, over 14813.00 frames. ], tot_loss[loss=0.07062, simple_loss=0.0933, pruned_loss=0.01445, audio_tagging_loss=0.009525, over 3053388.00 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:12:10,058 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353550 2023-11-23 11:12:15,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2357000.0, ans=0.0 2023-11-23 11:12:20,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2357000.0, ans=0.1 2023-11-23 11:12:23,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2357066.6666666665, ans=0.1 2023-11-23 11:12:51,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2357200.0, ans=0.125 2023-11-23 11:12:56,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2357200.0, ans=0.0 2023-11-23 11:12:57,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2357200.0, ans=0.125 2023-11-23 11:13:04,985 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4900, loss[loss=0.07261, simple_loss=0.09442, pruned_loss=0.01541, audio_tagging_loss=0.009991, over 14698.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09268, pruned_loss=0.01425, audio_tagging_loss=0.009447, over 3046662.99 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:13:05,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2357266.6666666665, ans=0.2 2023-11-23 11:13:15,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353600 2023-11-23 11:13:26,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2357333.3333333335, ans=0.1 2023-11-23 11:13:36,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2357400.0, ans=0.2 2023-11-23 11:13:57,846 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.207e+01 8.666e+01 9.499e+01 1.245e+02, threshold=1.733e+02, percent-clipped=0.0 2023-11-23 11:14:08,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2357533.3333333335, ans=0.0 2023-11-23 11:14:10,319 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 4950, loss[loss=0.08608, simple_loss=0.1195, pruned_loss=0.01933, audio_tagging_loss=0.007015, over 15557.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09384, pruned_loss=0.01439, audio_tagging_loss=0.009216, over 3049210.38 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:14:12,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2357600.0, ans=0.0 2023-11-23 11:14:16,425 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.09 vs. limit=15.0 2023-11-23 11:14:21,895 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353650 2023-11-23 11:14:23,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2357666.6666666665, ans=0.0 2023-11-23 11:14:23,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2357666.6666666665, ans=0.0 2023-11-23 11:14:28,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2357666.6666666665, ans=0.0 2023-11-23 11:15:12,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2357866.6666666665, ans=0.0 2023-11-23 11:15:17,621 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5000, loss[loss=0.06604, simple_loss=0.0872, pruned_loss=0.0119, audio_tagging_loss=0.01053, over 15715.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09307, pruned_loss=0.01425, audio_tagging_loss=0.009111, over 3052310.95 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:15:29,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353700 2023-11-23 11:15:43,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2358066.6666666665, ans=0.2 2023-11-23 11:15:50,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2358066.6666666665, ans=0.0 2023-11-23 11:15:57,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2358133.3333333335, ans=0.125 2023-11-23 11:16:00,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2358133.3333333335, ans=0.125 2023-11-23 11:16:10,298 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.172e+01 8.851e+01 9.559e+01 1.178e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-23 11:16:10,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2358200.0, ans=0.0 2023-11-23 11:16:24,075 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5050, loss[loss=0.06228, simple_loss=0.08188, pruned_loss=0.01223, audio_tagging_loss=0.009108, over 15128.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09217, pruned_loss=0.01394, audio_tagging_loss=0.009005, over 3050023.18 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:16:29,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2358266.6666666665, ans=0.0 2023-11-23 11:16:30,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.64 vs. limit=12.0 2023-11-23 11:16:34,161 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353750 2023-11-23 11:16:38,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2358333.3333333335, ans=0.125 2023-11-23 11:16:39,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.73 vs. limit=15.0 2023-11-23 11:16:42,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2358333.3333333335, ans=0.125 2023-11-23 11:16:44,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2358333.3333333335, ans=0.125 2023-11-23 11:16:50,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2358400.0, ans=0.1 2023-11-23 11:17:14,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2358466.6666666665, ans=0.0 2023-11-23 11:17:17,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.44 vs. limit=22.5 2023-11-23 11:17:29,515 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5100, loss[loss=0.0715, simple_loss=0.097, pruned_loss=0.01316, audio_tagging_loss=0.009846, over 16024.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.0923, pruned_loss=0.01401, audio_tagging_loss=0.00905, over 3056026.76 frames. ], batch size: 61, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:17:40,362 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353800 2023-11-23 11:17:42,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.03 vs. limit=22.5 2023-11-23 11:17:43,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2358666.6666666665, ans=0.125 2023-11-23 11:17:46,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2358666.6666666665, ans=0.0 2023-11-23 11:17:50,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.49 vs. limit=12.0 2023-11-23 11:17:50,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2358666.6666666665, ans=0.0 2023-11-23 11:17:51,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2358666.6666666665, ans=0.125 2023-11-23 11:18:08,982 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.79 vs. limit=15.0 2023-11-23 11:18:16,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2358800.0, ans=0.125 2023-11-23 11:18:23,225 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.214e+01 8.798e+01 9.658e+01 1.127e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-23 11:18:30,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2358866.6666666665, ans=0.2 2023-11-23 11:18:31,615 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 11:18:35,651 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5150, loss[loss=0.08092, simple_loss=0.1077, pruned_loss=0.02079, audio_tagging_loss=0.006281, over 15089.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09233, pruned_loss=0.01403, audio_tagging_loss=0.008965, over 3059027.47 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:18:46,395 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353850 2023-11-23 11:18:46,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2358933.3333333335, ans=0.125 2023-11-23 11:18:49,397 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.93 vs. limit=15.0 2023-11-23 11:19:07,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2359066.6666666665, ans=0.05 2023-11-23 11:19:10,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2359066.6666666665, ans=0.025 2023-11-23 11:19:23,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2359133.3333333335, ans=0.2 2023-11-23 11:19:25,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2359133.3333333335, ans=0.0 2023-11-23 11:19:34,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2359200.0, ans=0.1 2023-11-23 11:19:36,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2359200.0, ans=0.125 2023-11-23 11:19:36,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2359200.0, ans=0.125 2023-11-23 11:19:42,062 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5200, loss[loss=0.06304, simple_loss=0.07999, pruned_loss=0.01049, audio_tagging_loss=0.01255, over 15647.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09311, pruned_loss=0.01419, audio_tagging_loss=0.008895, over 3053573.20 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:19:52,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353900 2023-11-23 11:19:52,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2359266.6666666665, ans=0.125 2023-11-23 11:19:57,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2359333.3333333335, ans=0.1 2023-11-23 11:19:58,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2359333.3333333335, ans=0.125 2023-11-23 11:20:02,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2359333.3333333335, ans=0.2 2023-11-23 11:20:11,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.49 vs. limit=22.5 2023-11-23 11:20:28,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2359466.6666666665, ans=0.125 2023-11-23 11:20:36,393 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.445e+01 9.037e+01 9.922e+01 1.226e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-23 11:20:47,884 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5250, loss[loss=0.07133, simple_loss=0.1034, pruned_loss=0.01304, audio_tagging_loss=0.006588, over 15407.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09318, pruned_loss=0.0142, audio_tagging_loss=0.00891, over 3059429.72 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:20:52,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.73 vs. limit=15.0 2023-11-23 11:20:56,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2359600.0, ans=0.125 2023-11-23 11:20:57,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 353950 2023-11-23 11:21:12,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2359666.6666666665, ans=0.09899494936611666 2023-11-23 11:21:13,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2359733.3333333335, ans=0.0 2023-11-23 11:21:16,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=9.86 vs. limit=22.5 2023-11-23 11:21:54,233 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5300, loss[loss=0.07824, simple_loss=0.102, pruned_loss=0.01897, audio_tagging_loss=0.008249, over 15403.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09292, pruned_loss=0.01418, audio_tagging_loss=0.008874, over 3051243.57 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:22:04,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354000 2023-11-23 11:22:22,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2360066.6666666665, ans=0.0 2023-11-23 11:22:23,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2360066.6666666665, ans=0.0 2023-11-23 11:22:33,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2360133.3333333335, ans=0.0 2023-11-23 11:22:49,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.002e+01 8.452e+01 9.049e+01 9.694e+01 1.252e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 11:22:52,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2360200.0, ans=0.125 2023-11-23 11:23:00,048 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5350, loss[loss=0.05333, simple_loss=0.07073, pruned_loss=0.009668, audio_tagging_loss=0.008296, over 14980.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09278, pruned_loss=0.01411, audio_tagging_loss=0.008962, over 3055229.17 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:23:10,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354050 2023-11-23 11:23:19,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2360333.3333333335, ans=0.0 2023-11-23 11:23:26,152 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 11:23:37,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2360400.0, ans=0.0 2023-11-23 11:23:40,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2360466.6666666665, ans=0.125 2023-11-23 11:24:06,180 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.48 vs. limit=15.0 2023-11-23 11:24:06,735 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5400, loss[loss=0.09118, simple_loss=0.1307, pruned_loss=0.01738, audio_tagging_loss=0.008448, over 15527.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09285, pruned_loss=0.01416, audio_tagging_loss=0.009063, over 3056468.40 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:24:16,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354100 2023-11-23 11:24:17,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2360600.0, ans=0.0 2023-11-23 11:24:28,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.50 vs. limit=15.0 2023-11-23 11:24:33,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2360733.3333333335, ans=0.2 2023-11-23 11:24:48,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2360800.0, ans=0.95 2023-11-23 11:24:56,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.87 vs. limit=15.0 2023-11-23 11:25:01,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2360866.6666666665, ans=0.125 2023-11-23 11:25:02,418 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.248e+01 8.837e+01 9.737e+01 1.214e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 11:25:12,621 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5450, loss[loss=0.06594, simple_loss=0.08852, pruned_loss=0.01291, audio_tagging_loss=0.008771, over 15986.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09336, pruned_loss=0.01408, audio_tagging_loss=0.008981, over 3055418.91 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:25:14,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2360933.3333333335, ans=0.125 2023-11-23 11:25:24,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354150 2023-11-23 11:25:52,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2361133.3333333335, ans=10.0 2023-11-23 11:26:19,501 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5500, loss[loss=0.06767, simple_loss=0.09159, pruned_loss=0.0114, audio_tagging_loss=0.01047, over 15892.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09359, pruned_loss=0.01415, audio_tagging_loss=0.008978, over 3053997.87 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:26:30,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354200 2023-11-23 11:26:34,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2361333.3333333335, ans=0.0 2023-11-23 11:26:49,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2361400.0, ans=0.035 2023-11-23 11:27:08,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2361466.6666666665, ans=0.2 2023-11-23 11:27:16,272 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.480e+01 8.474e+01 9.083e+01 9.895e+01 1.258e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-23 11:27:26,423 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5550, loss[loss=0.08495, simple_loss=0.1151, pruned_loss=0.02167, audio_tagging_loss=0.005732, over 15321.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09351, pruned_loss=0.01431, audio_tagging_loss=0.00907, over 3057060.18 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:27:26,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2361600.0, ans=0.125 2023-11-23 11:27:29,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2361600.0, ans=0.1 2023-11-23 11:27:35,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2361600.0, ans=0.125 2023-11-23 11:27:37,406 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354250 2023-11-23 11:27:40,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2361666.6666666665, ans=0.125 2023-11-23 11:28:31,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2361933.3333333335, ans=0.2 2023-11-23 11:28:32,570 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5600, loss[loss=0.06213, simple_loss=0.0899, pruned_loss=0.008845, audio_tagging_loss=0.008337, over 14685.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09368, pruned_loss=0.01419, audio_tagging_loss=0.009233, over 3057108.76 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 32.0 2023-11-23 11:28:43,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354300 2023-11-23 11:28:47,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2362000.0, ans=0.0 2023-11-23 11:28:51,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2362000.0, ans=0.125 2023-11-23 11:28:54,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2362000.0, ans=0.125 2023-11-23 11:29:03,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2362066.6666666665, ans=0.125 2023-11-23 11:29:10,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2362133.3333333335, ans=0.0 2023-11-23 11:29:19,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2362133.3333333335, ans=0.125 2023-11-23 11:29:21,279 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 11:29:22,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2362133.3333333335, ans=0.0 2023-11-23 11:29:25,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2362200.0, ans=0.2 2023-11-23 11:29:27,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2362200.0, ans=0.1 2023-11-23 11:29:29,332 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.942e+01 8.297e+01 9.078e+01 9.642e+01 1.395e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-23 11:29:38,664 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5650, loss[loss=0.08026, simple_loss=0.1115, pruned_loss=0.01425, audio_tagging_loss=0.01028, over 15024.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.093, pruned_loss=0.01409, audio_tagging_loss=0.009357, over 3054842.06 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:29:39,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2362266.6666666665, ans=0.0 2023-11-23 11:29:44,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2362266.6666666665, ans=0.125 2023-11-23 11:29:49,521 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354350 2023-11-23 11:30:18,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.78 vs. limit=15.0 2023-11-23 11:30:43,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2362600.0, ans=0.0 2023-11-23 11:30:44,124 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5700, loss[loss=0.05574, simple_loss=0.07226, pruned_loss=0.008995, audio_tagging_loss=0.01061, over 16301.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09339, pruned_loss=0.01419, audio_tagging_loss=0.009369, over 3052640.84 frames. ], batch size: 64, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:30:50,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2362600.0, ans=0.0 2023-11-23 11:30:54,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354400 2023-11-23 11:31:07,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2362666.6666666665, ans=0.125 2023-11-23 11:31:09,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2362733.3333333335, ans=0.07 2023-11-23 11:31:12,254 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.75 vs. limit=22.5 2023-11-23 11:31:14,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2362733.3333333335, ans=0.0 2023-11-23 11:31:19,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2362733.3333333335, ans=0.0 2023-11-23 11:31:33,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2362800.0, ans=0.5 2023-11-23 11:31:40,369 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.224e+01 8.276e+01 8.914e+01 9.745e+01 1.403e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 11:31:49,769 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5750, loss[loss=0.05338, simple_loss=0.05802, pruned_loss=0.01311, audio_tagging_loss=0.01126, over 13984.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09279, pruned_loss=0.01405, audio_tagging_loss=0.009255, over 3053466.93 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:31:51,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2362933.3333333335, ans=0.0 2023-11-23 11:32:01,142 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354450 2023-11-23 11:32:09,299 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.64 vs. limit=15.0 2023-11-23 11:32:27,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2363066.6666666665, ans=0.125 2023-11-23 11:32:34,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2363133.3333333335, ans=0.0 2023-11-23 11:32:37,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2363133.3333333335, ans=0.125 2023-11-23 11:32:41,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2363200.0, ans=10.0 2023-11-23 11:32:56,522 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5800, loss[loss=0.04977, simple_loss=0.06589, pruned_loss=0.009113, audio_tagging_loss=0.007717, over 14427.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09251, pruned_loss=0.01391, audio_tagging_loss=0.009148, over 3055221.09 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:32:57,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.45 vs. limit=15.0 2023-11-23 11:33:07,354 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354500 2023-11-23 11:33:11,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.94 vs. limit=10.0 2023-11-23 11:33:29,351 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.67 vs. limit=15.0 2023-11-23 11:33:44,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2363466.6666666665, ans=0.125 2023-11-23 11:33:44,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2363466.6666666665, ans=15.0 2023-11-23 11:33:53,523 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.821e+01 8.322e+01 8.914e+01 9.568e+01 1.180e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 11:34:02,441 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5850, loss[loss=0.08048, simple_loss=0.1168, pruned_loss=0.01536, audio_tagging_loss=0.006698, over 15532.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09184, pruned_loss=0.01381, audio_tagging_loss=0.009163, over 3060194.50 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:34:12,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354550 2023-11-23 11:34:13,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2363666.6666666665, ans=0.1 2023-11-23 11:34:16,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2363666.6666666665, ans=0.125 2023-11-23 11:34:18,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2363666.6666666665, ans=0.2 2023-11-23 11:34:19,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2363666.6666666665, ans=0.1 2023-11-23 11:34:20,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2363666.6666666665, ans=0.1 2023-11-23 11:34:34,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.61 vs. limit=15.0 2023-11-23 11:34:39,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2363800.0, ans=0.0 2023-11-23 11:35:05,728 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.62 vs. limit=15.0 2023-11-23 11:35:06,291 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5900, loss[loss=0.08579, simple_loss=0.1138, pruned_loss=0.02046, audio_tagging_loss=0.008403, over 14765.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09181, pruned_loss=0.01375, audio_tagging_loss=0.009177, over 3056717.68 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0 2023-11-23 11:35:17,687 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354600 2023-11-23 11:35:45,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2364133.3333333335, ans=0.04949747468305833 2023-11-23 11:36:02,557 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.354e+01 8.917e+01 9.513e+01 1.247e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 11:36:12,772 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 5950, loss[loss=0.06104, simple_loss=0.07947, pruned_loss=0.01103, audio_tagging_loss=0.01028, over 15022.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09186, pruned_loss=0.01375, audio_tagging_loss=0.009119, over 3055624.50 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 11:36:23,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354650 2023-11-23 11:36:32,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2364333.3333333335, ans=0.07 2023-11-23 11:36:43,745 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.71 vs. limit=15.0 2023-11-23 11:37:08,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2364533.3333333335, ans=0.0 2023-11-23 11:37:15,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2364533.3333333335, ans=0.0 2023-11-23 11:37:17,287 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6000, loss[loss=0.07464, simple_loss=0.09704, pruned_loss=0.01901, audio_tagging_loss=0.007116, over 15812.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09197, pruned_loss=0.01368, audio_tagging_loss=0.009089, over 3042583.99 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:37:17,288 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 11:37:57,936 INFO [train_asr.py:1253] (3/4) Epoch 30, validation: loss=0.05791, simple_loss=0.05108, pruned_loss=0.005053, audio_tagging_loss=0.02732, over 4681554.00 frames. 2023-11-23 11:37:57,937 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 11:38:00,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2364600.0, ans=0.0 2023-11-23 11:38:02,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.62 vs. limit=22.5 2023-11-23 11:38:08,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354700 2023-11-23 11:38:26,515 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.01 vs. limit=15.0 2023-11-23 11:38:45,079 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 11:38:48,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.62 vs. limit=15.0 2023-11-23 11:38:52,917 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.297e+01 9.002e+01 9.719e+01 1.265e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-23 11:39:02,781 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6050, loss[loss=0.07635, simple_loss=0.09293, pruned_loss=0.01907, audio_tagging_loss=0.01081, over 15291.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09238, pruned_loss=0.01384, audio_tagging_loss=0.00907, over 3045443.46 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:39:05,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2364933.3333333335, ans=0.125 2023-11-23 11:39:07,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2364933.3333333335, ans=0.07 2023-11-23 11:39:13,273 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354750 2023-11-23 11:39:49,697 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.24 vs. limit=15.0 2023-11-23 11:39:50,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2365133.3333333335, ans=0.125 2023-11-23 11:40:07,196 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6100, loss[loss=0.06108, simple_loss=0.073, pruned_loss=0.01415, audio_tagging_loss=0.01043, over 15594.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09263, pruned_loss=0.01391, audio_tagging_loss=0.00899, over 3049293.67 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:40:08,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2365266.6666666665, ans=0.125 2023-11-23 11:40:17,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354800 2023-11-23 11:40:19,148 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.08 vs. limit=12.0 2023-11-23 11:40:39,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2365400.0, ans=0.125 2023-11-23 11:40:39,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2365400.0, ans=0.125 2023-11-23 11:40:42,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2365400.0, ans=0.05 2023-11-23 11:40:45,582 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.30 vs. limit=15.0 2023-11-23 11:40:58,411 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.72 vs. limit=15.0 2023-11-23 11:41:02,775 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.175e+01 8.198e+01 8.863e+01 9.563e+01 1.175e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-23 11:41:11,366 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6150, loss[loss=0.07012, simple_loss=0.09832, pruned_loss=0.01294, audio_tagging_loss=0.00802, over 16960.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09228, pruned_loss=0.01382, audio_tagging_loss=0.008989, over 3051108.35 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:41:21,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354850 2023-11-23 11:42:03,361 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.72 vs. limit=15.0 2023-11-23 11:42:05,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2365866.6666666665, ans=0.125 2023-11-23 11:42:16,176 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6200, loss[loss=0.05348, simple_loss=0.07223, pruned_loss=0.008759, audio_tagging_loss=0.008606, over 14361.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09121, pruned_loss=0.01377, audio_tagging_loss=0.00908, over 3052008.83 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:42:17,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2365933.3333333335, ans=0.0 2023-11-23 11:42:20,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2365933.3333333335, ans=0.2 2023-11-23 11:42:26,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2365933.3333333335, ans=0.1 2023-11-23 11:42:27,294 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354900 2023-11-23 11:42:29,043 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.85 vs. limit=15.0 2023-11-23 11:42:29,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.40 vs. limit=22.5 2023-11-23 11:42:38,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2366000.0, ans=0.0 2023-11-23 11:42:59,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2366133.3333333335, ans=0.0 2023-11-23 11:43:13,568 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.682e+01 8.235e+01 8.845e+01 9.606e+01 1.266e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 11:43:14,241 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.26 vs. limit=10.0 2023-11-23 11:43:21,078 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6250, loss[loss=0.07493, simple_loss=0.09757, pruned_loss=0.01678, audio_tagging_loss=0.009369, over 15172.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09163, pruned_loss=0.01389, audio_tagging_loss=0.009107, over 3053634.98 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 11:43:23,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2366266.6666666665, ans=0.125 2023-11-23 11:43:31,027 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 354950 2023-11-23 11:43:32,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2366333.3333333335, ans=0.125 2023-11-23 11:43:42,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2366333.3333333335, ans=0.0 2023-11-23 11:43:56,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2366400.0, ans=15.0 2023-11-23 11:44:11,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2366533.3333333335, ans=0.125 2023-11-23 11:44:21,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2366533.3333333335, ans=0.0 2023-11-23 11:44:24,653 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6300, loss[loss=0.07892, simple_loss=0.1078, pruned_loss=0.01779, audio_tagging_loss=0.007208, over 15251.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09286, pruned_loss=0.01408, audio_tagging_loss=0.009091, over 3057857.96 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 11:44:34,952 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355000 2023-11-23 11:45:01,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2366733.3333333335, ans=0.0 2023-11-23 11:45:20,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2366866.6666666665, ans=0.0 2023-11-23 11:45:20,942 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.278e+01 8.805e+01 9.594e+01 1.253e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-23 11:45:25,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2366866.6666666665, ans=0.125 2023-11-23 11:45:25,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2366866.6666666665, ans=0.0 2023-11-23 11:45:28,969 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6350, loss[loss=0.07471, simple_loss=0.09786, pruned_loss=0.01758, audio_tagging_loss=0.0082, over 14658.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09341, pruned_loss=0.01417, audio_tagging_loss=0.009122, over 3048659.64 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 11:45:38,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2366933.3333333335, ans=0.1 2023-11-23 11:45:40,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355050 2023-11-23 11:45:47,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2367000.0, ans=0.125 2023-11-23 11:46:00,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2367066.6666666665, ans=0.07 2023-11-23 11:46:00,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2367066.6666666665, ans=0.0 2023-11-23 11:46:13,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2367133.3333333335, ans=0.0 2023-11-23 11:46:19,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2367200.0, ans=0.2 2023-11-23 11:46:34,446 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6400, loss[loss=0.0786, simple_loss=0.1126, pruned_loss=0.0161, audio_tagging_loss=0.006207, over 14940.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09352, pruned_loss=0.01415, audio_tagging_loss=0.00916, over 3049443.88 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:46:37,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2367266.6666666665, ans=0.125 2023-11-23 11:46:44,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355100 2023-11-23 11:46:55,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2367333.3333333335, ans=0.125 2023-11-23 11:47:10,565 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.67 vs. limit=15.0 2023-11-23 11:47:30,743 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.031e+01 8.724e+01 9.486e+01 1.143e+02, threshold=1.745e+02, percent-clipped=0.0 2023-11-23 11:47:31,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2367533.3333333335, ans=0.125 2023-11-23 11:47:38,281 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6450, loss[loss=0.06175, simple_loss=0.08199, pruned_loss=0.01128, audio_tagging_loss=0.00948, over 15126.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09298, pruned_loss=0.01405, audio_tagging_loss=0.009358, over 3051340.79 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:47:44,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2367600.0, ans=0.125 2023-11-23 11:47:48,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355150 2023-11-23 11:47:53,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2367666.6666666665, ans=0.125 2023-11-23 11:48:09,171 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.17 vs. limit=15.0 2023-11-23 11:48:18,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2367800.0, ans=0.2 2023-11-23 11:48:31,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2367866.6666666665, ans=0.0 2023-11-23 11:48:43,083 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6500, loss[loss=0.07165, simple_loss=0.09376, pruned_loss=0.01575, audio_tagging_loss=0.009015, over 14309.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09202, pruned_loss=0.01396, audio_tagging_loss=0.009407, over 3050598.09 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:48:51,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.75 vs. limit=15.0 2023-11-23 11:48:53,521 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355200 2023-11-23 11:49:40,451 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.456e+01 9.159e+01 9.950e+01 1.283e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-23 11:49:48,682 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6550, loss[loss=0.07205, simple_loss=0.1005, pruned_loss=0.01532, audio_tagging_loss=0.006502, over 15726.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09286, pruned_loss=0.01412, audio_tagging_loss=0.009248, over 3049306.10 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:49:51,735 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.78 vs. limit=10.0 2023-11-23 11:49:55,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2368266.6666666665, ans=0.125 2023-11-23 11:49:59,217 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355250 2023-11-23 11:50:08,420 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.66 vs. limit=22.5 2023-11-23 11:50:45,990 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.88 vs. limit=12.0 2023-11-23 11:50:52,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2368600.0, ans=0.125 2023-11-23 11:50:52,348 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 11:50:53,276 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6600, loss[loss=0.05036, simple_loss=0.06569, pruned_loss=0.01013, audio_tagging_loss=0.007387, over 13975.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09186, pruned_loss=0.0139, audio_tagging_loss=0.009142, over 3046883.05 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:50:58,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.92 vs. limit=10.0 2023-11-23 11:51:03,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355300 2023-11-23 11:51:22,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.82 vs. limit=12.0 2023-11-23 11:51:22,591 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.66 vs. limit=22.5 2023-11-23 11:51:45,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2368866.6666666665, ans=0.0 2023-11-23 11:51:48,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2368866.6666666665, ans=0.0 2023-11-23 11:51:50,583 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.216e+01 8.892e+01 9.603e+01 1.376e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-23 11:51:57,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2368933.3333333335, ans=0.0 2023-11-23 11:51:58,398 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6650, loss[loss=0.06926, simple_loss=0.08933, pruned_loss=0.01675, audio_tagging_loss=0.00785, over 15447.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09178, pruned_loss=0.01396, audio_tagging_loss=0.00909, over 3042386.06 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:52:08,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355350 2023-11-23 11:52:18,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2369000.0, ans=0.04949747468305833 2023-11-23 11:52:23,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2369066.6666666665, ans=0.2 2023-11-23 11:52:34,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2369066.6666666665, ans=0.0 2023-11-23 11:52:54,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2369200.0, ans=0.2 2023-11-23 11:53:03,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2369266.6666666665, ans=0.125 2023-11-23 11:53:03,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2369266.6666666665, ans=0.125 2023-11-23 11:53:03,842 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6700, loss[loss=0.05932, simple_loss=0.07376, pruned_loss=0.01255, audio_tagging_loss=0.009886, over 14526.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09218, pruned_loss=0.01413, audio_tagging_loss=0.008907, over 3038416.90 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:53:11,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2369266.6666666665, ans=0.0 2023-11-23 11:53:13,930 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355400 2023-11-23 11:53:35,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.12 vs. limit=15.0 2023-11-23 11:53:36,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2369400.0, ans=0.05 2023-11-23 11:53:47,384 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.65 vs. limit=15.0 2023-11-23 11:53:48,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2369466.6666666665, ans=0.1 2023-11-23 11:53:55,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2369533.3333333335, ans=0.1 2023-11-23 11:53:56,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.42 vs. limit=22.5 2023-11-23 11:54:01,068 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.203e+01 8.988e+01 9.539e+01 1.363e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-23 11:54:08,658 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6750, loss[loss=0.0741, simple_loss=0.0995, pruned_loss=0.01453, audio_tagging_loss=0.009818, over 15321.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09183, pruned_loss=0.01415, audio_tagging_loss=0.008979, over 3032889.81 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:54:14,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2369600.0, ans=0.2 2023-11-23 11:54:16,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2369600.0, ans=0.125 2023-11-23 11:54:18,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2369600.0, ans=0.95 2023-11-23 11:54:19,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355450 2023-11-23 11:54:24,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2369666.6666666665, ans=0.0 2023-11-23 11:54:48,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2369800.0, ans=0.1 2023-11-23 11:54:51,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2369800.0, ans=0.0 2023-11-23 11:54:58,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2369800.0, ans=0.125 2023-11-23 11:55:13,497 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6800, loss[loss=0.07411, simple_loss=0.09115, pruned_loss=0.01895, audio_tagging_loss=0.009588, over 15644.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09195, pruned_loss=0.01415, audio_tagging_loss=0.008968, over 3036227.22 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:55:24,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355500 2023-11-23 11:55:28,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2370000.0, ans=0.125 2023-11-23 11:55:46,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.40 vs. limit=15.0 2023-11-23 11:55:48,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2370066.6666666665, ans=0.125 2023-11-23 11:55:51,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2370133.3333333335, ans=0.05 2023-11-23 11:55:57,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2370133.3333333335, ans=0.0 2023-11-23 11:56:01,389 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.43 vs. limit=15.0 2023-11-23 11:56:03,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2370133.3333333335, ans=0.0 2023-11-23 11:56:04,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2370200.0, ans=0.125 2023-11-23 11:56:12,193 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.201e+01 8.354e+01 8.978e+01 9.690e+01 1.235e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 11:56:19,043 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6850, loss[loss=0.0697, simple_loss=0.08746, pruned_loss=0.01616, audio_tagging_loss=0.009809, over 14876.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09205, pruned_loss=0.01405, audio_tagging_loss=0.008926, over 3030877.56 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:56:20,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2370266.6666666665, ans=0.1 2023-11-23 11:56:25,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.whiten.whitening_limit, batch_count=2370266.6666666665, ans=15.0 2023-11-23 11:56:29,494 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355550 2023-11-23 11:56:30,137 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.08 vs. limit=15.0 2023-11-23 11:56:35,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2370333.3333333335, ans=0.125 2023-11-23 11:56:35,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2370333.3333333335, ans=0.125 2023-11-23 11:56:58,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.17 vs. limit=15.0 2023-11-23 11:57:17,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2370533.3333333335, ans=0.1 2023-11-23 11:57:23,673 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.64 vs. limit=15.0 2023-11-23 11:57:24,327 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6900, loss[loss=0.06722, simple_loss=0.08439, pruned_loss=0.0142, audio_tagging_loss=0.01083, over 15359.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09126, pruned_loss=0.0138, audio_tagging_loss=0.008933, over 3026553.71 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:57:30,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2370600.0, ans=0.1 2023-11-23 11:57:34,119 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355600 2023-11-23 11:57:56,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2370733.3333333335, ans=0.125 2023-11-23 11:57:56,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2370733.3333333335, ans=0.2 2023-11-23 11:58:15,724 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 11:58:23,523 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.385e+01 8.957e+01 9.598e+01 1.316e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 11:58:29,834 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 6950, loss[loss=0.07143, simple_loss=0.08777, pruned_loss=0.01774, audio_tagging_loss=0.009806, over 15663.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09118, pruned_loss=0.0139, audio_tagging_loss=0.009018, over 3027872.05 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:58:38,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2370933.3333333335, ans=0.1 2023-11-23 11:58:41,256 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355650 2023-11-23 11:58:41,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2370933.3333333335, ans=0.1 2023-11-23 11:58:44,724 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.93 vs. limit=15.0 2023-11-23 11:58:56,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2371066.6666666665, ans=0.125 2023-11-23 11:59:36,282 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7000, loss[loss=0.04738, simple_loss=0.04759, pruned_loss=0.007519, audio_tagging_loss=0.01606, over 14639.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09077, pruned_loss=0.01376, audio_tagging_loss=0.009127, over 3029577.04 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 11:59:46,822 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355700 2023-11-23 11:59:54,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2371333.3333333335, ans=0.125 2023-11-23 11:59:59,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2371333.3333333335, ans=0.125 2023-11-23 12:00:26,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2371533.3333333335, ans=0.1 2023-11-23 12:00:34,908 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.338e+01 8.854e+01 9.577e+01 1.141e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-23 12:00:39,887 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7050, loss[loss=0.05951, simple_loss=0.07589, pruned_loss=0.01181, audio_tagging_loss=0.009755, over 15598.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09107, pruned_loss=0.01389, audio_tagging_loss=0.009179, over 3037010.26 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:00:41,669 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.54 vs. limit=15.0 2023-11-23 12:00:49,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355750 2023-11-23 12:00:49,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2371600.0, ans=0.0 2023-11-23 12:01:13,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2371733.3333333335, ans=0.125 2023-11-23 12:01:18,064 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.68 vs. limit=15.0 2023-11-23 12:01:21,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2371800.0, ans=0.125 2023-11-23 12:01:27,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.08 vs. limit=12.0 2023-11-23 12:01:34,660 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.50 vs. limit=10.0 2023-11-23 12:01:39,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2371866.6666666665, ans=0.125 2023-11-23 12:01:43,832 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7100, loss[loss=0.0916, simple_loss=0.1169, pruned_loss=0.02146, audio_tagging_loss=0.01169, over 15534.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09129, pruned_loss=0.01393, audio_tagging_loss=0.009234, over 3037227.05 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:01:53,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2371933.3333333335, ans=0.2 2023-11-23 12:01:54,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355800 2023-11-23 12:02:03,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2372000.0, ans=0.125 2023-11-23 12:02:26,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.23 vs. limit=22.5 2023-11-23 12:02:35,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2372200.0, ans=0.125 2023-11-23 12:02:43,357 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.520e+01 9.184e+01 9.861e+01 1.233e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-23 12:02:49,731 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7150, loss[loss=0.05092, simple_loss=0.06784, pruned_loss=0.004905, audio_tagging_loss=0.01209, over 14909.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09182, pruned_loss=0.01388, audio_tagging_loss=0.009201, over 3035601.17 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:02:52,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.66 vs. limit=12.0 2023-11-23 12:03:00,189 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355850 2023-11-23 12:03:02,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.69 vs. limit=12.0 2023-11-23 12:03:12,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2372333.3333333335, ans=0.125 2023-11-23 12:03:16,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.87 vs. limit=15.0 2023-11-23 12:03:19,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2372400.0, ans=0.2 2023-11-23 12:03:28,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2372466.6666666665, ans=0.0 2023-11-23 12:03:31,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2372466.6666666665, ans=0.07 2023-11-23 12:03:32,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2372466.6666666665, ans=0.07 2023-11-23 12:03:33,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2372466.6666666665, ans=0.0 2023-11-23 12:03:42,970 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.61 vs. limit=15.0 2023-11-23 12:03:50,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2372533.3333333335, ans=0.125 2023-11-23 12:03:53,740 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7200, loss[loss=0.05867, simple_loss=0.07572, pruned_loss=0.0114, audio_tagging_loss=0.009401, over 14876.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09182, pruned_loss=0.01391, audio_tagging_loss=0.009244, over 3033029.23 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:04:01,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2372600.0, ans=0.0 2023-11-23 12:04:02,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2372600.0, ans=0.125 2023-11-23 12:04:03,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355900 2023-11-23 12:04:03,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2372600.0, ans=0.2 2023-11-23 12:04:13,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2372666.6666666665, ans=0.1 2023-11-23 12:04:27,458 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.31 vs. limit=15.0 2023-11-23 12:04:37,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2372800.0, ans=0.0 2023-11-23 12:04:38,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2372800.0, ans=0.125 2023-11-23 12:04:52,178 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.220e+01 8.499e+01 9.077e+01 9.977e+01 1.549e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-23 12:04:57,115 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7250, loss[loss=0.06672, simple_loss=0.08586, pruned_loss=0.01369, audio_tagging_loss=0.01011, over 15307.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09172, pruned_loss=0.01388, audio_tagging_loss=0.009354, over 3027604.99 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:04:59,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2372933.3333333335, ans=0.2 2023-11-23 12:05:07,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 355950 2023-11-23 12:05:19,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2373000.0, ans=0.125 2023-11-23 12:05:20,332 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.92 vs. limit=22.5 2023-11-23 12:05:45,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.25 vs. limit=15.0 2023-11-23 12:05:57,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2373200.0, ans=0.04949747468305833 2023-11-23 12:06:01,450 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7300, loss[loss=0.05883, simple_loss=0.08414, pruned_loss=0.007843, audio_tagging_loss=0.008918, over 15986.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09196, pruned_loss=0.01383, audio_tagging_loss=0.00925, over 3031292.11 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:06:02,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.97 vs. limit=15.0 2023-11-23 12:06:12,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356000 2023-11-23 12:06:32,066 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.03 vs. limit=10.0 2023-11-23 12:06:36,936 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.47 vs. limit=22.5 2023-11-23 12:06:42,844 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.57 vs. limit=15.0 2023-11-23 12:06:52,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2373466.6666666665, ans=0.0 2023-11-23 12:07:05,006 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.072e+01 8.323e+01 8.907e+01 9.427e+01 1.100e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-23 12:07:10,575 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7350, loss[loss=0.06901, simple_loss=0.09328, pruned_loss=0.01315, audio_tagging_loss=0.009226, over 14833.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09382, pruned_loss=0.01397, audio_tagging_loss=0.009049, over 3032557.69 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:07:14,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2373600.0, ans=0.125 2023-11-23 12:07:15,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2373600.0, ans=0.125 2023-11-23 12:07:20,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356050 2023-11-23 12:07:22,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2373666.6666666665, ans=0.125 2023-11-23 12:07:47,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2373800.0, ans=0.125 2023-11-23 12:07:52,455 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0 2023-11-23 12:08:03,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2373866.6666666665, ans=0.2 2023-11-23 12:08:03,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2373866.6666666665, ans=0.1 2023-11-23 12:08:14,466 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7400, loss[loss=0.07028, simple_loss=0.1042, pruned_loss=0.01193, audio_tagging_loss=0.006244, over 16224.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09319, pruned_loss=0.01381, audio_tagging_loss=0.009015, over 3034345.01 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:08:24,205 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356100 2023-11-23 12:08:35,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2374000.0, ans=0.0 2023-11-23 12:08:37,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2374000.0, ans=0.1 2023-11-23 12:08:52,699 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.22 vs. limit=15.0 2023-11-23 12:08:52,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.88 vs. limit=15.0 2023-11-23 12:08:55,249 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.97 vs. limit=15.0 2023-11-23 12:09:00,137 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.24 vs. limit=15.0 2023-11-23 12:09:13,006 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.433e+01 8.940e+01 9.689e+01 1.513e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-23 12:09:16,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2374200.0, ans=0.125 2023-11-23 12:09:18,589 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7450, loss[loss=0.05675, simple_loss=0.07595, pruned_loss=0.01021, audio_tagging_loss=0.008562, over 15110.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.0934, pruned_loss=0.01403, audio_tagging_loss=0.008912, over 3045543.03 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:09:23,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.98 vs. limit=15.0 2023-11-23 12:09:29,117 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356150 2023-11-23 12:09:39,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2374333.3333333335, ans=0.125 2023-11-23 12:09:57,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2374466.6666666665, ans=0.0 2023-11-23 12:10:00,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2374466.6666666665, ans=0.125 2023-11-23 12:10:07,153 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.00 vs. limit=10.0 2023-11-23 12:10:17,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2374533.3333333335, ans=0.2 2023-11-23 12:10:20,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2374533.3333333335, ans=0.125 2023-11-23 12:10:23,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2374600.0, ans=0.0 2023-11-23 12:10:24,408 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7500, loss[loss=0.06871, simple_loss=0.09143, pruned_loss=0.0156, audio_tagging_loss=0.007398, over 15369.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09365, pruned_loss=0.01404, audio_tagging_loss=0.008915, over 3045599.82 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:10:34,378 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356200 2023-11-23 12:10:34,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2374600.0, ans=0.0 2023-11-23 12:10:39,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2374666.6666666665, ans=0.1 2023-11-23 12:10:51,472 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.76 vs. limit=22.5 2023-11-23 12:11:05,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.36 vs. limit=15.0 2023-11-23 12:11:18,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2374866.6666666665, ans=0.95 2023-11-23 12:11:23,843 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.224e+01 8.821e+01 9.416e+01 1.230e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-23 12:11:28,801 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7550, loss[loss=0.0634, simple_loss=0.08703, pruned_loss=0.009215, audio_tagging_loss=0.01067, over 15359.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09298, pruned_loss=0.01389, audio_tagging_loss=0.008986, over 3042008.76 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:11:38,795 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356250 2023-11-23 12:11:38,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2374933.3333333335, ans=0.1 2023-11-23 12:11:57,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2375066.6666666665, ans=0.125 2023-11-23 12:12:15,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2375133.3333333335, ans=0.125 2023-11-23 12:12:19,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2375133.3333333335, ans=0.5 2023-11-23 12:12:34,270 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7600, loss[loss=0.07528, simple_loss=0.103, pruned_loss=0.01423, audio_tagging_loss=0.009558, over 16492.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.093, pruned_loss=0.01391, audio_tagging_loss=0.009032, over 3044410.90 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:12:34,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2375266.6666666665, ans=0.125 2023-11-23 12:12:42,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0 2023-11-23 12:12:44,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.91 vs. limit=15.0 2023-11-23 12:12:44,666 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356300 2023-11-23 12:12:57,426 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.48 vs. limit=6.0 2023-11-23 12:13:33,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2375533.3333333335, ans=0.125 2023-11-23 12:13:34,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2375533.3333333335, ans=0.0 2023-11-23 12:13:34,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2375533.3333333335, ans=0.07 2023-11-23 12:13:35,219 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.003e+01 8.205e+01 8.929e+01 9.831e+01 1.216e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-23 12:13:39,066 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7650, loss[loss=0.05846, simple_loss=0.07889, pruned_loss=0.009122, audio_tagging_loss=0.00989, over 14321.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09231, pruned_loss=0.01392, audio_tagging_loss=0.008989, over 3039905.00 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:13:49,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2375600.0, ans=0.0 2023-11-23 12:13:50,306 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356350 2023-11-23 12:14:21,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2375800.0, ans=0.0 2023-11-23 12:14:32,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2375866.6666666665, ans=0.2 2023-11-23 12:14:40,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2375866.6666666665, ans=0.125 2023-11-23 12:14:44,731 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7700, loss[loss=0.05617, simple_loss=0.07523, pruned_loss=0.008811, audio_tagging_loss=0.009746, over 13900.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09178, pruned_loss=0.01373, audio_tagging_loss=0.009023, over 3043233.50 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:14:45,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.96 vs. limit=15.0 2023-11-23 12:14:54,585 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356400 2023-11-23 12:15:05,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2376000.0, ans=0.125 2023-11-23 12:15:12,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2376066.6666666665, ans=0.2 2023-11-23 12:15:13,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2376066.6666666665, ans=0.1 2023-11-23 12:15:44,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2376200.0, ans=0.0 2023-11-23 12:15:45,414 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.176e+01 8.917e+01 9.503e+01 1.166e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 12:15:49,148 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7750, loss[loss=0.08186, simple_loss=0.1105, pruned_loss=0.01918, audio_tagging_loss=0.007417, over 15860.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.0929, pruned_loss=0.01395, audio_tagging_loss=0.009052, over 3044916.65 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:15:55,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2376266.6666666665, ans=0.1 2023-11-23 12:16:00,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356450 2023-11-23 12:16:21,697 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.46 vs. limit=10.0 2023-11-23 12:16:24,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2376400.0, ans=0.0 2023-11-23 12:16:26,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2376400.0, ans=0.2 2023-11-23 12:16:54,277 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7800, loss[loss=0.0819, simple_loss=0.1088, pruned_loss=0.01966, audio_tagging_loss=0.007851, over 13809.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09267, pruned_loss=0.01393, audio_tagging_loss=0.009092, over 3035632.50 frames. ], batch size: 52, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:17:04,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356500 2023-11-23 12:17:26,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2376733.3333333335, ans=0.125 2023-11-23 12:17:54,731 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.181e+01 8.920e+01 9.526e+01 1.528e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 12:17:57,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2376933.3333333335, ans=0.125 2023-11-23 12:17:58,485 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7850, loss[loss=0.06539, simple_loss=0.09193, pruned_loss=0.01134, audio_tagging_loss=0.008094, over 16029.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09169, pruned_loss=0.01365, audio_tagging_loss=0.009134, over 3036386.81 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:18:04,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2376933.3333333335, ans=0.125 2023-11-23 12:18:05,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2376933.3333333335, ans=0.2 2023-11-23 12:18:05,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.85 vs. limit=15.0 2023-11-23 12:18:07,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2376933.3333333335, ans=0.0 2023-11-23 12:18:09,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356550 2023-11-23 12:18:35,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2377133.3333333335, ans=0.125 2023-11-23 12:18:39,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2377133.3333333335, ans=0.2 2023-11-23 12:19:02,632 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7900, loss[loss=0.07018, simple_loss=0.09792, pruned_loss=0.0129, audio_tagging_loss=0.008317, over 15421.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09278, pruned_loss=0.01394, audio_tagging_loss=0.009165, over 3038250.57 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:19:13,261 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356600 2023-11-23 12:19:18,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2377333.3333333335, ans=0.125 2023-11-23 12:19:20,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2377333.3333333335, ans=0.5 2023-11-23 12:19:58,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.23 vs. limit=15.0 2023-11-23 12:20:04,576 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.810e+01 8.633e+01 9.246e+01 1.018e+02 1.529e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-23 12:20:08,306 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 7950, loss[loss=0.08543, simple_loss=0.123, pruned_loss=0.01598, audio_tagging_loss=0.007954, over 15053.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09274, pruned_loss=0.01381, audio_tagging_loss=0.009241, over 3041116.68 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:20:10,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2377600.0, ans=0.025 2023-11-23 12:20:14,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2377600.0, ans=0.5 2023-11-23 12:20:18,753 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356650 2023-11-23 12:20:26,010 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 12:20:37,023 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.60 vs. limit=15.0 2023-11-23 12:21:03,137 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.04 vs. limit=15.0 2023-11-23 12:21:12,862 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8000, loss[loss=0.05898, simple_loss=0.07645, pruned_loss=0.00996, audio_tagging_loss=0.0108, over 14999.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.0919, pruned_loss=0.01375, audio_tagging_loss=0.009379, over 3036425.67 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:21:23,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356700 2023-11-23 12:22:15,960 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.250e+01 8.793e+01 9.379e+01 1.192e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-23 12:22:17,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2378266.6666666665, ans=0.07 2023-11-23 12:22:18,453 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8050, loss[loss=0.07043, simple_loss=0.09147, pruned_loss=0.01629, audio_tagging_loss=0.008406, over 14931.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09105, pruned_loss=0.01369, audio_tagging_loss=0.009436, over 3039161.32 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:22:28,989 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356750 2023-11-23 12:22:50,623 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0 2023-11-23 12:22:57,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2378466.6666666665, ans=0.2 2023-11-23 12:23:15,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2378533.3333333335, ans=0.0 2023-11-23 12:23:18,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.08 vs. limit=22.5 2023-11-23 12:23:20,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2378533.3333333335, ans=0.125 2023-11-23 12:23:21,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2378533.3333333335, ans=0.0 2023-11-23 12:23:24,500 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8100, loss[loss=0.07478, simple_loss=0.09967, pruned_loss=0.01586, audio_tagging_loss=0.00908, over 17251.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09116, pruned_loss=0.01381, audio_tagging_loss=0.009378, over 3037897.20 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:23:32,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2378600.0, ans=0.125 2023-11-23 12:23:34,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356800 2023-11-23 12:23:36,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2378666.6666666665, ans=0.0 2023-11-23 12:24:13,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2378800.0, ans=0.0 2023-11-23 12:24:13,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2378800.0, ans=0.125 2023-11-23 12:24:13,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2378800.0, ans=0.0 2023-11-23 12:24:24,581 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.01 vs. limit=15.0 2023-11-23 12:24:27,909 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.660e+01 9.350e+01 9.941e+01 1.281e+02, threshold=1.870e+02, percent-clipped=0.0 2023-11-23 12:24:30,484 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8150, loss[loss=0.08265, simple_loss=0.1192, pruned_loss=0.01729, audio_tagging_loss=0.005752, over 16222.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09162, pruned_loss=0.01394, audio_tagging_loss=0.009188, over 3040677.75 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:24:38,874 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.94 vs. limit=15.0 2023-11-23 12:24:40,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356850 2023-11-23 12:24:42,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.40 vs. limit=15.0 2023-11-23 12:25:10,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2379133.3333333335, ans=0.0 2023-11-23 12:25:21,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2379200.0, ans=0.1 2023-11-23 12:25:30,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2379200.0, ans=0.2 2023-11-23 12:25:34,967 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8200, loss[loss=0.07378, simple_loss=0.09721, pruned_loss=0.01511, audio_tagging_loss=0.01007, over 15195.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09162, pruned_loss=0.01407, audio_tagging_loss=0.009093, over 3044028.85 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:25:37,460 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 12:25:46,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356900 2023-11-23 12:26:18,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2379466.6666666665, ans=0.125 2023-11-23 12:26:29,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2379533.3333333335, ans=0.2 2023-11-23 12:26:33,154 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 12:26:38,260 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 8.430e+01 9.137e+01 9.990e+01 1.706e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-23 12:26:40,854 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8250, loss[loss=0.07035, simple_loss=0.09523, pruned_loss=0.01453, audio_tagging_loss=0.008198, over 16625.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.0912, pruned_loss=0.01403, audio_tagging_loss=0.00908, over 3039818.01 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:26:51,526 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 356950 2023-11-23 12:27:00,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2379666.6666666665, ans=0.2 2023-11-23 12:27:45,819 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8300, loss[loss=0.08166, simple_loss=0.1123, pruned_loss=0.01495, audio_tagging_loss=0.01055, over 14701.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09193, pruned_loss=0.01398, audio_tagging_loss=0.009012, over 3038286.43 frames. ], batch size: 53, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:27:46,571 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.38 vs. limit=6.0 2023-11-23 12:27:55,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357000 2023-11-23 12:27:56,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.01 vs. limit=22.5 2023-11-23 12:28:02,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2380000.0, ans=0.2 2023-11-23 12:28:08,024 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0 2023-11-23 12:28:10,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2380066.6666666665, ans=0.125 2023-11-23 12:28:17,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2380066.6666666665, ans=0.125 2023-11-23 12:28:28,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2380133.3333333335, ans=0.2 2023-11-23 12:28:33,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2380133.3333333335, ans=0.0 2023-11-23 12:28:47,202 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0 2023-11-23 12:28:47,636 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.282e+01 8.543e+01 9.003e+01 9.782e+01 1.205e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 12:28:48,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2380200.0, ans=0.1 2023-11-23 12:28:50,059 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8350, loss[loss=0.07052, simple_loss=0.09374, pruned_loss=0.01598, audio_tagging_loss=0.007668, over 15654.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09231, pruned_loss=0.014, audio_tagging_loss=0.008917, over 3046492.63 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:28:52,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2380266.6666666665, ans=0.125 2023-11-23 12:29:00,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357050 2023-11-23 12:29:39,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2380466.6666666665, ans=0.5 2023-11-23 12:29:49,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2380533.3333333335, ans=0.1 2023-11-23 12:29:54,906 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8400, loss[loss=0.07973, simple_loss=0.1076, pruned_loss=0.01783, audio_tagging_loss=0.008091, over 14888.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09309, pruned_loss=0.01396, audio_tagging_loss=0.00884, over 3046323.82 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:30:05,183 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357100 2023-11-23 12:30:13,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2380666.6666666665, ans=0.125 2023-11-23 12:30:30,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2380733.3333333335, ans=0.0 2023-11-23 12:30:39,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2380800.0, ans=0.125 2023-11-23 12:30:49,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2380866.6666666665, ans=0.125 2023-11-23 12:30:57,938 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.179e+01 8.935e+01 9.716e+01 1.287e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 12:30:59,219 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8450, loss[loss=0.04336, simple_loss=0.05176, pruned_loss=0.008102, audio_tagging_loss=0.009381, over 14916.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09194, pruned_loss=0.01388, audio_tagging_loss=0.008851, over 3047223.67 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:31:01,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2380933.3333333335, ans=0.025 2023-11-23 12:31:09,775 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357150 2023-11-23 12:31:16,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2381000.0, ans=0.125 2023-11-23 12:31:22,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2381000.0, ans=0.2 2023-11-23 12:31:26,165 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.21 vs. limit=15.0 2023-11-23 12:31:27,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2381066.6666666665, ans=0.2 2023-11-23 12:31:47,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2381133.3333333335, ans=0.125 2023-11-23 12:32:03,573 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8500, loss[loss=0.06397, simple_loss=0.08323, pruned_loss=0.01307, audio_tagging_loss=0.009278, over 16000.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09196, pruned_loss=0.01394, audio_tagging_loss=0.008858, over 3048962.11 frames. ], batch size: 62, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:32:07,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2381266.6666666665, ans=0.2 2023-11-23 12:32:11,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2381266.6666666665, ans=0.2 2023-11-23 12:32:12,774 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.85 vs. limit=10.0 2023-11-23 12:32:13,540 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357200 2023-11-23 12:32:30,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2381400.0, ans=0.125 2023-11-23 12:32:35,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2381400.0, ans=0.125 2023-11-23 12:32:41,860 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.94 vs. limit=15.0 2023-11-23 12:32:54,511 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.09 vs. limit=15.0 2023-11-23 12:33:06,340 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.293e+01 8.960e+01 9.755e+01 1.208e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-23 12:33:08,143 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8550, loss[loss=0.07465, simple_loss=0.09225, pruned_loss=0.01645, audio_tagging_loss=0.01207, over 14957.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09267, pruned_loss=0.01398, audio_tagging_loss=0.008897, over 3047271.77 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:33:08,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.36 vs. limit=15.0 2023-11-23 12:33:15,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2381600.0, ans=0.0 2023-11-23 12:33:18,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2381600.0, ans=0.125 2023-11-23 12:33:19,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357250 2023-11-23 12:33:19,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2381600.0, ans=0.125 2023-11-23 12:33:41,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2381733.3333333335, ans=0.0 2023-11-23 12:33:46,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2381800.0, ans=0.07 2023-11-23 12:33:51,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2381800.0, ans=0.125 2023-11-23 12:33:58,437 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.95 vs. limit=22.5 2023-11-23 12:34:09,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2381866.6666666665, ans=0.0 2023-11-23 12:34:13,822 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8600, loss[loss=0.08491, simple_loss=0.1148, pruned_loss=0.01904, audio_tagging_loss=0.008474, over 15547.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09287, pruned_loss=0.01407, audio_tagging_loss=0.008965, over 3050134.48 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:34:24,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357300 2023-11-23 12:34:33,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2382000.0, ans=0.2 2023-11-23 12:34:34,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2382000.0, ans=0.015 2023-11-23 12:34:42,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2382066.6666666665, ans=0.0 2023-11-23 12:35:17,575 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.105e+01 8.339e+01 9.203e+01 9.998e+01 1.297e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-23 12:35:18,842 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8650, loss[loss=0.08146, simple_loss=0.1107, pruned_loss=0.01721, audio_tagging_loss=0.008909, over 15973.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09361, pruned_loss=0.01417, audio_tagging_loss=0.009025, over 3056381.69 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:35:28,751 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357350 2023-11-23 12:35:37,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2382333.3333333335, ans=0.125 2023-11-23 12:35:43,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2382400.0, ans=0.125 2023-11-23 12:35:44,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2382400.0, ans=0.0 2023-11-23 12:35:46,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2382400.0, ans=0.0 2023-11-23 12:35:57,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2382466.6666666665, ans=0.2 2023-11-23 12:36:22,800 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8700, loss[loss=0.07569, simple_loss=0.09854, pruned_loss=0.01721, audio_tagging_loss=0.009206, over 14965.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09375, pruned_loss=0.0142, audio_tagging_loss=0.009121, over 3048245.63 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:36:27,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2382600.0, ans=0.1 2023-11-23 12:36:32,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2382600.0, ans=0.2 2023-11-23 12:36:33,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357400 2023-11-23 12:36:34,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2382600.0, ans=0.125 2023-11-23 12:36:34,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2382600.0, ans=0.125 2023-11-23 12:36:35,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2382666.6666666665, ans=0.2 2023-11-23 12:36:53,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2382733.3333333335, ans=0.0 2023-11-23 12:36:56,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2382733.3333333335, ans=0.2 2023-11-23 12:37:27,564 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.508e+01 9.323e+01 9.917e+01 1.440e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-23 12:37:28,915 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8750, loss[loss=0.04963, simple_loss=0.06531, pruned_loss=0.007118, audio_tagging_loss=0.009852, over 14098.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09471, pruned_loss=0.01444, audio_tagging_loss=0.009105, over 3044492.23 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:37:40,205 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357450 2023-11-23 12:38:00,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.50 vs. limit=15.0 2023-11-23 12:38:00,954 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.16 vs. limit=15.0 2023-11-23 12:38:21,913 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.78 vs. limit=22.5 2023-11-23 12:38:31,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2383200.0, ans=0.0 2023-11-23 12:38:32,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2383200.0, ans=15.0 2023-11-23 12:38:35,189 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8800, loss[loss=0.08367, simple_loss=0.1099, pruned_loss=0.02044, audio_tagging_loss=0.008266, over 15562.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09483, pruned_loss=0.01446, audio_tagging_loss=0.009265, over 3049355.38 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0 2023-11-23 12:38:44,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2383266.6666666665, ans=0.0 2023-11-23 12:38:44,995 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357500 2023-11-23 12:38:48,152 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.04 vs. limit=15.0 2023-11-23 12:38:53,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2383333.3333333335, ans=0.05 2023-11-23 12:39:25,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2383466.6666666665, ans=0.0 2023-11-23 12:39:33,005 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.46 vs. limit=15.0 2023-11-23 12:39:39,751 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.582e+01 9.222e+01 9.925e+01 1.869e+02, threshold=1.844e+02, percent-clipped=1.0 2023-11-23 12:39:39,801 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8850, loss[loss=0.05991, simple_loss=0.07352, pruned_loss=0.01305, audio_tagging_loss=0.0101, over 14735.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09404, pruned_loss=0.01442, audio_tagging_loss=0.009358, over 3048127.48 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:39:45,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2383600.0, ans=0.125 2023-11-23 12:39:50,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357550 2023-11-23 12:39:54,652 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 12:40:33,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2383866.6666666665, ans=0.125 2023-11-23 12:40:42,469 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0 2023-11-23 12:40:44,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2383933.3333333335, ans=0.125 2023-11-23 12:40:45,538 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8900, loss[loss=0.07188, simple_loss=0.1019, pruned_loss=0.01403, audio_tagging_loss=0.006924, over 14778.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09459, pruned_loss=0.01458, audio_tagging_loss=0.009177, over 3046433.57 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:40:46,437 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.22 vs. limit=22.5 2023-11-23 12:40:55,934 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357600 2023-11-23 12:41:44,889 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.37 vs. limit=15.0 2023-11-23 12:41:51,026 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.078e+01 8.629e+01 9.554e+01 1.140e+02, threshold=1.726e+02, percent-clipped=0.0 2023-11-23 12:41:51,084 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 8950, loss[loss=0.05666, simple_loss=0.07363, pruned_loss=0.01222, audio_tagging_loss=0.007622, over 15571.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09418, pruned_loss=0.01448, audio_tagging_loss=0.009043, over 3050002.96 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:41:59,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2384266.6666666665, ans=0.125 2023-11-23 12:42:01,018 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357650 2023-11-23 12:42:03,968 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.15 vs. limit=15.0 2023-11-23 12:42:30,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.13 vs. limit=15.0 2023-11-23 12:42:52,667 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.08 vs. limit=22.5 2023-11-23 12:42:54,535 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9000, loss[loss=0.06957, simple_loss=0.09918, pruned_loss=0.01264, audio_tagging_loss=0.007348, over 16042.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.0943, pruned_loss=0.01445, audio_tagging_loss=0.008966, over 3054097.51 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:42:54,536 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 12:43:16,057 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9376, 3.7122, 4.9017, 4.4317], device='cuda:3') 2023-11-23 12:43:36,342 INFO [train_asr.py:1253] (3/4) Epoch 30, validation: loss=0.05877, simple_loss=0.051, pruned_loss=0.005026, audio_tagging_loss=0.02824, over 4681554.00 frames. 2023-11-23 12:43:36,343 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 12:43:46,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357700 2023-11-23 12:44:16,137 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 12:44:25,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2384800.0, ans=0.0 2023-11-23 12:44:29,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2384866.6666666665, ans=0.0 2023-11-23 12:44:35,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2384866.6666666665, ans=0.015 2023-11-23 12:44:41,115 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.551e+01 8.530e+01 8.986e+01 9.928e+01 1.282e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 12:44:41,163 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9050, loss[loss=0.05751, simple_loss=0.07257, pruned_loss=0.01282, audio_tagging_loss=0.0084, over 14763.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.0935, pruned_loss=0.01416, audio_tagging_loss=0.008938, over 3053519.13 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0 2023-11-23 12:44:51,060 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357750 2023-11-23 12:44:54,180 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.31 vs. limit=15.0 2023-11-23 12:44:58,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2385000.0, ans=0.125 2023-11-23 12:45:11,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.20 vs. limit=10.0 2023-11-23 12:45:44,796 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9100, loss[loss=0.07884, simple_loss=0.1111, pruned_loss=0.01531, audio_tagging_loss=0.007987, over 15920.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09274, pruned_loss=0.01405, audio_tagging_loss=0.008959, over 3049291.76 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 12:45:53,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.09 vs. limit=12.0 2023-11-23 12:45:55,236 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357800 2023-11-23 12:46:17,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2385400.0, ans=0.0 2023-11-23 12:46:19,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.03 vs. limit=10.0 2023-11-23 12:46:24,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2385466.6666666665, ans=0.125 2023-11-23 12:46:42,612 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.28 vs. limit=22.5 2023-11-23 12:46:44,092 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=15.0 2023-11-23 12:46:49,805 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.972e+01 8.272e+01 8.936e+01 9.614e+01 1.250e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 12:46:49,852 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9150, loss[loss=0.0546, simple_loss=0.07015, pruned_loss=0.01062, audio_tagging_loss=0.008907, over 15684.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09287, pruned_loss=0.01401, audio_tagging_loss=0.008918, over 3045754.72 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 12:47:00,189 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357850 2023-11-23 12:47:00,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2385600.0, ans=0.125 2023-11-23 12:47:05,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2385666.6666666665, ans=0.125 2023-11-23 12:47:06,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2385666.6666666665, ans=0.0 2023-11-23 12:47:14,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2385733.3333333335, ans=0.125 2023-11-23 12:47:29,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2385800.0, ans=0.125 2023-11-23 12:47:34,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2385800.0, ans=0.0 2023-11-23 12:47:37,474 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.03 vs. limit=15.0 2023-11-23 12:47:46,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2385866.6666666665, ans=0.125 2023-11-23 12:47:53,051 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9200, loss[loss=0.07663, simple_loss=0.1002, pruned_loss=0.01608, audio_tagging_loss=0.01042, over 14638.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09325, pruned_loss=0.01417, audio_tagging_loss=0.008866, over 3051589.17 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 12:47:53,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.73 vs. limit=15.0 2023-11-23 12:47:58,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2385933.3333333335, ans=0.125 2023-11-23 12:48:03,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357900 2023-11-23 12:48:04,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2386000.0, ans=0.125 2023-11-23 12:48:12,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2386000.0, ans=0.125 2023-11-23 12:48:44,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2386200.0, ans=0.2 2023-11-23 12:48:56,514 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.180e+01 8.978e+01 9.494e+01 1.363e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 12:48:56,558 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9250, loss[loss=0.05598, simple_loss=0.06745, pruned_loss=0.01033, audio_tagging_loss=0.01192, over 15211.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09301, pruned_loss=0.01405, audio_tagging_loss=0.008823, over 3053810.54 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 12:49:06,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 357950 2023-11-23 12:49:23,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2386400.0, ans=0.2 2023-11-23 12:49:31,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2386400.0, ans=0.125 2023-11-23 12:49:37,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2386466.6666666665, ans=0.1 2023-11-23 12:49:42,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.05 vs. limit=15.0 2023-11-23 12:49:47,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2386533.3333333335, ans=0.125 2023-11-23 12:50:00,437 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9300, loss[loss=0.05805, simple_loss=0.0652, pruned_loss=0.01161, audio_tagging_loss=0.01385, over 15032.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.0925, pruned_loss=0.01395, audio_tagging_loss=0.008916, over 3053428.02 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 12:50:06,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.59 vs. limit=12.0 2023-11-23 12:50:10,753 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358000 2023-11-23 12:50:25,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2386733.3333333335, ans=0.0 2023-11-23 12:50:37,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2386800.0, ans=0.2 2023-11-23 12:50:45,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2386800.0, ans=0.125 2023-11-23 12:50:45,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2386800.0, ans=0.2 2023-11-23 12:51:00,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2386866.6666666665, ans=0.0 2023-11-23 12:51:04,207 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.283e+01 8.829e+01 9.752e+01 1.235e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 12:51:04,252 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9350, loss[loss=0.06824, simple_loss=0.09674, pruned_loss=0.01104, audio_tagging_loss=0.008831, over 15612.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09174, pruned_loss=0.01386, audio_tagging_loss=0.009153, over 3050745.00 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 12:51:09,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2386933.3333333335, ans=0.125 2023-11-23 12:51:10,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.62 vs. limit=22.5 2023-11-23 12:51:13,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358050 2023-11-23 12:51:28,551 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.96 vs. limit=22.5 2023-11-23 12:51:29,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2387066.6666666665, ans=0.125 2023-11-23 12:51:30,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.59 vs. limit=15.0 2023-11-23 12:51:38,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.92 vs. limit=15.0 2023-11-23 12:51:43,290 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.99 vs. limit=12.0 2023-11-23 12:52:07,411 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9400, loss[loss=0.06829, simple_loss=0.09017, pruned_loss=0.01364, audio_tagging_loss=0.009566, over 16190.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09179, pruned_loss=0.0138, audio_tagging_loss=0.009241, over 3046091.70 frames. ], batch size: 61, lr: 2.25e-03, grad_scale: 8.0 2023-11-23 12:52:17,968 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358100 2023-11-23 12:52:22,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2387333.3333333335, ans=0.04949747468305833 2023-11-23 12:52:51,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2387466.6666666665, ans=0.1 2023-11-23 12:53:10,720 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 12:53:11,894 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9450, loss[loss=0.04694, simple_loss=0.06467, pruned_loss=0.00464, audio_tagging_loss=0.009967, over 14698.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09192, pruned_loss=0.01385, audio_tagging_loss=0.009237, over 3047182.05 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 8.0 2023-11-23 12:53:14,838 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.523e+01 9.315e+01 1.049e+02 1.250e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-23 12:53:22,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358150 2023-11-23 12:53:22,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2387600.0, ans=0.0 2023-11-23 12:53:29,171 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 12:53:41,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2387733.3333333335, ans=0.1 2023-11-23 12:53:44,099 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.87 vs. limit=15.0 2023-11-23 12:53:59,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.15 vs. limit=12.0 2023-11-23 12:54:08,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2387866.6666666665, ans=0.0 2023-11-23 12:54:16,502 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9500, loss[loss=0.06656, simple_loss=0.08435, pruned_loss=0.01421, audio_tagging_loss=0.01018, over 15502.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09204, pruned_loss=0.01393, audio_tagging_loss=0.009261, over 3045335.84 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 8.0 2023-11-23 12:54:21,111 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=15.0 2023-11-23 12:54:21,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2387933.3333333335, ans=0.0 2023-11-23 12:54:26,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358200 2023-11-23 12:54:35,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2388000.0, ans=0.025 2023-11-23 12:54:36,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2388000.0, ans=0.125 2023-11-23 12:55:11,088 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.34 vs. limit=15.0 2023-11-23 12:55:19,862 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9550, loss[loss=0.06026, simple_loss=0.06686, pruned_loss=0.01076, audio_tagging_loss=0.01607, over 14485.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09238, pruned_loss=0.01406, audio_tagging_loss=0.009368, over 3053066.94 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 8.0 2023-11-23 12:55:22,284 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.501e+01 9.046e+01 9.743e+01 1.461e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 12:55:25,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2388266.6666666665, ans=0.125 2023-11-23 12:55:28,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2388266.6666666665, ans=0.015 2023-11-23 12:55:29,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358250 2023-11-23 12:55:31,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2388333.3333333335, ans=0.125 2023-11-23 12:55:47,797 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.80 vs. limit=22.5 2023-11-23 12:55:55,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2388400.0, ans=0.0 2023-11-23 12:55:57,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2388466.6666666665, ans=0.125 2023-11-23 12:56:03,107 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0 2023-11-23 12:56:03,210 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.40 vs. limit=22.5 2023-11-23 12:56:07,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.98 vs. limit=15.0 2023-11-23 12:56:15,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2388533.3333333335, ans=0.2 2023-11-23 12:56:21,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2388533.3333333335, ans=0.0 2023-11-23 12:56:24,042 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9600, loss[loss=0.05688, simple_loss=0.06859, pruned_loss=0.01334, audio_tagging_loss=0.00924, over 15037.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09207, pruned_loss=0.01404, audio_tagging_loss=0.009433, over 3043698.61 frames. ], batch size: 61, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 12:56:24,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2388600.0, ans=0.125 2023-11-23 12:56:30,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2388600.0, ans=0.125 2023-11-23 12:56:30,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2388600.0, ans=0.0 2023-11-23 12:56:34,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358300 2023-11-23 12:56:51,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2388733.3333333335, ans=0.0 2023-11-23 12:57:06,086 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.47 vs. limit=15.0 2023-11-23 12:57:21,267 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.39 vs. limit=15.0 2023-11-23 12:57:23,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2388866.6666666665, ans=0.125 2023-11-23 12:57:24,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2388866.6666666665, ans=0.1 2023-11-23 12:57:27,874 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9650, loss[loss=0.07097, simple_loss=0.09248, pruned_loss=0.01565, audio_tagging_loss=0.009075, over 14831.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09245, pruned_loss=0.01403, audio_tagging_loss=0.009444, over 3046088.87 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 12:57:30,914 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.376e+01 8.856e+01 9.540e+01 1.217e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-23 12:57:32,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2388933.3333333335, ans=0.0 2023-11-23 12:57:38,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358350 2023-11-23 12:57:56,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2389066.6666666665, ans=0.1 2023-11-23 12:58:21,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2389200.0, ans=0.035 2023-11-23 12:58:29,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2389200.0, ans=0.125 2023-11-23 12:58:31,819 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9700, loss[loss=0.0558, simple_loss=0.07203, pruned_loss=0.01179, audio_tagging_loss=0.00799, over 15329.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09311, pruned_loss=0.0141, audio_tagging_loss=0.009298, over 3040509.52 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 12:58:41,915 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358400 2023-11-23 12:59:11,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2389466.6666666665, ans=0.025 2023-11-23 12:59:36,168 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9750, loss[loss=0.07347, simple_loss=0.09297, pruned_loss=0.01745, audio_tagging_loss=0.009536, over 15531.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09283, pruned_loss=0.01411, audio_tagging_loss=0.009202, over 3040442.96 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 12:59:39,182 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.327e+01 8.730e+01 9.519e+01 2.872e+02, threshold=1.746e+02, percent-clipped=1.0 2023-11-23 12:59:47,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358450 2023-11-23 12:59:52,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2389666.6666666665, ans=0.125 2023-11-23 12:59:57,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2389666.6666666665, ans=0.2 2023-11-23 13:00:01,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2389733.3333333335, ans=0.125 2023-11-23 13:00:40,793 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9800, loss[loss=0.06778, simple_loss=0.08889, pruned_loss=0.01066, audio_tagging_loss=0.01268, over 16796.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09182, pruned_loss=0.01407, audio_tagging_loss=0.009191, over 3042855.64 frames. ], batch size: 65, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:00:51,364 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358500 2023-11-23 13:01:10,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2390066.6666666665, ans=0.0 2023-11-23 13:01:19,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2390133.3333333335, ans=0.125 2023-11-23 13:01:34,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2390200.0, ans=0.2 2023-11-23 13:01:38,481 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.20 vs. limit=15.0 2023-11-23 13:01:38,906 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 13:01:45,074 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9850, loss[loss=0.07179, simple_loss=0.09751, pruned_loss=0.01736, audio_tagging_loss=0.00568, over 15359.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09289, pruned_loss=0.01431, audio_tagging_loss=0.009017, over 3051761.77 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:01:47,536 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.668e+01 8.339e+01 8.951e+01 9.575e+01 1.283e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-23 13:01:51,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.57 vs. limit=6.0 2023-11-23 13:01:55,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358550 2023-11-23 13:02:06,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2390333.3333333335, ans=0.2 2023-11-23 13:02:07,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2390333.3333333335, ans=0.125 2023-11-23 13:02:25,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2390466.6666666665, ans=0.125 2023-11-23 13:02:49,010 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9900, loss[loss=0.07138, simple_loss=0.09627, pruned_loss=0.01427, audio_tagging_loss=0.008974, over 14882.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09222, pruned_loss=0.01405, audio_tagging_loss=0.00901, over 3050710.57 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:02:50,928 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.73 vs. limit=15.0 2023-11-23 13:02:52,944 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:02:59,929 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358600 2023-11-23 13:03:10,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2390666.6666666665, ans=0.035 2023-11-23 13:03:11,658 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.51 vs. limit=15.0 2023-11-23 13:03:21,383 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.16 vs. limit=15.0 2023-11-23 13:03:50,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2390866.6666666665, ans=0.2 2023-11-23 13:03:53,919 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 9950, loss[loss=0.07868, simple_loss=0.09867, pruned_loss=0.0199, audio_tagging_loss=0.009447, over 16248.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09199, pruned_loss=0.01386, audio_tagging_loss=0.008979, over 3045383.07 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:03:56,388 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.331e+01 8.358e+01 9.006e+01 9.900e+01 1.170e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 13:04:04,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358650 2023-11-23 13:04:34,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2391133.3333333335, ans=0.125 2023-11-23 13:04:40,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2391133.3333333335, ans=0.0 2023-11-23 13:04:51,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2391200.0, ans=0.1 2023-11-23 13:04:57,636 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10000, loss[loss=0.06102, simple_loss=0.08446, pruned_loss=0.009487, audio_tagging_loss=0.009304, over 15155.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.0917, pruned_loss=0.01377, audio_tagging_loss=0.008979, over 3047240.60 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:05:06,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2391266.6666666665, ans=0.125 2023-11-23 13:05:07,311 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358700 2023-11-23 13:05:33,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2391400.0, ans=0.125 2023-11-23 13:05:39,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2391466.6666666665, ans=0.0 2023-11-23 13:05:42,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2391466.6666666665, ans=0.0 2023-11-23 13:06:01,423 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10050, loss[loss=0.07044, simple_loss=0.08967, pruned_loss=0.01732, audio_tagging_loss=0.008281, over 15286.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09284, pruned_loss=0.01409, audio_tagging_loss=0.008946, over 3045063.60 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:06:03,783 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.755e+01 8.322e+01 8.907e+01 9.751e+01 1.198e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-23 13:06:06,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2391600.0, ans=0.5 2023-11-23 13:06:11,106 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358750 2023-11-23 13:06:45,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0 2023-11-23 13:06:58,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2391866.6666666665, ans=0.2 2023-11-23 13:07:04,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2391933.3333333335, ans=0.0 2023-11-23 13:07:05,476 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10100, loss[loss=0.04969, simple_loss=0.06887, pruned_loss=0.005513, audio_tagging_loss=0.009739, over 15163.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09258, pruned_loss=0.01377, audio_tagging_loss=0.009052, over 3048670.43 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:07:07,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2391933.3333333335, ans=0.0 2023-11-23 13:07:16,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358800 2023-11-23 13:07:30,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2392066.6666666665, ans=0.125 2023-11-23 13:07:43,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2392133.3333333335, ans=0.125 2023-11-23 13:07:47,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2392133.3333333335, ans=0.125 2023-11-23 13:07:57,873 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 13:08:10,174 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10150, loss[loss=0.0625, simple_loss=0.08727, pruned_loss=0.01025, audio_tagging_loss=0.008616, over 16466.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09331, pruned_loss=0.01401, audio_tagging_loss=0.009076, over 3054227.17 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:08:13,710 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 8.548e+01 9.170e+01 9.737e+01 1.223e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-23 13:08:14,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2392266.6666666665, ans=0.05 2023-11-23 13:08:15,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2392266.6666666665, ans=0.0 2023-11-23 13:08:18,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2392266.6666666665, ans=0.125 2023-11-23 13:08:19,854 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358850 2023-11-23 13:08:34,139 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=12.0 2023-11-23 13:08:40,893 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 13:08:41,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2392400.0, ans=0.125 2023-11-23 13:09:13,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2392600.0, ans=0.125 2023-11-23 13:09:13,911 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10200, loss[loss=0.06217, simple_loss=0.08568, pruned_loss=0.01178, audio_tagging_loss=0.00755, over 15236.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.0928, pruned_loss=0.01391, audio_tagging_loss=0.009095, over 3055731.42 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:09:23,750 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358900 2023-11-23 13:09:39,595 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 13:09:54,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2392800.0, ans=0.2 2023-11-23 13:09:59,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2392800.0, ans=0.0 2023-11-23 13:09:59,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2392800.0, ans=0.2 2023-11-23 13:10:18,489 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10250, loss[loss=0.07358, simple_loss=0.08889, pruned_loss=0.0158, audio_tagging_loss=0.01333, over 15417.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09215, pruned_loss=0.01387, audio_tagging_loss=0.009189, over 3051614.51 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:10:21,514 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.43 vs. limit=22.5 2023-11-23 13:10:22,077 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.314e+01 9.044e+01 9.616e+01 1.457e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 13:10:29,067 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 358950 2023-11-23 13:10:34,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2393000.0, ans=0.125 2023-11-23 13:10:39,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2393000.0, ans=0.125 2023-11-23 13:10:43,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.13 vs. limit=6.0 2023-11-23 13:11:22,490 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.23 vs. limit=15.0 2023-11-23 13:11:23,021 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10300, loss[loss=0.06541, simple_loss=0.09426, pruned_loss=0.01128, audio_tagging_loss=0.006998, over 14352.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09174, pruned_loss=0.01384, audio_tagging_loss=0.009264, over 3047916.80 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:11:33,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359000 2023-11-23 13:12:25,797 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:12:26,629 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10350, loss[loss=0.08567, simple_loss=0.1139, pruned_loss=0.01922, audio_tagging_loss=0.009511, over 15298.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09206, pruned_loss=0.01396, audio_tagging_loss=0.009272, over 3048971.77 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:12:30,158 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.344e+01 8.692e+01 9.326e+01 1.149e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-23 13:12:30,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2393600.0, ans=0.2 2023-11-23 13:12:35,652 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.51 vs. limit=22.5 2023-11-23 13:12:36,449 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359050 2023-11-23 13:12:43,283 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:12:57,596 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.32 vs. limit=15.0 2023-11-23 13:13:16,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2393866.6666666665, ans=0.0 2023-11-23 13:13:17,139 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.58 vs. limit=15.0 2023-11-23 13:13:20,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2393866.6666666665, ans=0.0 2023-11-23 13:13:22,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2393866.6666666665, ans=0.125 2023-11-23 13:13:22,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2393866.6666666665, ans=0.0 2023-11-23 13:13:30,175 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10400, loss[loss=0.04346, simple_loss=0.05003, pruned_loss=0.006227, audio_tagging_loss=0.01222, over 14243.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09197, pruned_loss=0.01383, audio_tagging_loss=0.009293, over 3043578.64 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:13:34,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2393933.3333333335, ans=0.125 2023-11-23 13:13:39,964 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359100 2023-11-23 13:13:40,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2393933.3333333335, ans=0.1 2023-11-23 13:13:40,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0 2023-11-23 13:13:41,696 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.09 vs. limit=15.0 2023-11-23 13:13:49,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2394000.0, ans=0.125 2023-11-23 13:13:51,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2394000.0, ans=0.125 2023-11-23 13:14:26,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2394200.0, ans=0.0 2023-11-23 13:14:29,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.08 vs. limit=22.5 2023-11-23 13:14:33,789 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10450, loss[loss=0.06119, simple_loss=0.08551, pruned_loss=0.009227, audio_tagging_loss=0.009204, over 14979.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09152, pruned_loss=0.01371, audio_tagging_loss=0.009316, over 3038059.83 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:14:37,446 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.401e+01 9.132e+01 1.001e+02 1.338e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-23 13:14:43,742 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359150 2023-11-23 13:14:56,141 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.41 vs. limit=12.0 2023-11-23 13:15:36,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2394600.0, ans=0.0 2023-11-23 13:15:37,335 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10500, loss[loss=0.08686, simple_loss=0.1149, pruned_loss=0.02204, audio_tagging_loss=0.00738, over 15497.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09207, pruned_loss=0.0139, audio_tagging_loss=0.009174, over 3037635.77 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:15:47,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359200 2023-11-23 13:15:51,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2394666.6666666665, ans=0.1 2023-11-23 13:15:53,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2394666.6666666665, ans=0.125 2023-11-23 13:16:05,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2394733.3333333335, ans=0.2 2023-11-23 13:16:12,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2394733.3333333335, ans=0.07 2023-11-23 13:16:12,729 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:16:12,764 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:16:18,086 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.90 vs. limit=15.0 2023-11-23 13:16:35,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.69 vs. limit=6.0 2023-11-23 13:16:41,123 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10550, loss[loss=0.06494, simple_loss=0.0853, pruned_loss=0.01409, audio_tagging_loss=0.008202, over 15255.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09239, pruned_loss=0.01403, audio_tagging_loss=0.009085, over 3034153.06 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:16:44,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2394933.3333333335, ans=0.125 2023-11-23 13:16:45,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.241e+01 8.844e+01 9.462e+01 1.240e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 13:16:51,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359250 2023-11-23 13:17:13,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2395066.6666666665, ans=0.125 2023-11-23 13:17:45,214 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10600, loss[loss=0.07238, simple_loss=0.09669, pruned_loss=0.014, audio_tagging_loss=0.01003, over 14949.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09262, pruned_loss=0.01402, audio_tagging_loss=0.00903, over 3032328.82 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:17:55,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359300 2023-11-23 13:18:06,143 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.97 vs. limit=6.0 2023-11-23 13:18:25,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.12 vs. limit=15.0 2023-11-23 13:18:29,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2395466.6666666665, ans=0.0 2023-11-23 13:18:49,238 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10650, loss[loss=0.06485, simple_loss=0.09172, pruned_loss=0.01108, audio_tagging_loss=0.007912, over 14923.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09264, pruned_loss=0.01404, audio_tagging_loss=0.008979, over 3035451.68 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:18:53,997 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.219e+01 8.921e+01 9.796e+01 1.166e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 13:18:59,586 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359350 2023-11-23 13:19:07,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2395666.6666666665, ans=0.0 2023-11-23 13:19:13,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2395666.6666666665, ans=0.1 2023-11-23 13:19:40,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2395866.6666666665, ans=0.95 2023-11-23 13:19:50,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2395866.6666666665, ans=0.125 2023-11-23 13:19:52,825 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10700, loss[loss=0.07998, simple_loss=0.1075, pruned_loss=0.01731, audio_tagging_loss=0.008919, over 15500.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09283, pruned_loss=0.01401, audio_tagging_loss=0.008921, over 3045184.41 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:20:03,363 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359400 2023-11-23 13:20:04,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2396000.0, ans=0.125 2023-11-23 13:20:13,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0 2023-11-23 13:20:21,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2396066.6666666665, ans=0.0 2023-11-23 13:20:30,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2396133.3333333335, ans=0.1 2023-11-23 13:20:51,196 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:20:54,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2396200.0, ans=0.125 2023-11-23 13:20:54,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2396200.0, ans=0.0 2023-11-23 13:20:57,065 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10750, loss[loss=0.05563, simple_loss=0.07105, pruned_loss=0.01057, audio_tagging_loss=0.009537, over 15288.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09279, pruned_loss=0.01395, audio_tagging_loss=0.008974, over 3047573.19 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:21:02,309 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.422e+01 8.871e+01 9.738e+01 1.299e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-23 13:21:07,225 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359450 2023-11-23 13:21:07,376 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:21:14,922 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0 2023-11-23 13:22:01,022 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10800, loss[loss=0.05363, simple_loss=0.07141, pruned_loss=0.008834, audio_tagging_loss=0.009093, over 14715.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09262, pruned_loss=0.01383, audio_tagging_loss=0.008897, over 3052253.88 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:22:01,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2396600.0, ans=0.0 2023-11-23 13:22:11,216 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359500 2023-11-23 13:22:13,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.68 vs. limit=15.0 2023-11-23 13:22:13,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2396666.6666666665, ans=0.0 2023-11-23 13:22:18,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2396666.6666666665, ans=0.125 2023-11-23 13:22:24,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2396666.6666666665, ans=0.125 2023-11-23 13:22:26,133 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:22:38,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.00 vs. limit=15.0 2023-11-23 13:22:40,842 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.16 vs. limit=5.0 2023-11-23 13:23:03,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2396933.3333333335, ans=0.125 2023-11-23 13:23:04,155 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10850, loss[loss=0.05832, simple_loss=0.07311, pruned_loss=0.01187, audio_tagging_loss=0.009896, over 14317.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09292, pruned_loss=0.0139, audio_tagging_loss=0.008896, over 3050488.98 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:23:11,465 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.664e+01 8.074e+01 8.715e+01 9.516e+01 1.146e+02, threshold=1.743e+02, percent-clipped=0.0 2023-11-23 13:23:15,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359550 2023-11-23 13:23:29,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2397066.6666666665, ans=0.125 2023-11-23 13:23:34,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2397066.6666666665, ans=0.0 2023-11-23 13:23:39,619 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.51 vs. limit=22.5 2023-11-23 13:23:51,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2397133.3333333335, ans=0.0 2023-11-23 13:23:52,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2397133.3333333335, ans=0.1 2023-11-23 13:24:05,401 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 13:24:07,834 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10900, loss[loss=0.06677, simple_loss=0.09407, pruned_loss=0.0114, audio_tagging_loss=0.008334, over 15605.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09284, pruned_loss=0.0139, audio_tagging_loss=0.009051, over 3042559.27 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:24:18,503 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359600 2023-11-23 13:24:31,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2397333.3333333335, ans=0.125 2023-11-23 13:24:37,501 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:24:42,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2397400.0, ans=0.125 2023-11-23 13:24:55,168 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.92 vs. limit=15.0 2023-11-23 13:25:00,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2397533.3333333335, ans=0.125 2023-11-23 13:25:12,141 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 10950, loss[loss=0.05195, simple_loss=0.07143, pruned_loss=0.009278, audio_tagging_loss=0.006954, over 15301.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09222, pruned_loss=0.01384, audio_tagging_loss=0.009105, over 3044301.88 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:25:18,228 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.360e+01 9.047e+01 9.764e+01 1.279e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 13:25:22,009 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359650 2023-11-23 13:25:24,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2397666.6666666665, ans=0.125 2023-11-23 13:25:30,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2397666.6666666665, ans=0.0 2023-11-23 13:25:33,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2397666.6666666665, ans=0.1 2023-11-23 13:25:38,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2397733.3333333335, ans=0.125 2023-11-23 13:25:48,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2397733.3333333335, ans=0.125 2023-11-23 13:25:53,114 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:26:09,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2397866.6666666665, ans=0.0 2023-11-23 13:26:16,465 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11000, loss[loss=0.06673, simple_loss=0.09127, pruned_loss=0.01344, audio_tagging_loss=0.007645, over 14094.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.0912, pruned_loss=0.01373, audio_tagging_loss=0.00923, over 3050447.13 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:26:19,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2397933.3333333335, ans=0.125 2023-11-23 13:26:27,191 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359700 2023-11-23 13:26:27,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2397933.3333333335, ans=0.2 2023-11-23 13:26:27,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2397933.3333333335, ans=0.1 2023-11-23 13:26:28,868 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 13:26:44,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2398066.6666666665, ans=0.125 2023-11-23 13:27:22,168 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11050, loss[loss=0.07832, simple_loss=0.1043, pruned_loss=0.01627, audio_tagging_loss=0.009918, over 15145.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09172, pruned_loss=0.01373, audio_tagging_loss=0.009243, over 3057038.61 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:27:28,286 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.620e+01 9.222e+01 9.990e+01 1.274e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-23 13:27:29,838 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:27:32,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359750 2023-11-23 13:27:36,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2398333.3333333335, ans=0.125 2023-11-23 13:27:37,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2398333.3333333335, ans=0.125 2023-11-23 13:27:51,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2398400.0, ans=0.125 2023-11-23 13:28:17,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2398533.3333333335, ans=0.125 2023-11-23 13:28:27,277 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11100, loss[loss=0.06834, simple_loss=0.09665, pruned_loss=0.0141, audio_tagging_loss=0.005909, over 15943.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09099, pruned_loss=0.0136, audio_tagging_loss=0.009314, over 3054613.46 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:28:30,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2398600.0, ans=0.0 2023-11-23 13:28:37,338 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359800 2023-11-23 13:28:43,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2398666.6666666665, ans=0.0 2023-11-23 13:28:47,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2398666.6666666665, ans=0.1 2023-11-23 13:29:09,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2398800.0, ans=0.0 2023-11-23 13:29:13,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2398800.0, ans=0.0 2023-11-23 13:29:20,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2398866.6666666665, ans=0.1 2023-11-23 13:29:31,468 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11150, loss[loss=0.07449, simple_loss=0.09953, pruned_loss=0.01659, audio_tagging_loss=0.008134, over 16355.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.0908, pruned_loss=0.01353, audio_tagging_loss=0.009424, over 3059127.71 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:29:37,471 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.369e+01 8.812e+01 9.424e+01 1.136e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-23 13:29:38,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2398933.3333333335, ans=0.1 2023-11-23 13:29:41,407 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359850 2023-11-23 13:29:41,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2398933.3333333335, ans=0.0 2023-11-23 13:29:54,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=2399000.0, ans=6.0 2023-11-23 13:29:56,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2399066.6666666665, ans=0.125 2023-11-23 13:30:08,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2399066.6666666665, ans=0.125 2023-11-23 13:30:13,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2399133.3333333335, ans=0.2 2023-11-23 13:30:23,919 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:30:35,627 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11200, loss[loss=0.05157, simple_loss=0.05837, pruned_loss=0.00961, audio_tagging_loss=0.01277, over 14085.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09072, pruned_loss=0.01353, audio_tagging_loss=0.009474, over 3056796.97 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:30:37,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2399266.6666666665, ans=0.1 2023-11-23 13:30:43,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2399266.6666666665, ans=0.125 2023-11-23 13:30:46,064 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359900 2023-11-23 13:30:59,348 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.98 vs. limit=15.0 2023-11-23 13:31:22,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2399466.6666666665, ans=0.125 2023-11-23 13:31:35,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2399533.3333333335, ans=0.125 2023-11-23 13:31:37,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2399533.3333333335, ans=0.125 2023-11-23 13:31:39,965 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11250, loss[loss=0.06827, simple_loss=0.08237, pruned_loss=0.01789, audio_tagging_loss=0.009194, over 14613.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09116, pruned_loss=0.01362, audio_tagging_loss=0.009383, over 3050280.65 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:31:47,330 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.154e+01 8.449e+01 9.308e+01 1.008e+02 1.171e+02, threshold=1.862e+02, percent-clipped=0.0 2023-11-23 13:31:49,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 359950 2023-11-23 13:31:50,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2399600.0, ans=0.125 2023-11-23 13:31:52,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2399666.6666666665, ans=0.0 2023-11-23 13:31:54,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=22.5 2023-11-23 13:32:12,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=15.0 2023-11-23 13:32:35,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2399866.6666666665, ans=0.2 2023-11-23 13:32:43,481 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11300, loss[loss=0.07138, simple_loss=0.09948, pruned_loss=0.01476, audio_tagging_loss=0.006872, over 16366.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09141, pruned_loss=0.01384, audio_tagging_loss=0.009268, over 3054589.40 frames. ], batch size: 62, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:32:48,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2399933.3333333335, ans=0.0 2023-11-23 13:32:51,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2399933.3333333335, ans=0.0 2023-11-23 13:32:53,445 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360000 2023-11-23 13:33:08,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2400000.0, ans=0.125 2023-11-23 13:33:22,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2400066.6666666665, ans=0.125 2023-11-23 13:33:27,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2400133.3333333335, ans=0.125 2023-11-23 13:33:36,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2400200.0, ans=0.125 2023-11-23 13:33:36,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2400200.0, ans=0.0 2023-11-23 13:33:50,673 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11350, loss[loss=0.07352, simple_loss=0.1059, pruned_loss=0.0148, audio_tagging_loss=0.005783, over 16070.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09129, pruned_loss=0.01383, audio_tagging_loss=0.00906, over 3051036.76 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:33:58,000 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.206e+01 9.153e+01 9.834e+01 1.397e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-23 13:34:00,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360050 2023-11-23 13:34:07,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2400333.3333333335, ans=10.0 2023-11-23 13:34:09,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2400333.3333333335, ans=0.0 2023-11-23 13:34:29,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2400466.6666666665, ans=0.1 2023-11-23 13:34:34,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2400466.6666666665, ans=6.0 2023-11-23 13:34:54,323 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11400, loss[loss=0.07338, simple_loss=0.09639, pruned_loss=0.01592, audio_tagging_loss=0.009271, over 15564.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09197, pruned_loss=0.01408, audio_tagging_loss=0.00892, over 3043668.73 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:35:03,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2400600.0, ans=0.125 2023-11-23 13:35:04,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360100 2023-11-23 13:35:15,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2400666.6666666665, ans=0.1 2023-11-23 13:35:30,117 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.63 vs. limit=22.5 2023-11-23 13:35:39,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.77 vs. limit=10.0 2023-11-23 13:35:51,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2400866.6666666665, ans=0.1 2023-11-23 13:35:57,435 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11450, loss[loss=0.05207, simple_loss=0.06926, pruned_loss=0.008725, audio_tagging_loss=0.008721, over 14245.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09248, pruned_loss=0.01408, audio_tagging_loss=0.008916, over 3044721.16 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:36:02,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2400933.3333333335, ans=0.0 2023-11-23 13:36:04,744 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.159e+01 8.794e+01 9.452e+01 1.261e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-23 13:36:07,375 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360150 2023-11-23 13:36:18,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2401000.0, ans=0.2 2023-11-23 13:36:23,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2401066.6666666665, ans=0.0 2023-11-23 13:36:24,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2401066.6666666665, ans=0.125 2023-11-23 13:36:37,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2401133.3333333335, ans=0.0 2023-11-23 13:36:39,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2401133.3333333335, ans=0.125 2023-11-23 13:36:45,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2401133.3333333335, ans=0.1 2023-11-23 13:36:57,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2401200.0, ans=0.0 2023-11-23 13:37:01,740 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11500, loss[loss=0.04835, simple_loss=0.06393, pruned_loss=0.007917, audio_tagging_loss=0.008473, over 14827.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09221, pruned_loss=0.01392, audio_tagging_loss=0.008906, over 3046390.91 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:37:12,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360200 2023-11-23 13:37:17,383 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.25 vs. limit=10.0 2023-11-23 13:37:21,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2401333.3333333335, ans=0.1 2023-11-23 13:37:23,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2401333.3333333335, ans=0.0 2023-11-23 13:37:28,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2401400.0, ans=0.125 2023-11-23 13:37:38,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2401400.0, ans=0.125 2023-11-23 13:37:47,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2401466.6666666665, ans=0.125 2023-11-23 13:38:04,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.30 vs. limit=15.0 2023-11-23 13:38:06,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2401600.0, ans=0.125 2023-11-23 13:38:07,661 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11550, loss[loss=0.06455, simple_loss=0.08737, pruned_loss=0.01373, audio_tagging_loss=0.007133, over 14578.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09226, pruned_loss=0.01405, audio_tagging_loss=0.008986, over 3048550.86 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:38:15,049 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.476e+01 9.086e+01 9.753e+01 1.372e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-23 13:38:18,188 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360250 2023-11-23 13:38:26,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.56 vs. limit=15.0 2023-11-23 13:38:38,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2401733.3333333335, ans=0.125 2023-11-23 13:38:47,277 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 13:38:49,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2401800.0, ans=0.0 2023-11-23 13:39:01,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2401866.6666666665, ans=0.05 2023-11-23 13:39:05,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2401866.6666666665, ans=0.125 2023-11-23 13:39:09,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2401866.6666666665, ans=0.2 2023-11-23 13:39:11,793 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11600, loss[loss=0.07808, simple_loss=0.1083, pruned_loss=0.01492, audio_tagging_loss=0.008988, over 15468.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09245, pruned_loss=0.01399, audio_tagging_loss=0.009038, over 3051852.38 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:39:21,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360300 2023-11-23 13:40:00,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2402133.3333333335, ans=0.0 2023-11-23 13:40:14,609 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11650, loss[loss=0.07887, simple_loss=0.1128, pruned_loss=0.01635, audio_tagging_loss=0.006103, over 16453.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09131, pruned_loss=0.01382, audio_tagging_loss=0.009062, over 3047285.98 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:40:16,565 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.27 vs. limit=10.0 2023-11-23 13:40:19,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2402266.6666666665, ans=0.1 2023-11-23 13:40:22,543 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.635e+01 8.364e+01 9.300e+01 1.004e+02 1.361e+02, threshold=1.860e+02, percent-clipped=0.0 2023-11-23 13:40:25,054 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360350 2023-11-23 13:40:45,997 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0 2023-11-23 13:40:49,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2402400.0, ans=0.125 2023-11-23 13:41:01,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.08 vs. limit=15.0 2023-11-23 13:41:05,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2402533.3333333335, ans=0.0 2023-11-23 13:41:18,501 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11700, loss[loss=0.07168, simple_loss=0.1037, pruned_loss=0.01243, audio_tagging_loss=0.007386, over 16318.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.0912, pruned_loss=0.01382, audio_tagging_loss=0.009045, over 3039376.28 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:41:29,171 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360400 2023-11-23 13:41:29,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2402600.0, ans=0.1 2023-11-23 13:41:40,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2402666.6666666665, ans=0.2 2023-11-23 13:41:43,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2402733.3333333335, ans=0.0 2023-11-23 13:41:51,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2402733.3333333335, ans=0.2 2023-11-23 13:41:58,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2402800.0, ans=6.0 2023-11-23 13:42:05,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2402800.0, ans=0.015 2023-11-23 13:42:23,019 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11750, loss[loss=0.05817, simple_loss=0.06999, pruned_loss=0.01321, audio_tagging_loss=0.009966, over 14946.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09035, pruned_loss=0.01365, audio_tagging_loss=0.009167, over 3039435.12 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:42:32,128 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.747e+01 8.361e+01 8.957e+01 9.749e+01 1.339e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 13:42:33,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360450 2023-11-23 13:42:37,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2403000.0, ans=0.2 2023-11-23 13:42:37,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2403000.0, ans=0.125 2023-11-23 13:42:52,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2403066.6666666665, ans=0.0 2023-11-23 13:42:59,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2403066.6666666665, ans=0.125 2023-11-23 13:43:03,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.44 vs. limit=10.0 2023-11-23 13:43:05,697 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.23 vs. limit=6.0 2023-11-23 13:43:12,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2403200.0, ans=0.125 2023-11-23 13:43:14,147 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:43:26,979 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11800, loss[loss=0.07067, simple_loss=0.09442, pruned_loss=0.01278, audio_tagging_loss=0.01068, over 15107.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09185, pruned_loss=0.014, audio_tagging_loss=0.009089, over 3035826.80 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:43:31,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2403266.6666666665, ans=0.2 2023-11-23 13:43:34,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2403266.6666666665, ans=0.95 2023-11-23 13:43:36,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2403266.6666666665, ans=0.0 2023-11-23 13:43:37,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360500 2023-11-23 13:43:49,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2403333.3333333335, ans=0.0 2023-11-23 13:44:01,993 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.18 vs. limit=22.5 2023-11-23 13:44:04,638 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.47 vs. limit=15.0 2023-11-23 13:44:31,222 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11850, loss[loss=0.06488, simple_loss=0.09057, pruned_loss=0.01024, audio_tagging_loss=0.009361, over 15764.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09141, pruned_loss=0.01391, audio_tagging_loss=0.009168, over 3037381.45 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:44:40,400 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.325e+01 8.959e+01 9.729e+01 1.337e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-23 13:44:41,731 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360550 2023-11-23 13:44:46,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2403666.6666666665, ans=0.125 2023-11-23 13:44:48,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2403666.6666666665, ans=0.125 2023-11-23 13:45:00,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2403733.3333333335, ans=0.05 2023-11-23 13:45:18,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2403800.0, ans=0.1 2023-11-23 13:45:33,458 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.76 vs. limit=22.5 2023-11-23 13:45:35,379 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11900, loss[loss=0.07586, simple_loss=0.1042, pruned_loss=0.01561, audio_tagging_loss=0.00816, over 14679.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.0917, pruned_loss=0.01396, audio_tagging_loss=0.009247, over 3036694.49 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:45:35,872 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.06 vs. limit=15.0 2023-11-23 13:45:39,342 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 13:45:39,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2403933.3333333335, ans=0.125 2023-11-23 13:45:45,373 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360600 2023-11-23 13:45:52,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2404000.0, ans=0.125 2023-11-23 13:45:57,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2404000.0, ans=0.125 2023-11-23 13:46:05,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.74 vs. limit=10.0 2023-11-23 13:46:07,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2404066.6666666665, ans=0.1 2023-11-23 13:46:40,771 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 11950, loss[loss=0.05745, simple_loss=0.07344, pruned_loss=0.007549, audio_tagging_loss=0.01318, over 15803.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09104, pruned_loss=0.01388, audio_tagging_loss=0.009372, over 3040365.19 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0 2023-11-23 13:46:42,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2404266.6666666665, ans=0.07 2023-11-23 13:46:49,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2404266.6666666665, ans=0.0 2023-11-23 13:46:50,092 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.271e+01 8.961e+01 9.651e+01 1.560e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-23 13:46:51,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360650 2023-11-23 13:46:54,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2404333.3333333335, ans=0.125 2023-11-23 13:47:02,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2404333.3333333335, ans=0.125 2023-11-23 13:47:22,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2404466.6666666665, ans=0.0 2023-11-23 13:47:42,941 INFO [train_asr.py:1221] (3/4) Epoch 30, batch 12000, loss[loss=0.06291, simple_loss=0.07957, pruned_loss=0.01304, audio_tagging_loss=0.01008, over 15137.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09137, pruned_loss=0.01387, audio_tagging_loss=0.009421, over 3040179.01 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 32.0 2023-11-23 13:47:42,942 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 13:48:08,079 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.5968, 3.4664, 3.9343, 3.4786], device='cuda:3') 2023-11-23 13:48:25,615 INFO [train_asr.py:1253] (3/4) Epoch 30, validation: loss=0.05798, simple_loss=0.05115, pruned_loss=0.00515, audio_tagging_loss=0.02725, over 4681554.00 frames. 2023-11-23 13:48:25,616 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 13:48:35,657 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360700 2023-11-23 13:48:35,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2404600.0, ans=0.125 2023-11-23 13:48:35,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2404600.0, ans=0.125 2023-11-23 13:48:41,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2404666.6666666665, ans=0.1 2023-11-23 13:48:47,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2404666.6666666665, ans=0.125 2023-11-23 13:49:30,373 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 0, loss[loss=0.06683, simple_loss=0.07758, pruned_loss=0.009283, audio_tagging_loss=0.01876, over 14872.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.07758, pruned_loss=0.009283, audio_tagging_loss=0.01876, over 14872.00 frames. ], batch size: 57, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:49:30,374 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 13:50:05,321 INFO [train_asr.py:1253] (3/4) Epoch 31, validation: loss=0.05797, simple_loss=0.05105, pruned_loss=0.005059, audio_tagging_loss=0.02738, over 4681554.00 frames. 2023-11-23 13:50:05,322 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 13:50:41,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2404900.0, ans=0.125 2023-11-23 13:50:46,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2404966.6666666665, ans=0.125 2023-11-23 13:50:47,536 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.427e+01 8.784e+01 9.433e+01 1.048e+02 1.296e+02, threshold=1.887e+02, percent-clipped=0.0 2023-11-23 13:50:48,931 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360750 2023-11-23 13:50:54,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2404966.6666666665, ans=15.0 2023-11-23 13:50:56,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2405033.3333333335, ans=0.1 2023-11-23 13:51:10,467 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 50, loss[loss=0.06459, simple_loss=0.0714, pruned_loss=0.01004, audio_tagging_loss=0.01885, over 15521.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.0922, pruned_loss=0.0134, audio_tagging_loss=0.01754, over 689076.94 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:51:28,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=22.5 2023-11-23 13:51:34,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2405166.6666666665, ans=0.125 2023-11-23 13:51:38,909 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.62 vs. limit=15.0 2023-11-23 13:51:46,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2405233.3333333335, ans=0.0 2023-11-23 13:51:53,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360800 2023-11-23 13:51:59,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.91 vs. limit=10.0 2023-11-23 13:52:16,521 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 100, loss[loss=0.07966, simple_loss=0.1031, pruned_loss=0.01484, audio_tagging_loss=0.01329, over 16002.00 frames. ], tot_loss[loss=0.07693, simple_loss=0.09307, pruned_loss=0.01372, audio_tagging_loss=0.01667, over 1210745.99 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:52:16,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2405433.3333333335, ans=0.125 2023-11-23 13:52:18,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2405433.3333333335, ans=0.0 2023-11-23 13:52:25,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2405433.3333333335, ans=0.125 2023-11-23 13:52:45,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2405566.6666666665, ans=0.1 2023-11-23 13:52:57,690 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.885e+01 9.566e+01 1.092e+02 1.525e+02, threshold=1.913e+02, percent-clipped=0.0 2023-11-23 13:52:59,749 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360850 2023-11-23 13:52:59,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2405633.3333333335, ans=0.0 2023-11-23 13:53:01,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.31 vs. limit=15.0 2023-11-23 13:53:20,699 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 150, loss[loss=0.05424, simple_loss=0.06951, pruned_loss=0.006302, audio_tagging_loss=0.01318, over 14460.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09169, pruned_loss=0.01356, audio_tagging_loss=0.01505, over 1617118.20 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:53:58,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2405966.6666666665, ans=0.0 2023-11-23 13:54:01,619 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.21 vs. limit=6.0 2023-11-23 13:54:03,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360900 2023-11-23 13:54:03,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2405966.6666666665, ans=0.0 2023-11-23 13:54:15,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=12.0 2023-11-23 13:54:25,032 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 200, loss[loss=0.0925, simple_loss=0.1311, pruned_loss=0.01943, audio_tagging_loss=0.007526, over 16800.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09136, pruned_loss=0.0135, audio_tagging_loss=0.01338, over 1933484.32 frames. ], batch size: 60, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:55:06,204 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.350e+01 9.246e+01 9.961e+01 1.409e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-23 13:55:07,135 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.06 vs. limit=12.0 2023-11-23 13:55:07,602 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 360950 2023-11-23 13:55:13,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2406300.0, ans=0.0 2023-11-23 13:55:22,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2406366.6666666665, ans=0.2 2023-11-23 13:55:30,781 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 250, loss[loss=0.08531, simple_loss=0.1238, pruned_loss=0.01472, audio_tagging_loss=0.008679, over 15511.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.09192, pruned_loss=0.01372, audio_tagging_loss=0.01206, over 2180084.96 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:55:35,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.26 vs. limit=15.0 2023-11-23 13:55:39,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2406433.3333333335, ans=0.125 2023-11-23 13:55:45,063 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.81 vs. limit=15.0 2023-11-23 13:56:13,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361000 2023-11-23 13:56:16,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.31 vs. limit=10.0 2023-11-23 13:56:35,032 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 300, loss[loss=0.06416, simple_loss=0.08425, pruned_loss=0.01307, audio_tagging_loss=0.008963, over 14567.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09232, pruned_loss=0.01404, audio_tagging_loss=0.0112, over 2367338.33 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:56:45,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2406766.6666666665, ans=0.025 2023-11-23 13:56:46,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2406833.3333333335, ans=0.0 2023-11-23 13:56:51,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2406833.3333333335, ans=0.0 2023-11-23 13:56:54,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.59 vs. limit=15.0 2023-11-23 13:57:16,330 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.477e+01 9.124e+01 9.859e+01 1.340e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-23 13:57:17,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361050 2023-11-23 13:57:19,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2406966.6666666665, ans=0.125 2023-11-23 13:57:21,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2406966.6666666665, ans=0.0 2023-11-23 13:57:26,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2407033.3333333335, ans=0.0 2023-11-23 13:57:29,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2407033.3333333335, ans=0.125 2023-11-23 13:57:38,722 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 350, loss[loss=0.07173, simple_loss=0.1046, pruned_loss=0.0133, audio_tagging_loss=0.006131, over 15221.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09329, pruned_loss=0.014, audio_tagging_loss=0.0105, over 2517907.60 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:57:58,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2407166.6666666665, ans=0.035 2023-11-23 13:57:59,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2407166.6666666665, ans=0.2 2023-11-23 13:58:05,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2407233.3333333335, ans=0.0 2023-11-23 13:58:21,940 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361100 2023-11-23 13:58:44,379 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 400, loss[loss=0.07315, simple_loss=0.09622, pruned_loss=0.01728, audio_tagging_loss=0.007753, over 14776.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09437, pruned_loss=0.01435, audio_tagging_loss=0.01006, over 2641319.10 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 13:59:00,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2407500.0, ans=0.125 2023-11-23 13:59:05,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2407500.0, ans=0.125 2023-11-23 13:59:11,678 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.88 vs. limit=12.0 2023-11-23 13:59:26,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2407633.3333333335, ans=0.1 2023-11-23 13:59:27,042 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.381e+01 8.890e+01 9.709e+01 1.192e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-23 13:59:27,199 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361150 2023-11-23 13:59:29,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2407633.3333333335, ans=0.1 2023-11-23 13:59:49,036 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 450, loss[loss=0.07481, simple_loss=0.09748, pruned_loss=0.01625, audio_tagging_loss=0.009813, over 14199.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09249, pruned_loss=0.01394, audio_tagging_loss=0.009895, over 2723936.61 frames. ], batch size: 54, lr: 2.21e-03, grad_scale: 32.0 2023-11-23 14:00:01,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2407833.3333333335, ans=0.125 2023-11-23 14:00:11,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2407833.3333333335, ans=0.125 2023-11-23 14:00:16,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2407900.0, ans=0.125 2023-11-23 14:00:21,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2407900.0, ans=0.0 2023-11-23 14:00:27,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2407966.6666666665, ans=0.1 2023-11-23 14:00:31,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361200 2023-11-23 14:00:41,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2408033.3333333335, ans=0.0 2023-11-23 14:00:52,577 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 500, loss[loss=0.05526, simple_loss=0.08029, pruned_loss=0.008711, audio_tagging_loss=0.006403, over 15455.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09304, pruned_loss=0.01385, audio_tagging_loss=0.009663, over 2798264.88 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 8.0 2023-11-23 14:01:09,212 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.30 vs. limit=15.0 2023-11-23 14:01:12,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2408166.6666666665, ans=0.125 2023-11-23 14:01:17,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2408166.6666666665, ans=0.125 2023-11-23 14:01:24,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2408233.3333333335, ans=0.125 2023-11-23 14:01:35,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361250 2023-11-23 14:01:38,186 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.202e+01 8.738e+01 9.216e+01 1.113e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-23 14:01:40,472 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0 2023-11-23 14:01:45,063 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.38 vs. limit=22.5 2023-11-23 14:01:57,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2408433.3333333335, ans=0.125 2023-11-23 14:01:58,076 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 550, loss[loss=0.06674, simple_loss=0.09486, pruned_loss=0.01136, audio_tagging_loss=0.007956, over 16012.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09291, pruned_loss=0.01406, audio_tagging_loss=0.009525, over 2853058.12 frames. ], batch size: 60, lr: 2.21e-03, grad_scale: 8.0 2023-11-23 14:01:58,914 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.87 vs. limit=22.5 2023-11-23 14:02:09,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2408433.3333333335, ans=0.1 2023-11-23 14:02:18,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.37 vs. limit=15.0 2023-11-23 14:02:26,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2408566.6666666665, ans=0.125 2023-11-23 14:02:31,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2408566.6666666665, ans=0.0 2023-11-23 14:02:40,370 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361300 2023-11-23 14:02:51,494 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:03:02,859 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 600, loss[loss=0.07242, simple_loss=0.09585, pruned_loss=0.01414, audio_tagging_loss=0.01036, over 15012.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09321, pruned_loss=0.01405, audio_tagging_loss=0.009445, over 2888019.87 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 8.0 2023-11-23 14:03:20,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2408833.3333333335, ans=0.125 2023-11-23 14:03:34,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2408900.0, ans=0.04949747468305833 2023-11-23 14:03:38,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2408900.0, ans=0.125 2023-11-23 14:03:45,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361350 2023-11-23 14:03:46,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2408966.6666666665, ans=0.0 2023-11-23 14:03:48,041 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.522e+01 9.044e+01 1.001e+02 1.721e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 14:04:03,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.40 vs. limit=15.0 2023-11-23 14:04:06,503 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 650, loss[loss=0.06135, simple_loss=0.08497, pruned_loss=0.009207, audio_tagging_loss=0.009655, over 16191.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09407, pruned_loss=0.01422, audio_tagging_loss=0.009329, over 2922979.94 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 8.0 2023-11-23 14:04:15,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2409100.0, ans=0.0 2023-11-23 14:04:40,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2409233.3333333335, ans=0.125 2023-11-23 14:04:46,793 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.56 vs. limit=15.0 2023-11-23 14:04:49,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361400 2023-11-23 14:05:01,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2409366.6666666665, ans=0.0 2023-11-23 14:05:04,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2409366.6666666665, ans=0.0 2023-11-23 14:05:08,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.53 vs. limit=15.0 2023-11-23 14:05:12,024 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 700, loss[loss=0.05469, simple_loss=0.06636, pruned_loss=0.01016, audio_tagging_loss=0.01134, over 15001.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09443, pruned_loss=0.01428, audio_tagging_loss=0.009289, over 2957642.31 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 8.0 2023-11-23 14:05:18,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2409433.3333333335, ans=0.125 2023-11-23 14:05:25,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2409500.0, ans=0.0 2023-11-23 14:05:50,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2409633.3333333335, ans=0.0 2023-11-23 14:05:54,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361450 2023-11-23 14:05:57,076 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.383e+01 9.026e+01 9.852e+01 1.230e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 14:05:59,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2409633.3333333335, ans=0.125 2023-11-23 14:06:11,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.10 vs. limit=15.0 2023-11-23 14:06:17,525 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 750, loss[loss=0.06825, simple_loss=0.08739, pruned_loss=0.01522, audio_tagging_loss=0.009342, over 14860.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09334, pruned_loss=0.01412, audio_tagging_loss=0.009275, over 2978053.22 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 8.0 2023-11-23 14:06:20,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2409766.6666666665, ans=0.0 2023-11-23 14:06:25,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.91 vs. limit=10.0 2023-11-23 14:06:26,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2409766.6666666665, ans=0.0 2023-11-23 14:06:32,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2409833.3333333335, ans=0.125 2023-11-23 14:06:37,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2409833.3333333335, ans=0.0 2023-11-23 14:06:57,338 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:06:59,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361500 2023-11-23 14:07:09,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2410033.3333333335, ans=0.125 2023-11-23 14:07:12,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2410033.3333333335, ans=0.0 2023-11-23 14:07:17,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2410033.3333333335, ans=0.0 2023-11-23 14:07:21,102 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 800, loss[loss=0.05765, simple_loss=0.07203, pruned_loss=0.01129, audio_tagging_loss=0.01034, over 15122.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09352, pruned_loss=0.01412, audio_tagging_loss=0.009282, over 2991607.63 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:07:24,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2410100.0, ans=0.1 2023-11-23 14:07:34,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2410166.6666666665, ans=0.0 2023-11-23 14:07:35,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2410166.6666666665, ans=0.125 2023-11-23 14:08:04,605 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361550 2023-11-23 14:08:04,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2410300.0, ans=0.125 2023-11-23 14:08:06,976 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.503e+01 9.179e+01 9.718e+01 1.363e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-23 14:08:26,904 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 850, loss[loss=0.07983, simple_loss=0.1066, pruned_loss=0.01729, audio_tagging_loss=0.009233, over 14610.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.0931, pruned_loss=0.01414, audio_tagging_loss=0.009314, over 3008858.47 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:08:32,139 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:09:09,284 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361600 2023-11-23 14:09:32,510 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 900, loss[loss=0.07151, simple_loss=0.09004, pruned_loss=0.01562, audio_tagging_loss=0.01087, over 15721.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09291, pruned_loss=0.01404, audio_tagging_loss=0.009344, over 3015046.11 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:09:35,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2410766.6666666665, ans=0.1 2023-11-23 14:09:38,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2410766.6666666665, ans=0.1 2023-11-23 14:09:40,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2410766.6666666665, ans=0.125 2023-11-23 14:10:01,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2410900.0, ans=0.125 2023-11-23 14:10:15,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361650 2023-11-23 14:10:15,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2410966.6666666665, ans=0.125 2023-11-23 14:10:18,171 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.422e+01 9.031e+01 9.667e+01 1.145e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-23 14:10:22,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2410966.6666666665, ans=0.125 2023-11-23 14:10:36,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2411100.0, ans=0.125 2023-11-23 14:10:37,461 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 950, loss[loss=0.0593, simple_loss=0.0744, pruned_loss=0.01159, audio_tagging_loss=0.01051, over 14228.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.0931, pruned_loss=0.01403, audio_tagging_loss=0.00929, over 3020316.93 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:10:54,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2411166.6666666665, ans=0.0 2023-11-23 14:11:12,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2411233.3333333335, ans=0.125 2023-11-23 14:11:19,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361700 2023-11-23 14:11:31,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2411366.6666666665, ans=0.2 2023-11-23 14:11:41,909 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1000, loss[loss=0.07441, simple_loss=0.09758, pruned_loss=0.01697, audio_tagging_loss=0.008652, over 14450.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09202, pruned_loss=0.0138, audio_tagging_loss=0.009109, over 3015683.24 frames. ], batch size: 54, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:11:47,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2411433.3333333335, ans=0.125 2023-11-23 14:11:48,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2411433.3333333335, ans=0.0 2023-11-23 14:12:08,907 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 14:12:17,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2411566.6666666665, ans=15.0 2023-11-23 14:12:24,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361750 2023-11-23 14:12:26,485 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.336e+01 9.096e+01 9.576e+01 1.309e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-23 14:12:29,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2411633.3333333335, ans=10.0 2023-11-23 14:12:36,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2411700.0, ans=0.125 2023-11-23 14:12:39,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2411700.0, ans=0.125 2023-11-23 14:12:44,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2411700.0, ans=0.09899494936611666 2023-11-23 14:12:47,008 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1050, loss[loss=0.04294, simple_loss=0.04904, pruned_loss=0.007755, audio_tagging_loss=0.01067, over 16717.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.0928, pruned_loss=0.01385, audio_tagging_loss=0.008985, over 3025462.33 frames. ], batch size: 66, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:12:56,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2411766.6666666665, ans=0.125 2023-11-23 14:13:05,541 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:13:10,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2411833.3333333335, ans=0.0 2023-11-23 14:13:30,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361800 2023-11-23 14:13:34,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2411966.6666666665, ans=0.0 2023-11-23 14:13:51,456 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1100, loss[loss=0.08394, simple_loss=0.1179, pruned_loss=0.01737, audio_tagging_loss=0.007641, over 14703.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09288, pruned_loss=0.01382, audio_tagging_loss=0.008908, over 3030086.29 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:13:53,969 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 14:13:57,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2412100.0, ans=0.125 2023-11-23 14:14:14,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2412166.6666666665, ans=0.125 2023-11-23 14:14:16,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2412233.3333333335, ans=0.0 2023-11-23 14:14:22,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2412233.3333333335, ans=0.0 2023-11-23 14:14:34,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361850 2023-11-23 14:14:37,265 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.240e+01 8.744e+01 9.286e+01 1.135e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-23 14:14:41,805 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.61 vs. limit=22.5 2023-11-23 14:14:56,298 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1150, loss[loss=0.06109, simple_loss=0.08421, pruned_loss=0.01223, audio_tagging_loss=0.00676, over 14879.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09237, pruned_loss=0.01381, audio_tagging_loss=0.00891, over 3035672.85 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 16.0 2023-11-23 14:14:56,935 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.01 vs. limit=15.0 2023-11-23 14:15:05,535 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.40 vs. limit=15.0 2023-11-23 14:15:06,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2412433.3333333335, ans=0.1 2023-11-23 14:15:30,969 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:15:39,862 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361900 2023-11-23 14:15:57,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2412700.0, ans=0.1 2023-11-23 14:16:02,451 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1200, loss[loss=0.06077, simple_loss=0.08147, pruned_loss=0.01134, audio_tagging_loss=0.008687, over 15015.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09239, pruned_loss=0.01381, audio_tagging_loss=0.008891, over 3046338.93 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:16:24,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2412833.3333333335, ans=0.0 2023-11-23 14:16:35,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2412900.0, ans=0.0 2023-11-23 14:16:44,777 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 361950 2023-11-23 14:16:48,317 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.465e+01 9.389e+01 1.006e+02 1.334e+02, threshold=1.878e+02, percent-clipped=0.0 2023-11-23 14:16:54,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2413033.3333333335, ans=0.125 2023-11-23 14:17:00,737 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.69 vs. limit=15.0 2023-11-23 14:17:06,330 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1250, loss[loss=0.07041, simple_loss=0.0923, pruned_loss=0.01531, audio_tagging_loss=0.008953, over 15082.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09176, pruned_loss=0.0137, audio_tagging_loss=0.00901, over 3047139.19 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:17:06,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=14.60 vs. limit=15.0 2023-11-23 14:17:16,540 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:17:47,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2413300.0, ans=0.125 2023-11-23 14:17:50,036 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362000 2023-11-23 14:18:05,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2413366.6666666665, ans=0.125 2023-11-23 14:18:11,322 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1300, loss[loss=0.07632, simple_loss=0.106, pruned_loss=0.01579, audio_tagging_loss=0.007517, over 14749.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09249, pruned_loss=0.01377, audio_tagging_loss=0.008921, over 3043064.15 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:18:11,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2413433.3333333335, ans=0.125 2023-11-23 14:18:14,995 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0 2023-11-23 14:18:18,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2413433.3333333335, ans=0.0 2023-11-23 14:18:23,493 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5 2023-11-23 14:18:28,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2413500.0, ans=0.0 2023-11-23 14:18:35,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2413500.0, ans=0.07 2023-11-23 14:18:38,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.19 vs. limit=6.0 2023-11-23 14:18:45,598 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0 2023-11-23 14:18:54,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362050 2023-11-23 14:18:58,460 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.540e+01 8.227e+01 8.895e+01 9.502e+01 1.940e+02, threshold=1.779e+02, percent-clipped=1.0 2023-11-23 14:19:17,511 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1350, loss[loss=0.04792, simple_loss=0.0541, pruned_loss=0.009429, audio_tagging_loss=0.01144, over 14552.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09301, pruned_loss=0.014, audio_tagging_loss=0.008825, over 3046851.82 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:19:17,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2413766.6666666665, ans=0.125 2023-11-23 14:19:19,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2413766.6666666665, ans=0.0 2023-11-23 14:19:29,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.93 vs. limit=22.5 2023-11-23 14:19:42,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2413900.0, ans=0.0 2023-11-23 14:19:43,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2413900.0, ans=0.1 2023-11-23 14:19:45,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2413900.0, ans=15.0 2023-11-23 14:19:51,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2413900.0, ans=0.125 2023-11-23 14:20:01,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362100 2023-11-23 14:20:04,686 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 14:20:12,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2414033.3333333335, ans=0.0 2023-11-23 14:20:22,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.78 vs. limit=15.0 2023-11-23 14:20:22,794 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1400, loss[loss=0.0746, simple_loss=0.09518, pruned_loss=0.01795, audio_tagging_loss=0.009063, over 15246.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.0928, pruned_loss=0.01406, audio_tagging_loss=0.008918, over 3045694.33 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:20:28,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.58 vs. limit=12.0 2023-11-23 14:20:31,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2414100.0, ans=0.5 2023-11-23 14:20:38,240 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.69 vs. limit=15.0 2023-11-23 14:20:56,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.82 vs. limit=22.5 2023-11-23 14:20:57,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2414233.3333333335, ans=0.2 2023-11-23 14:21:06,253 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362150 2023-11-23 14:21:09,859 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.379e+01 8.906e+01 9.568e+01 1.474e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-23 14:21:27,363 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1450, loss[loss=0.07591, simple_loss=0.1064, pruned_loss=0.01666, audio_tagging_loss=0.006058, over 15958.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09233, pruned_loss=0.01402, audio_tagging_loss=0.009043, over 3045672.79 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:21:28,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2414433.3333333335, ans=0.1 2023-11-23 14:21:33,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.20 vs. limit=15.0 2023-11-23 14:21:58,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2414566.6666666665, ans=0.125 2023-11-23 14:22:04,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2414566.6666666665, ans=0.2 2023-11-23 14:22:10,951 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362200 2023-11-23 14:22:34,372 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1500, loss[loss=0.05786, simple_loss=0.07134, pruned_loss=0.01019, audio_tagging_loss=0.012, over 15058.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09251, pruned_loss=0.01412, audio_tagging_loss=0.009101, over 3041969.87 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:22:41,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2414766.6666666665, ans=10.0 2023-11-23 14:22:50,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.34 vs. limit=15.0 2023-11-23 14:22:58,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.99 vs. limit=22.5 2023-11-23 14:23:07,758 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.10 vs. limit=15.0 2023-11-23 14:23:09,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2414900.0, ans=0.0 2023-11-23 14:23:16,548 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362250 2023-11-23 14:23:20,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2414966.6666666665, ans=0.125 2023-11-23 14:23:20,759 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.572e+01 9.255e+01 1.007e+02 1.352e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-23 14:23:38,723 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1550, loss[loss=0.0532, simple_loss=0.06463, pruned_loss=0.01157, audio_tagging_loss=0.00931, over 13887.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09267, pruned_loss=0.01417, audio_tagging_loss=0.009156, over 3046182.04 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:24:03,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2415233.3333333335, ans=0.125 2023-11-23 14:24:05,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2415233.3333333335, ans=0.0 2023-11-23 14:24:22,278 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362300 2023-11-23 14:24:42,971 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1600, loss[loss=0.06047, simple_loss=0.07973, pruned_loss=0.01175, audio_tagging_loss=0.008849, over 14799.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09271, pruned_loss=0.0141, audio_tagging_loss=0.009226, over 3048147.44 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:24:50,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2415433.3333333335, ans=0.125 2023-11-23 14:24:56,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2415500.0, ans=0.5 2023-11-23 14:25:06,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2415500.0, ans=0.2 2023-11-23 14:25:26,559 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362350 2023-11-23 14:25:30,066 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.201e+01 8.953e+01 9.579e+01 1.488e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 14:25:48,517 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1650, loss[loss=0.06609, simple_loss=0.08134, pruned_loss=0.01584, audio_tagging_loss=0.009577, over 13514.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09199, pruned_loss=0.01386, audio_tagging_loss=0.009329, over 3039109.60 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:26:14,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.69 vs. limit=22.5 2023-11-23 14:26:27,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2415966.6666666665, ans=0.125 2023-11-23 14:26:31,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362400 2023-11-23 14:26:53,728 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1700, loss[loss=0.06503, simple_loss=0.07994, pruned_loss=0.01264, audio_tagging_loss=0.01242, over 15446.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09272, pruned_loss=0.01388, audio_tagging_loss=0.009324, over 3047520.99 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:27:01,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2416100.0, ans=0.1 2023-11-23 14:27:02,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2416100.0, ans=0.125 2023-11-23 14:27:02,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2416100.0, ans=0.125 2023-11-23 14:27:14,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2416166.6666666665, ans=0.0 2023-11-23 14:27:20,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2416233.3333333335, ans=0.125 2023-11-23 14:27:21,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2416233.3333333335, ans=0.0 2023-11-23 14:27:28,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2416233.3333333335, ans=0.1 2023-11-23 14:27:29,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2416233.3333333335, ans=0.125 2023-11-23 14:27:36,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362450 2023-11-23 14:27:40,063 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.211e+01 9.052e+01 9.581e+01 1.125e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 14:27:46,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2416366.6666666665, ans=0.125 2023-11-23 14:27:57,283 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1750, loss[loss=0.07214, simple_loss=0.09448, pruned_loss=0.01576, audio_tagging_loss=0.009142, over 16693.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09191, pruned_loss=0.01372, audio_tagging_loss=0.009252, over 3043994.82 frames. ], batch size: 63, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:28:00,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2416433.3333333335, ans=0.0 2023-11-23 14:28:01,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2416433.3333333335, ans=0.1 2023-11-23 14:28:09,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2416500.0, ans=0.0 2023-11-23 14:28:21,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2416500.0, ans=0.1 2023-11-23 14:28:40,523 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362500 2023-11-23 14:28:53,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.76 vs. limit=22.5 2023-11-23 14:29:02,035 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1800, loss[loss=0.08032, simple_loss=0.1115, pruned_loss=0.01775, audio_tagging_loss=0.006827, over 14864.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09266, pruned_loss=0.01398, audio_tagging_loss=0.009085, over 3047986.35 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:29:22,062 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.19 vs. limit=22.5 2023-11-23 14:29:44,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362550 2023-11-23 14:29:49,002 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 8.278e+01 8.801e+01 9.362e+01 1.432e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-23 14:29:49,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2416966.6666666665, ans=0.125 2023-11-23 14:30:08,002 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1850, loss[loss=0.05461, simple_loss=0.07886, pruned_loss=0.006343, audio_tagging_loss=0.008836, over 13560.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.0922, pruned_loss=0.01375, audio_tagging_loss=0.009062, over 3046679.84 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:30:17,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2417100.0, ans=0.0 2023-11-23 14:30:43,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2417233.3333333335, ans=0.07 2023-11-23 14:30:50,699 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362600 2023-11-23 14:30:52,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2417300.0, ans=0.0 2023-11-23 14:31:12,193 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1900, loss[loss=0.05417, simple_loss=0.07029, pruned_loss=0.01006, audio_tagging_loss=0.008969, over 15181.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09189, pruned_loss=0.01368, audio_tagging_loss=0.008965, over 3054073.48 frames. ], batch size: 60, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:31:18,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2417433.3333333335, ans=0.125 2023-11-23 14:31:26,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2417500.0, ans=0.0 2023-11-23 14:31:54,873 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362650 2023-11-23 14:31:58,353 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.275e+01 8.858e+01 9.597e+01 1.368e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 14:32:03,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2417700.0, ans=0.0 2023-11-23 14:32:16,153 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 1950, loss[loss=0.06694, simple_loss=0.09109, pruned_loss=0.01098, audio_tagging_loss=0.01041, over 15167.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09279, pruned_loss=0.01388, audio_tagging_loss=0.008933, over 3056853.33 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:32:20,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2417766.6666666665, ans=0.125 2023-11-23 14:32:22,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2417766.6666666665, ans=0.0 2023-11-23 14:32:53,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.08 vs. limit=10.0 2023-11-23 14:32:57,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2417966.6666666665, ans=0.0 2023-11-23 14:32:58,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362700 2023-11-23 14:33:02,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.60 vs. limit=15.0 2023-11-23 14:33:13,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2418033.3333333335, ans=0.95 2023-11-23 14:33:20,876 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2000, loss[loss=0.07818, simple_loss=0.104, pruned_loss=0.01687, audio_tagging_loss=0.009302, over 14657.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09209, pruned_loss=0.01378, audio_tagging_loss=0.008939, over 3050708.33 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:33:31,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2418100.0, ans=0.125 2023-11-23 14:34:01,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2418300.0, ans=0.125 2023-11-23 14:34:03,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362750 2023-11-23 14:34:10,256 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.308e+01 8.872e+01 9.432e+01 1.250e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-23 14:34:12,178 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.56 vs. limit=22.5 2023-11-23 14:34:17,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.86 vs. limit=15.0 2023-11-23 14:34:25,999 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2050, loss[loss=0.06749, simple_loss=0.0925, pruned_loss=0.01349, audio_tagging_loss=0.007744, over 14853.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09172, pruned_loss=0.01366, audio_tagging_loss=0.008967, over 3054157.90 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:34:28,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2418433.3333333335, ans=0.125 2023-11-23 14:34:38,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2418500.0, ans=0.125 2023-11-23 14:34:45,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2418500.0, ans=0.125 2023-11-23 14:35:09,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362800 2023-11-23 14:35:16,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2418633.3333333335, ans=0.125 2023-11-23 14:35:18,217 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.23 vs. limit=15.0 2023-11-23 14:35:31,106 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2100, loss[loss=0.06766, simple_loss=0.09347, pruned_loss=0.009631, audio_tagging_loss=0.0113, over 14048.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.0917, pruned_loss=0.01367, audio_tagging_loss=0.008983, over 3050764.34 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:35:37,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2418766.6666666665, ans=0.125 2023-11-23 14:35:44,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2418833.3333333335, ans=0.125 2023-11-23 14:35:47,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2418833.3333333335, ans=0.125 2023-11-23 14:35:48,163 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0 2023-11-23 14:35:49,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2418833.3333333335, ans=0.125 2023-11-23 14:35:52,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0 2023-11-23 14:35:54,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2418833.3333333335, ans=0.0 2023-11-23 14:35:56,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2418900.0, ans=0.2 2023-11-23 14:35:59,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2418900.0, ans=0.125 2023-11-23 14:36:07,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2418900.0, ans=0.125 2023-11-23 14:36:07,741 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.47 vs. limit=15.0 2023-11-23 14:36:11,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2418966.6666666665, ans=0.1 2023-11-23 14:36:14,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362850 2023-11-23 14:36:14,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2418966.6666666665, ans=0.125 2023-11-23 14:36:20,838 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 8.402e+01 8.985e+01 9.676e+01 1.115e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 14:36:31,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2419033.3333333335, ans=0.125 2023-11-23 14:36:32,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2419033.3333333335, ans=0.0 2023-11-23 14:36:36,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2419100.0, ans=0.0 2023-11-23 14:36:37,683 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2150, loss[loss=0.07763, simple_loss=0.1019, pruned_loss=0.01688, audio_tagging_loss=0.00978, over 15441.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09184, pruned_loss=0.0138, audio_tagging_loss=0.009042, over 3045055.52 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:37:14,945 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 14:37:20,602 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362900 2023-11-23 14:37:38,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.60 vs. limit=8.0 2023-11-23 14:37:42,014 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2200, loss[loss=0.06894, simple_loss=0.08465, pruned_loss=0.01726, audio_tagging_loss=0.009362, over 14960.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09263, pruned_loss=0.01399, audio_tagging_loss=0.009018, over 3043622.03 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:37:45,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2419433.3333333335, ans=10.0 2023-11-23 14:37:45,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2419433.3333333335, ans=0.0 2023-11-23 14:38:05,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2419500.0, ans=0.1 2023-11-23 14:38:10,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2419566.6666666665, ans=0.125 2023-11-23 14:38:11,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2419566.6666666665, ans=0.125 2023-11-23 14:38:19,828 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.49 vs. limit=15.0 2023-11-23 14:38:22,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2419633.3333333335, ans=10.0 2023-11-23 14:38:25,463 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 362950 2023-11-23 14:38:25,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2419633.3333333335, ans=0.0 2023-11-23 14:38:29,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2419633.3333333335, ans=0.0 2023-11-23 14:38:31,618 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.445e+01 9.122e+01 9.777e+01 1.344e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-23 14:38:33,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2419700.0, ans=0.125 2023-11-23 14:38:47,341 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2250, loss[loss=0.04585, simple_loss=0.05849, pruned_loss=0.007402, audio_tagging_loss=0.009202, over 14619.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09307, pruned_loss=0.01401, audio_tagging_loss=0.009104, over 3038880.76 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:38:49,189 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.85 vs. limit=15.0 2023-11-23 14:38:50,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2419766.6666666665, ans=0.2 2023-11-23 14:39:00,062 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:39:18,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2419900.0, ans=0.0 2023-11-23 14:39:31,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363000 2023-11-23 14:39:40,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2420033.3333333335, ans=0.2 2023-11-23 14:39:47,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2420033.3333333335, ans=0.0 2023-11-23 14:39:54,477 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2300, loss[loss=0.07205, simple_loss=0.09838, pruned_loss=0.01309, audio_tagging_loss=0.009773, over 15154.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09349, pruned_loss=0.01411, audio_tagging_loss=0.009155, over 3041974.42 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:39:56,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2420100.0, ans=0.125 2023-11-23 14:40:36,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363050 2023-11-23 14:40:43,600 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.427e+01 9.145e+01 9.809e+01 1.490e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-23 14:40:49,918 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 14:40:52,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2420366.6666666665, ans=0.0 2023-11-23 14:40:52,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2420366.6666666665, ans=0.125 2023-11-23 14:40:56,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2420366.6666666665, ans=0.125 2023-11-23 14:40:58,420 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2350, loss[loss=0.0647, simple_loss=0.07891, pruned_loss=0.01472, audio_tagging_loss=0.01053, over 13930.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09255, pruned_loss=0.01399, audio_tagging_loss=0.009304, over 3044224.88 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:41:03,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2420433.3333333335, ans=0.125 2023-11-23 14:41:17,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2420500.0, ans=0.125 2023-11-23 14:41:18,349 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.68 vs. limit=22.5 2023-11-23 14:41:25,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2420566.6666666665, ans=0.0 2023-11-23 14:41:40,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2420633.3333333335, ans=0.035 2023-11-23 14:41:41,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363100 2023-11-23 14:41:41,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2420633.3333333335, ans=0.125 2023-11-23 14:41:41,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2420633.3333333335, ans=0.2 2023-11-23 14:41:44,549 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.66 vs. limit=22.5 2023-11-23 14:41:54,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2420700.0, ans=0.125 2023-11-23 14:41:56,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2420700.0, ans=0.0 2023-11-23 14:42:00,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2420700.0, ans=0.04949747468305833 2023-11-23 14:42:02,811 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2400, loss[loss=0.07192, simple_loss=0.09005, pruned_loss=0.01326, audio_tagging_loss=0.01363, over 16214.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.0928, pruned_loss=0.0141, audio_tagging_loss=0.009368, over 3042931.42 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:42:22,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2420833.3333333335, ans=0.2 2023-11-23 14:42:25,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2420833.3333333335, ans=0.125 2023-11-23 14:42:27,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.57 vs. limit=10.0 2023-11-23 14:42:34,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2420900.0, ans=0.125 2023-11-23 14:42:45,457 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363150 2023-11-23 14:42:51,923 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.436e+01 9.019e+01 9.718e+01 1.173e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 14:43:03,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2421033.3333333335, ans=0.2 2023-11-23 14:43:07,843 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2450, loss[loss=0.06018, simple_loss=0.08036, pruned_loss=0.01091, audio_tagging_loss=0.009089, over 14867.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09319, pruned_loss=0.01408, audio_tagging_loss=0.00931, over 3049569.81 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:43:46,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.43 vs. limit=15.0 2023-11-23 14:43:50,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363200 2023-11-23 14:43:57,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2421300.0, ans=0.2 2023-11-23 14:44:11,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2421433.3333333335, ans=0.0 2023-11-23 14:44:12,624 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2500, loss[loss=0.09296, simple_loss=0.1323, pruned_loss=0.01886, audio_tagging_loss=0.007966, over 15251.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09277, pruned_loss=0.01385, audio_tagging_loss=0.009304, over 3047669.61 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:44:15,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2421433.3333333335, ans=0.125 2023-11-23 14:44:26,541 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:44:56,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363250 2023-11-23 14:45:02,005 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.211e+01 9.083e+01 9.843e+01 1.373e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-23 14:45:06,142 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:45:17,038 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2550, loss[loss=0.09676, simple_loss=0.1361, pruned_loss=0.02031, audio_tagging_loss=0.008413, over 17037.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09379, pruned_loss=0.01396, audio_tagging_loss=0.009188, over 3049791.13 frames. ], batch size: 62, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:45:18,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2421766.6666666665, ans=0.125 2023-11-23 14:45:18,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.15 vs. limit=15.0 2023-11-23 14:45:42,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2421833.3333333335, ans=0.0 2023-11-23 14:45:53,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2421900.0, ans=0.125 2023-11-23 14:45:54,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2421900.0, ans=0.0 2023-11-23 14:46:00,370 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363300 2023-11-23 14:46:11,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2422033.3333333335, ans=0.125 2023-11-23 14:46:21,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2422100.0, ans=0.0 2023-11-23 14:46:22,966 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2600, loss[loss=0.05865, simple_loss=0.08196, pruned_loss=0.008455, audio_tagging_loss=0.009216, over 14379.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09277, pruned_loss=0.01393, audio_tagging_loss=0.009128, over 3046385.03 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:46:34,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2422100.0, ans=0.0 2023-11-23 14:46:41,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2422166.6666666665, ans=0.125 2023-11-23 14:46:45,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2422166.6666666665, ans=0.125 2023-11-23 14:46:45,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.48 vs. limit=15.0 2023-11-23 14:47:05,793 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363350 2023-11-23 14:47:05,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2422300.0, ans=0.125 2023-11-23 14:47:12,905 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.370e+01 9.064e+01 9.837e+01 1.240e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-23 14:47:23,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2422366.6666666665, ans=0.125 2023-11-23 14:47:28,330 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2650, loss[loss=0.09489, simple_loss=0.1334, pruned_loss=0.02127, audio_tagging_loss=0.006924, over 14636.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09287, pruned_loss=0.01382, audio_tagging_loss=0.009044, over 3041248.17 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:47:31,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2422433.3333333335, ans=0.125 2023-11-23 14:47:53,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.75 vs. limit=22.5 2023-11-23 14:48:11,213 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363400 2023-11-23 14:48:28,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2422700.0, ans=0.0 2023-11-23 14:48:32,436 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2700, loss[loss=0.07412, simple_loss=0.09897, pruned_loss=0.01699, audio_tagging_loss=0.007643, over 14625.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09257, pruned_loss=0.01383, audio_tagging_loss=0.009009, over 3043044.78 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:48:32,741 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:49:07,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2422900.0, ans=0.09899494936611666 2023-11-23 14:49:15,911 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363450 2023-11-23 14:49:23,351 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.399e+01 8.847e+01 9.605e+01 1.237e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 14:49:23,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2423033.3333333335, ans=0.125 2023-11-23 14:49:24,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.60 vs. limit=22.5 2023-11-23 14:49:31,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2423033.3333333335, ans=0.125 2023-11-23 14:49:37,801 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2750, loss[loss=0.07535, simple_loss=0.1056, pruned_loss=0.01357, audio_tagging_loss=0.008979, over 14137.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09236, pruned_loss=0.01381, audio_tagging_loss=0.00896, over 3046446.65 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:49:46,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2423100.0, ans=0.05 2023-11-23 14:49:46,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2423100.0, ans=0.1 2023-11-23 14:50:20,867 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363500 2023-11-23 14:50:29,687 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.12 vs. limit=15.0 2023-11-23 14:50:33,624 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 14:50:43,986 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2800, loss[loss=0.0619, simple_loss=0.07644, pruned_loss=0.01258, audio_tagging_loss=0.01111, over 16149.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09193, pruned_loss=0.01377, audio_tagging_loss=0.008985, over 3051927.83 frames. ], batch size: 64, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 14:50:44,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2423433.3333333335, ans=0.125 2023-11-23 14:50:44,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2423433.3333333335, ans=0.125 2023-11-23 14:51:21,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2423633.3333333335, ans=0.0 2023-11-23 14:51:27,475 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363550 2023-11-23 14:51:30,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2423633.3333333335, ans=0.0 2023-11-23 14:51:36,079 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.140e+01 8.811e+01 9.422e+01 1.418e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-23 14:51:48,472 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2850, loss[loss=0.09029, simple_loss=0.1232, pruned_loss=0.02139, audio_tagging_loss=0.007305, over 15534.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09123, pruned_loss=0.01349, audio_tagging_loss=0.009066, over 3045253.34 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:52:06,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2423833.3333333335, ans=0.2 2023-11-23 14:52:29,627 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.75 vs. limit=15.0 2023-11-23 14:52:31,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363600 2023-11-23 14:52:31,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2423966.6666666665, ans=0.125 2023-11-23 14:52:34,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2423966.6666666665, ans=0.1 2023-11-23 14:52:48,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.11 vs. limit=15.0 2023-11-23 14:52:52,488 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2900, loss[loss=0.06392, simple_loss=0.08898, pruned_loss=0.01238, audio_tagging_loss=0.007053, over 15699.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09104, pruned_loss=0.01359, audio_tagging_loss=0.009131, over 3050628.24 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:52:59,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=22.5 2023-11-23 14:53:06,510 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:53:11,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.17 vs. limit=22.5 2023-11-23 14:53:35,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363650 2023-11-23 14:53:37,765 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.41 vs. limit=15.0 2023-11-23 14:53:44,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.203e+01 8.514e+01 9.320e+01 9.888e+01 1.457e+02, threshold=1.864e+02, percent-clipped=0.0 2023-11-23 14:53:59,191 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 2950, loss[loss=0.05551, simple_loss=0.07443, pruned_loss=0.009842, audio_tagging_loss=0.00845, over 15370.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09142, pruned_loss=0.01363, audio_tagging_loss=0.009143, over 3046710.90 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:54:10,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2424500.0, ans=0.125 2023-11-23 14:54:23,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2424566.6666666665, ans=0.2 2023-11-23 14:54:41,451 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363700 2023-11-23 14:54:55,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.41 vs. limit=8.0 2023-11-23 14:55:03,648 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3000, loss[loss=0.07506, simple_loss=0.09784, pruned_loss=0.01919, audio_tagging_loss=0.006951, over 15130.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.0922, pruned_loss=0.01369, audio_tagging_loss=0.00911, over 3047032.18 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:55:03,649 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 14:55:45,739 INFO [train_asr.py:1253] (3/4) Epoch 31, validation: loss=0.0577, simple_loss=0.05103, pruned_loss=0.005016, audio_tagging_loss=0.02717, over 4681554.00 frames. 2023-11-23 14:55:45,740 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 14:56:17,704 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.43 vs. limit=15.0 2023-11-23 14:56:26,368 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.27 vs. limit=6.0 2023-11-23 14:56:28,229 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363750 2023-11-23 14:56:29,941 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.25 vs. limit=12.0 2023-11-23 14:56:32,790 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 14:56:36,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2425033.3333333335, ans=15.0 2023-11-23 14:56:37,295 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.570e+01 9.368e+01 9.997e+01 1.561e+02, threshold=1.874e+02, percent-clipped=0.0 2023-11-23 14:56:50,888 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3050, loss[loss=0.0871, simple_loss=0.1157, pruned_loss=0.0201, audio_tagging_loss=0.009127, over 14202.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09223, pruned_loss=0.01378, audio_tagging_loss=0.009143, over 3042709.97 frames. ], batch size: 51, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:56:59,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2425100.0, ans=15.0 2023-11-23 14:57:02,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2425166.6666666665, ans=0.125 2023-11-23 14:57:11,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2425166.6666666665, ans=0.2 2023-11-23 14:57:22,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2425233.3333333335, ans=0.125 2023-11-23 14:57:25,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2425233.3333333335, ans=0.125 2023-11-23 14:57:26,218 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.29 vs. limit=15.0 2023-11-23 14:57:28,000 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 14:57:32,437 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.64 vs. limit=15.0 2023-11-23 14:57:34,131 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363800 2023-11-23 14:57:56,165 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3100, loss[loss=0.0603, simple_loss=0.08602, pruned_loss=0.008103, audio_tagging_loss=0.009186, over 15270.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.0935, pruned_loss=0.01387, audio_tagging_loss=0.009098, over 3047342.78 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:58:14,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2425500.0, ans=0.125 2023-11-23 14:58:24,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2425566.6666666665, ans=0.5 2023-11-23 14:58:37,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2425633.3333333335, ans=0.125 2023-11-23 14:58:39,611 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363850 2023-11-23 14:58:43,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2425633.3333333335, ans=0.125 2023-11-23 14:58:43,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2425633.3333333335, ans=0.125 2023-11-23 14:58:47,919 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.127e+01 8.546e+01 8.908e+01 9.738e+01 1.357e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 14:59:00,948 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3150, loss[loss=0.06605, simple_loss=0.0835, pruned_loss=0.0136, audio_tagging_loss=0.0107, over 15014.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09308, pruned_loss=0.014, audio_tagging_loss=0.009181, over 3044523.06 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 14:59:03,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2425766.6666666665, ans=10.0 2023-11-23 14:59:19,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2425833.3333333335, ans=0.0 2023-11-23 14:59:32,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2425900.0, ans=0.125 2023-11-23 14:59:34,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2425900.0, ans=10.0 2023-11-23 14:59:37,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2425900.0, ans=0.1 2023-11-23 14:59:43,591 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363900 2023-11-23 14:59:45,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2425966.6666666665, ans=0.125 2023-11-23 14:59:56,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2426033.3333333335, ans=0.0 2023-11-23 15:00:06,086 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3200, loss[loss=0.0576, simple_loss=0.0765, pruned_loss=0.0105, audio_tagging_loss=0.008851, over 15188.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09331, pruned_loss=0.01403, audio_tagging_loss=0.009295, over 3043926.79 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:00:13,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2426100.0, ans=0.125 2023-11-23 15:00:15,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2426100.0, ans=0.07 2023-11-23 15:00:18,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2426166.6666666665, ans=0.0 2023-11-23 15:00:20,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2426166.6666666665, ans=0.0 2023-11-23 15:00:39,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2426233.3333333335, ans=0.1 2023-11-23 15:00:49,081 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 363950 2023-11-23 15:00:58,116 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.380e+01 9.015e+01 9.537e+01 1.251e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-23 15:01:07,043 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:01:11,016 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3250, loss[loss=0.06827, simple_loss=0.09823, pruned_loss=0.01118, audio_tagging_loss=0.007972, over 15572.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09266, pruned_loss=0.01392, audio_tagging_loss=0.009353, over 3042817.43 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:01:12,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2426433.3333333335, ans=0.0 2023-11-23 15:01:21,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2426433.3333333335, ans=0.1 2023-11-23 15:01:37,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2426566.6666666665, ans=0.1 2023-11-23 15:01:47,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2426566.6666666665, ans=0.2 2023-11-23 15:01:53,574 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364000 2023-11-23 15:01:53,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2426633.3333333335, ans=0.0 2023-11-23 15:01:54,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2426633.3333333335, ans=0.125 2023-11-23 15:02:00,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2426633.3333333335, ans=0.1 2023-11-23 15:02:01,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2426633.3333333335, ans=0.1 2023-11-23 15:02:18,581 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3300, loss[loss=0.07821, simple_loss=0.1046, pruned_loss=0.01417, audio_tagging_loss=0.01172, over 15900.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09251, pruned_loss=0.01391, audio_tagging_loss=0.009342, over 3041454.08 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:02:21,864 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:02:26,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2426766.6666666665, ans=0.125 2023-11-23 15:02:29,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2426766.6666666665, ans=0.04949747468305833 2023-11-23 15:02:34,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2426833.3333333335, ans=0.0 2023-11-23 15:02:34,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.59 vs. limit=15.0 2023-11-23 15:02:41,778 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.75 vs. limit=15.0 2023-11-23 15:02:43,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2426900.0, ans=0.125 2023-11-23 15:03:00,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2426966.6666666665, ans=0.2 2023-11-23 15:03:01,210 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364050 2023-11-23 15:03:10,313 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.350e+01 9.048e+01 9.771e+01 1.182e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 15:03:13,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2427033.3333333335, ans=0.2 2023-11-23 15:03:23,932 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3350, loss[loss=0.07108, simple_loss=0.09143, pruned_loss=0.01597, audio_tagging_loss=0.009398, over 14608.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09232, pruned_loss=0.01402, audio_tagging_loss=0.009279, over 3045263.60 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:03:43,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2427166.6666666665, ans=0.125 2023-11-23 15:03:45,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2427166.6666666665, ans=0.5 2023-11-23 15:04:04,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2427300.0, ans=0.125 2023-11-23 15:04:06,606 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364100 2023-11-23 15:04:27,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2427433.3333333335, ans=0.125 2023-11-23 15:04:28,240 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3400, loss[loss=0.06006, simple_loss=0.08277, pruned_loss=0.01043, audio_tagging_loss=0.008247, over 15106.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09241, pruned_loss=0.01399, audio_tagging_loss=0.009134, over 3042563.15 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:04:29,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2427433.3333333335, ans=0.0 2023-11-23 15:04:41,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2427500.0, ans=0.04949747468305833 2023-11-23 15:04:54,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2427566.6666666665, ans=0.0 2023-11-23 15:05:11,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364150 2023-11-23 15:05:13,870 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:05:17,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2427633.3333333335, ans=0.125 2023-11-23 15:05:20,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2427700.0, ans=0.2 2023-11-23 15:05:21,246 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.657e+01 9.005e+01 9.760e+01 1.355e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 15:05:23,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2427700.0, ans=0.0 2023-11-23 15:05:32,854 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3450, loss[loss=0.08626, simple_loss=0.1109, pruned_loss=0.02072, audio_tagging_loss=0.01007, over 15378.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09325, pruned_loss=0.01415, audio_tagging_loss=0.009016, over 3049268.18 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:05:59,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2427900.0, ans=0.0 2023-11-23 15:06:12,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2427966.6666666665, ans=0.125 2023-11-23 15:06:15,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2427966.6666666665, ans=0.0 2023-11-23 15:06:16,709 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364200 2023-11-23 15:06:21,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2427966.6666666665, ans=0.05 2023-11-23 15:06:39,850 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3500, loss[loss=0.08052, simple_loss=0.1151, pruned_loss=0.01667, audio_tagging_loss=0.006315, over 15259.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09395, pruned_loss=0.01417, audio_tagging_loss=0.008942, over 3050317.95 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:06:49,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.71 vs. limit=15.0 2023-11-23 15:07:10,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2428233.3333333335, ans=0.125 2023-11-23 15:07:11,438 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 15:07:21,995 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364250 2023-11-23 15:07:32,827 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.333e+01 8.735e+01 9.330e+01 1.206e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-23 15:07:43,896 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3550, loss[loss=0.07021, simple_loss=0.09262, pruned_loss=0.01483, audio_tagging_loss=0.009068, over 15306.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09274, pruned_loss=0.01413, audio_tagging_loss=0.008916, over 3041362.21 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:07:57,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2428500.0, ans=0.0 2023-11-23 15:07:58,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2428500.0, ans=0.125 2023-11-23 15:07:58,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2428500.0, ans=0.2 2023-11-23 15:08:01,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2428500.0, ans=0.0 2023-11-23 15:08:10,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2428566.6666666665, ans=0.125 2023-11-23 15:08:12,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2428566.6666666665, ans=0.0 2023-11-23 15:08:27,305 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364300 2023-11-23 15:08:41,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2428700.0, ans=0.125 2023-11-23 15:08:41,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2428700.0, ans=0.2 2023-11-23 15:08:49,094 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3600, loss[loss=0.0606, simple_loss=0.08022, pruned_loss=0.01109, audio_tagging_loss=0.009396, over 14093.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09294, pruned_loss=0.01411, audio_tagging_loss=0.008869, over 3050222.30 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:08:51,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2428766.6666666665, ans=0.1 2023-11-23 15:09:26,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2428966.6666666665, ans=0.125 2023-11-23 15:09:31,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364350 2023-11-23 15:09:33,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2428966.6666666665, ans=0.0 2023-11-23 15:09:42,099 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.038e+01 8.312e+01 8.847e+01 9.562e+01 1.157e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-23 15:09:54,402 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3650, loss[loss=0.07959, simple_loss=0.1067, pruned_loss=0.01731, audio_tagging_loss=0.008923, over 15293.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09406, pruned_loss=0.01439, audio_tagging_loss=0.00886, over 3042776.59 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:09:58,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2429100.0, ans=0.1 2023-11-23 15:10:36,903 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364400 2023-11-23 15:10:57,789 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.51 vs. limit=15.0 2023-11-23 15:10:59,375 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3700, loss[loss=0.08893, simple_loss=0.1112, pruned_loss=0.02328, audio_tagging_loss=0.01005, over 15275.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09431, pruned_loss=0.0144, audio_tagging_loss=0.008849, over 3048458.20 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:11:00,212 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0 2023-11-23 15:11:09,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2429433.3333333335, ans=0.125 2023-11-23 15:11:27,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2429566.6666666665, ans=0.125 2023-11-23 15:11:40,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2429633.3333333335, ans=0.125 2023-11-23 15:11:42,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364450 2023-11-23 15:11:52,583 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.697e+01 8.537e+01 9.041e+01 9.821e+01 1.313e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-23 15:12:03,713 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3750, loss[loss=0.06888, simple_loss=0.08522, pruned_loss=0.01428, audio_tagging_loss=0.01199, over 14827.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09456, pruned_loss=0.01437, audio_tagging_loss=0.008935, over 3050555.16 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:12:11,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2429766.6666666665, ans=0.09899494936611666 2023-11-23 15:12:16,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2429833.3333333335, ans=0.0 2023-11-23 15:12:23,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2429833.3333333335, ans=0.0 2023-11-23 15:12:24,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2429833.3333333335, ans=0.125 2023-11-23 15:12:32,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2429900.0, ans=0.1 2023-11-23 15:12:46,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364500 2023-11-23 15:12:47,574 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 15:13:06,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.20 vs. limit=10.0 2023-11-23 15:13:08,473 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3800, loss[loss=0.06449, simple_loss=0.07548, pruned_loss=0.01275, audio_tagging_loss=0.01401, over 13684.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09362, pruned_loss=0.01421, audio_tagging_loss=0.009027, over 3047510.87 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:13:08,745 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:13:08,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2430100.0, ans=0.1 2023-11-23 15:13:18,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2430100.0, ans=0.125 2023-11-23 15:13:47,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.89 vs. limit=15.0 2023-11-23 15:13:50,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364550 2023-11-23 15:13:57,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2430300.0, ans=0.125 2023-11-23 15:14:03,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2430366.6666666665, ans=0.125 2023-11-23 15:14:04,561 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.445e+01 8.986e+01 9.684e+01 1.334e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 15:14:10,485 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.21 vs. limit=12.0 2023-11-23 15:14:14,495 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3850, loss[loss=0.06691, simple_loss=0.08779, pruned_loss=0.01281, audio_tagging_loss=0.0102, over 14551.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09208, pruned_loss=0.01398, audio_tagging_loss=0.009138, over 3032697.90 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:14:41,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2430566.6666666665, ans=0.125 2023-11-23 15:14:41,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2430566.6666666665, ans=0.1 2023-11-23 15:14:48,838 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.24 vs. limit=15.0 2023-11-23 15:14:57,806 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364600 2023-11-23 15:15:18,956 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3900, loss[loss=0.05428, simple_loss=0.06971, pruned_loss=0.009909, audio_tagging_loss=0.009514, over 14220.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09252, pruned_loss=0.01406, audio_tagging_loss=0.009106, over 3034683.76 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:15:34,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2430833.3333333335, ans=0.2 2023-11-23 15:15:39,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2430833.3333333335, ans=0.125 2023-11-23 15:16:02,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364650 2023-11-23 15:16:14,753 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.954e+01 8.382e+01 9.163e+01 9.741e+01 1.346e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-23 15:16:17,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2431033.3333333335, ans=0.1 2023-11-23 15:16:23,939 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 3950, loss[loss=0.07531, simple_loss=0.1036, pruned_loss=0.01486, audio_tagging_loss=0.008636, over 16597.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09267, pruned_loss=0.01402, audio_tagging_loss=0.009233, over 3034815.68 frames. ], batch size: 63, lr: 2.20e-03, grad_scale: 8.0 2023-11-23 15:16:38,196 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:16:45,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2431166.6666666665, ans=0.125 2023-11-23 15:16:46,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2431166.6666666665, ans=0.125 2023-11-23 15:16:46,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2431166.6666666665, ans=0.125 2023-11-23 15:16:55,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2431233.3333333335, ans=0.1 2023-11-23 15:16:59,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2431233.3333333335, ans=0.2 2023-11-23 15:17:03,582 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.69 vs. limit=15.0 2023-11-23 15:17:06,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364700 2023-11-23 15:17:20,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.45 vs. limit=22.5 2023-11-23 15:17:30,008 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4000, loss[loss=0.07234, simple_loss=0.1006, pruned_loss=0.01263, audio_tagging_loss=0.009418, over 15171.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.0915, pruned_loss=0.01386, audio_tagging_loss=0.009295, over 3025440.77 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:17:44,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2431500.0, ans=0.125 2023-11-23 15:17:47,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2431500.0, ans=0.5 2023-11-23 15:17:58,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2431566.6666666665, ans=0.125 2023-11-23 15:18:09,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.15 vs. limit=15.0 2023-11-23 15:18:12,464 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.33 vs. limit=10.0 2023-11-23 15:18:12,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364750 2023-11-23 15:18:24,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2431700.0, ans=0.0 2023-11-23 15:18:25,117 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.345e+01 8.888e+01 9.598e+01 2.808e+02, threshold=1.778e+02, percent-clipped=1.0 2023-11-23 15:18:27,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2431700.0, ans=0.015 2023-11-23 15:18:32,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.22 vs. limit=15.0 2023-11-23 15:18:32,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2431766.6666666665, ans=0.125 2023-11-23 15:18:33,729 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4050, loss[loss=0.07253, simple_loss=0.08733, pruned_loss=0.01649, audio_tagging_loss=0.01238, over 13898.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09273, pruned_loss=0.01394, audio_tagging_loss=0.009283, over 3035463.33 frames. ], batch size: 52, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:18:36,216 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 15:18:40,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2431766.6666666665, ans=0.2 2023-11-23 15:19:04,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2431900.0, ans=0.2 2023-11-23 15:19:16,199 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364800 2023-11-23 15:19:37,453 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4100, loss[loss=0.08037, simple_loss=0.1176, pruned_loss=0.01245, audio_tagging_loss=0.009125, over 15874.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09376, pruned_loss=0.01401, audio_tagging_loss=0.009211, over 3040524.32 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:19:43,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2432100.0, ans=0.125 2023-11-23 15:19:55,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2432166.6666666665, ans=0.04949747468305833 2023-11-23 15:20:16,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2432300.0, ans=0.125 2023-11-23 15:20:17,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2432300.0, ans=0.025 2023-11-23 15:20:19,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364850 2023-11-23 15:20:32,695 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.349e+01 8.595e+01 9.054e+01 9.904e+01 2.115e+02, threshold=1.811e+02, percent-clipped=1.0 2023-11-23 15:20:35,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2432366.6666666665, ans=0.035 2023-11-23 15:20:43,288 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4150, loss[loss=0.07574, simple_loss=0.09545, pruned_loss=0.01975, audio_tagging_loss=0.008268, over 14111.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.0945, pruned_loss=0.01437, audio_tagging_loss=0.008996, over 3049064.16 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:21:08,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2432566.6666666665, ans=0.125 2023-11-23 15:21:10,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2432566.6666666665, ans=0.0 2023-11-23 15:21:25,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364900 2023-11-23 15:21:29,316 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 15:21:39,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.01 vs. limit=22.5 2023-11-23 15:21:45,065 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:21:47,305 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4200, loss[loss=0.08386, simple_loss=0.1151, pruned_loss=0.01723, audio_tagging_loss=0.009091, over 15544.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09359, pruned_loss=0.01404, audio_tagging_loss=0.008922, over 3048806.83 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:21:47,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2432766.6666666665, ans=0.2 2023-11-23 15:21:56,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2432766.6666666665, ans=0.125 2023-11-23 15:22:30,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 364950 2023-11-23 15:22:42,430 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.314e+01 9.166e+01 9.730e+01 1.323e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-23 15:22:51,148 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4250, loss[loss=0.06672, simple_loss=0.09369, pruned_loss=0.01334, audio_tagging_loss=0.006537, over 15017.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09374, pruned_loss=0.01397, audio_tagging_loss=0.008976, over 3043667.82 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:23:08,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2433166.6666666665, ans=0.1 2023-11-23 15:23:13,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.97 vs. limit=15.0 2023-11-23 15:23:21,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=2433233.3333333335, ans=0.02 2023-11-23 15:23:30,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2433300.0, ans=0.125 2023-11-23 15:23:33,575 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365000 2023-11-23 15:23:50,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.29 vs. limit=22.5 2023-11-23 15:23:56,035 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4300, loss[loss=0.05342, simple_loss=0.0652, pruned_loss=0.0124, audio_tagging_loss=0.008414, over 15713.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.0938, pruned_loss=0.01403, audio_tagging_loss=0.00892, over 3045712.33 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:24:12,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2433500.0, ans=0.0 2023-11-23 15:24:20,240 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:24:21,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2433566.6666666665, ans=0.1 2023-11-23 15:24:25,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2433566.6666666665, ans=0.0 2023-11-23 15:24:38,097 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365050 2023-11-23 15:24:50,780 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.539e+01 9.202e+01 9.861e+01 1.335e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-23 15:24:52,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2433700.0, ans=0.1 2023-11-23 15:25:00,106 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4350, loss[loss=0.08786, simple_loss=0.1222, pruned_loss=0.01647, audio_tagging_loss=0.01028, over 15275.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09416, pruned_loss=0.01409, audio_tagging_loss=0.009029, over 3046581.45 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0 2023-11-23 15:25:06,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2433766.6666666665, ans=0.125 2023-11-23 15:25:12,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2433833.3333333335, ans=0.2 2023-11-23 15:25:13,740 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:25:31,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2433900.0, ans=0.125 2023-11-23 15:25:42,417 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365100 2023-11-23 15:26:03,789 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4400, loss[loss=0.062, simple_loss=0.08472, pruned_loss=0.01102, audio_tagging_loss=0.008621, over 15398.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.0935, pruned_loss=0.01397, audio_tagging_loss=0.008901, over 3044142.78 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:26:29,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2434233.3333333335, ans=0.0 2023-11-23 15:26:35,788 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.95 vs. limit=15.0 2023-11-23 15:26:42,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2434300.0, ans=0.1 2023-11-23 15:26:46,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365150 2023-11-23 15:26:47,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.25 vs. limit=15.0 2023-11-23 15:26:49,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2434300.0, ans=0.1 2023-11-23 15:26:55,786 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:26:59,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2434366.6666666665, ans=0.0 2023-11-23 15:26:59,942 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.264e+01 9.016e+01 9.697e+01 1.171e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-23 15:27:09,432 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4450, loss[loss=0.07655, simple_loss=0.1055, pruned_loss=0.01519, audio_tagging_loss=0.008627, over 15198.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09159, pruned_loss=0.01382, audio_tagging_loss=0.008925, over 3046375.30 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0 2023-11-23 15:27:26,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.44 vs. limit=15.0 2023-11-23 15:27:31,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2434500.0, ans=0.125 2023-11-23 15:27:37,239 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.53 vs. limit=12.0 2023-11-23 15:27:44,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.53 vs. limit=10.0 2023-11-23 15:27:45,911 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.12 vs. limit=10.0 2023-11-23 15:27:51,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365200 2023-11-23 15:27:58,707 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.72 vs. limit=22.5 2023-11-23 15:28:03,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2434700.0, ans=0.1 2023-11-23 15:28:07,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2434700.0, ans=0.125 2023-11-23 15:28:14,263 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4500, loss[loss=0.0525, simple_loss=0.06623, pruned_loss=0.009353, audio_tagging_loss=0.01004, over 15453.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09197, pruned_loss=0.01373, audio_tagging_loss=0.008833, over 3049677.28 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:28:14,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2434766.6666666665, ans=0.125 2023-11-23 15:28:20,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2434766.6666666665, ans=0.125 2023-11-23 15:28:56,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2434966.6666666665, ans=0.07 2023-11-23 15:28:57,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365250 2023-11-23 15:29:01,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2434966.6666666665, ans=0.125 2023-11-23 15:29:11,961 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.432e+01 8.316e+01 8.938e+01 9.967e+01 1.364e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-23 15:29:19,397 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4550, loss[loss=0.06355, simple_loss=0.08286, pruned_loss=0.01439, audio_tagging_loss=0.007738, over 14772.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09228, pruned_loss=0.01377, audio_tagging_loss=0.008909, over 3049862.43 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:29:32,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2435166.6666666665, ans=0.125 2023-11-23 15:29:36,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2435166.6666666665, ans=0.1 2023-11-23 15:29:46,429 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:29:49,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2435233.3333333335, ans=0.07 2023-11-23 15:29:54,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2435233.3333333335, ans=0.0 2023-11-23 15:30:03,039 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365300 2023-11-23 15:30:09,137 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 15:30:13,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2435366.6666666665, ans=0.0 2023-11-23 15:30:17,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2435366.6666666665, ans=0.125 2023-11-23 15:30:24,791 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4600, loss[loss=0.05644, simple_loss=0.05795, pruned_loss=0.01388, audio_tagging_loss=0.01359, over 14760.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09221, pruned_loss=0.01373, audio_tagging_loss=0.008954, over 3047699.90 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:30:30,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2435433.3333333335, ans=0.125 2023-11-23 15:30:46,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2435500.0, ans=0.0 2023-11-23 15:31:08,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365350 2023-11-23 15:31:22,441 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.583e+01 9.120e+01 9.726e+01 1.437e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-23 15:31:29,883 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4650, loss[loss=0.09137, simple_loss=0.1338, pruned_loss=0.01741, audio_tagging_loss=0.007047, over 16708.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09178, pruned_loss=0.01372, audio_tagging_loss=0.00907, over 3045092.76 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:31:33,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2435766.6666666665, ans=0.125 2023-11-23 15:31:33,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2435766.6666666665, ans=0.1 2023-11-23 15:32:12,775 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365400 2023-11-23 15:32:26,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2436033.3333333335, ans=0.0 2023-11-23 15:32:26,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2436033.3333333335, ans=0.2 2023-11-23 15:32:33,926 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4700, loss[loss=0.07344, simple_loss=0.09013, pruned_loss=0.01654, audio_tagging_loss=0.01183, over 16313.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09125, pruned_loss=0.01373, audio_tagging_loss=0.009139, over 3050267.28 frames. ], batch size: 62, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:32:34,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2436100.0, ans=0.125 2023-11-23 15:32:37,597 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.86 vs. limit=22.5 2023-11-23 15:32:38,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2436100.0, ans=0.0 2023-11-23 15:32:43,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2436100.0, ans=0.125 2023-11-23 15:32:53,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.11 vs. limit=15.0 2023-11-23 15:33:01,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2436233.3333333335, ans=0.0 2023-11-23 15:33:17,200 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365450 2023-11-23 15:33:31,147 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.245e+01 8.834e+01 9.507e+01 1.184e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 15:33:32,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2436366.6666666665, ans=0.0 2023-11-23 15:33:39,246 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4750, loss[loss=0.06957, simple_loss=0.09464, pruned_loss=0.01281, audio_tagging_loss=0.009444, over 14467.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09124, pruned_loss=0.01367, audio_tagging_loss=0.009193, over 3047240.41 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:34:03,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2436500.0, ans=0.0 2023-11-23 15:34:22,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365500 2023-11-23 15:34:44,612 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4800, loss[loss=0.07694, simple_loss=0.1021, pruned_loss=0.01524, audio_tagging_loss=0.01064, over 14806.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.0913, pruned_loss=0.01362, audio_tagging_loss=0.009236, over 3051364.44 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 15:34:44,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2436766.6666666665, ans=0.2 2023-11-23 15:35:19,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2436900.0, ans=0.0 2023-11-23 15:35:22,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.76 vs. limit=12.0 2023-11-23 15:35:27,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365550 2023-11-23 15:35:42,122 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.333e+01 8.912e+01 9.620e+01 1.211e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 15:35:48,324 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4850, loss[loss=0.07015, simple_loss=0.09761, pruned_loss=0.01222, audio_tagging_loss=0.009125, over 15214.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09171, pruned_loss=0.01362, audio_tagging_loss=0.009307, over 3046976.72 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:36:31,585 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365600 2023-11-23 15:36:44,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2437366.6666666665, ans=0.0 2023-11-23 15:36:52,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2437433.3333333335, ans=0.1 2023-11-23 15:36:53,499 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4900, loss[loss=0.06593, simple_loss=0.09169, pruned_loss=0.0143, audio_tagging_loss=0.005779, over 14550.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09211, pruned_loss=0.01381, audio_tagging_loss=0.00928, over 3046647.99 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:37:35,599 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365650 2023-11-23 15:37:53,798 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.240e+01 8.743e+01 9.594e+01 1.319e+02, threshold=1.749e+02, percent-clipped=0.0 2023-11-23 15:37:58,822 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 4950, loss[loss=0.0693, simple_loss=0.1, pruned_loss=0.01127, audio_tagging_loss=0.008025, over 15170.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09141, pruned_loss=0.01356, audio_tagging_loss=0.00922, over 3040375.82 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 15:38:10,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2437833.3333333335, ans=0.07 2023-11-23 15:38:14,361 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.40 vs. limit=22.5 2023-11-23 15:38:38,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2437966.6666666665, ans=0.125 2023-11-23 15:38:41,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365700 2023-11-23 15:38:48,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.64 vs. limit=15.0 2023-11-23 15:38:51,209 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=13.63 vs. limit=15.0 2023-11-23 15:38:51,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2438033.3333333335, ans=0.1 2023-11-23 15:38:56,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2438033.3333333335, ans=0.125 2023-11-23 15:39:02,598 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5000, loss[loss=0.06677, simple_loss=0.09158, pruned_loss=0.01188, audio_tagging_loss=0.009096, over 16607.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09114, pruned_loss=0.01357, audio_tagging_loss=0.009081, over 3042310.78 frames. ], batch size: 64, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 15:39:19,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2438166.6666666665, ans=0.0 2023-11-23 15:39:38,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2438233.3333333335, ans=0.0 2023-11-23 15:39:45,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365750 2023-11-23 15:39:47,488 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.22 vs. limit=12.0 2023-11-23 15:39:57,655 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.11 vs. limit=22.5 2023-11-23 15:40:01,798 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.229e+01 8.801e+01 9.337e+01 1.229e+02, threshold=1.760e+02, percent-clipped=0.0 2023-11-23 15:40:07,588 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5050, loss[loss=0.05893, simple_loss=0.07577, pruned_loss=0.01055, audio_tagging_loss=0.01049, over 14737.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09187, pruned_loss=0.01374, audio_tagging_loss=0.008936, over 3043704.66 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 15:40:14,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2438433.3333333335, ans=0.125 2023-11-23 15:40:44,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2438633.3333333335, ans=0.125 2023-11-23 15:40:49,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365800 2023-11-23 15:40:52,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2438633.3333333335, ans=0.95 2023-11-23 15:41:04,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2438700.0, ans=0.125 2023-11-23 15:41:12,967 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5100, loss[loss=0.0801, simple_loss=0.1092, pruned_loss=0.01585, audio_tagging_loss=0.009665, over 15616.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09172, pruned_loss=0.01376, audio_tagging_loss=0.008867, over 3042106.97 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 15:41:24,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2438833.3333333335, ans=0.1 2023-11-23 15:41:30,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2438833.3333333335, ans=0.0 2023-11-23 15:41:37,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2438900.0, ans=0.0 2023-11-23 15:41:53,831 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365850 2023-11-23 15:41:53,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2438966.6666666665, ans=0.125 2023-11-23 15:41:56,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2438966.6666666665, ans=0.125 2023-11-23 15:42:01,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2438966.6666666665, ans=0.125 2023-11-23 15:42:11,232 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.471e+01 8.600e+01 9.200e+01 9.982e+01 4.087e+02, threshold=1.840e+02, percent-clipped=1.0 2023-11-23 15:42:14,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2439033.3333333335, ans=0.5 2023-11-23 15:42:16,218 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5150, loss[loss=0.08581, simple_loss=0.1213, pruned_loss=0.01835, audio_tagging_loss=0.006819, over 15479.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09087, pruned_loss=0.01367, audio_tagging_loss=0.008914, over 3042899.92 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 15:42:22,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2439100.0, ans=0.1 2023-11-23 15:42:50,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2439233.3333333335, ans=0.2 2023-11-23 15:42:59,317 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365900 2023-11-23 15:43:20,543 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5200, loss[loss=0.06605, simple_loss=0.09276, pruned_loss=0.01276, audio_tagging_loss=0.006907, over 15377.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09121, pruned_loss=0.01374, audio_tagging_loss=0.008971, over 3047286.73 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:43:54,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2439566.6666666665, ans=0.125 2023-11-23 15:44:04,189 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 365950 2023-11-23 15:44:21,109 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.221e+01 8.985e+01 9.483e+01 1.522e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 15:44:27,300 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5250, loss[loss=0.07237, simple_loss=0.09611, pruned_loss=0.01732, audio_tagging_loss=0.006997, over 14445.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09168, pruned_loss=0.01357, audio_tagging_loss=0.008827, over 3049732.78 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:44:44,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2439833.3333333335, ans=0.05 2023-11-23 15:44:55,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2439900.0, ans=0.125 2023-11-23 15:45:09,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366000 2023-11-23 15:45:27,909 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.42 vs. limit=15.0 2023-11-23 15:45:32,562 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 15:45:33,378 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5300, loss[loss=0.06236, simple_loss=0.0801, pruned_loss=0.01293, audio_tagging_loss=0.009377, over 14590.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09153, pruned_loss=0.01355, audio_tagging_loss=0.008853, over 3048218.71 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:45:43,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2440100.0, ans=0.2 2023-11-23 15:46:05,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2440233.3333333335, ans=0.2 2023-11-23 15:46:17,188 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366050 2023-11-23 15:46:30,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=2440366.6666666665, ans=0.02 2023-11-23 15:46:32,949 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.490e+01 8.834e+01 9.644e+01 1.833e+02, threshold=1.767e+02, percent-clipped=1.0 2023-11-23 15:46:37,995 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5350, loss[loss=0.06954, simple_loss=0.09553, pruned_loss=0.01201, audio_tagging_loss=0.009771, over 16034.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09251, pruned_loss=0.01378, audio_tagging_loss=0.008832, over 3041040.30 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:46:43,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.74 vs. limit=15.0 2023-11-23 15:46:49,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2440500.0, ans=0.5 2023-11-23 15:47:12,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.63 vs. limit=15.0 2023-11-23 15:47:14,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2440566.6666666665, ans=0.07 2023-11-23 15:47:21,105 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366100 2023-11-23 15:47:24,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.13 vs. limit=12.0 2023-11-23 15:47:39,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2440700.0, ans=0.0 2023-11-23 15:47:42,591 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5400, loss[loss=0.08618, simple_loss=0.1244, pruned_loss=0.01916, audio_tagging_loss=0.00482, over 15425.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09248, pruned_loss=0.01376, audio_tagging_loss=0.008952, over 3046040.89 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:47:58,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2440833.3333333335, ans=0.0 2023-11-23 15:48:13,706 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.97 vs. limit=15.0 2023-11-23 15:48:15,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2440900.0, ans=0.0 2023-11-23 15:48:26,207 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366150 2023-11-23 15:48:28,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2440966.6666666665, ans=0.125 2023-11-23 15:48:40,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2441033.3333333335, ans=0.2 2023-11-23 15:48:42,966 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.475e+01 9.130e+01 9.785e+01 1.452e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-23 15:48:48,710 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5450, loss[loss=0.07082, simple_loss=0.1025, pruned_loss=0.01306, audio_tagging_loss=0.006521, over 14671.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09293, pruned_loss=0.01386, audio_tagging_loss=0.008957, over 3042647.50 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:48:56,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2441100.0, ans=0.0 2023-11-23 15:49:31,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366200 2023-11-23 15:49:36,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2441300.0, ans=0.0 2023-11-23 15:49:52,014 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.12 vs. limit=22.5 2023-11-23 15:49:52,718 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5500, loss[loss=0.06696, simple_loss=0.08998, pruned_loss=0.009226, audio_tagging_loss=0.01275, over 14196.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.0929, pruned_loss=0.01391, audio_tagging_loss=0.00909, over 3042597.41 frames. ], batch size: 52, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:50:02,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2441433.3333333335, ans=0.0 2023-11-23 15:50:04,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2441500.0, ans=0.2 2023-11-23 15:50:33,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2441633.3333333335, ans=0.125 2023-11-23 15:50:35,846 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366250 2023-11-23 15:50:45,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2441700.0, ans=0.0 2023-11-23 15:50:52,278 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.602e+01 8.303e+01 8.937e+01 9.609e+01 1.241e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 15:50:57,381 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5550, loss[loss=0.07214, simple_loss=0.106, pruned_loss=0.01188, audio_tagging_loss=0.007256, over 15995.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09389, pruned_loss=0.01407, audio_tagging_loss=0.009066, over 3046950.19 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:51:09,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2441833.3333333335, ans=0.0 2023-11-23 15:51:22,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2441900.0, ans=0.125 2023-11-23 15:51:40,017 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366300 2023-11-23 15:51:40,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2441966.6666666665, ans=0.125 2023-11-23 15:51:53,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.84 vs. limit=15.0 2023-11-23 15:52:01,535 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5600, loss[loss=0.09396, simple_loss=0.134, pruned_loss=0.02011, audio_tagging_loss=0.006867, over 14580.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09462, pruned_loss=0.01421, audio_tagging_loss=0.009117, over 3048710.45 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 15:52:10,553 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.67 vs. limit=15.0 2023-11-23 15:52:44,401 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366350 2023-11-23 15:52:48,047 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 15:53:00,692 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.372e+01 8.291e+01 8.935e+01 9.655e+01 1.271e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 15:53:05,691 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5650, loss[loss=0.06485, simple_loss=0.08858, pruned_loss=0.01204, audio_tagging_loss=0.008516, over 15194.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09385, pruned_loss=0.01418, audio_tagging_loss=0.00919, over 3054343.66 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 15:53:07,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2442433.3333333335, ans=0.125 2023-11-23 15:53:16,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2442433.3333333335, ans=0.1 2023-11-23 15:53:47,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366400 2023-11-23 15:53:58,891 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.02 vs. limit=10.0 2023-11-23 15:54:07,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2442700.0, ans=0.125 2023-11-23 15:54:09,827 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5700, loss[loss=0.0629, simple_loss=0.07327, pruned_loss=0.01395, audio_tagging_loss=0.01233, over 14985.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09307, pruned_loss=0.01405, audio_tagging_loss=0.009265, over 3053355.66 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 15:54:51,487 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366450 2023-11-23 15:55:00,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2443033.3333333335, ans=0.5 2023-11-23 15:55:01,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2443033.3333333335, ans=0.125 2023-11-23 15:55:08,825 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.337e+01 8.162e+01 8.678e+01 9.471e+01 1.109e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-23 15:55:13,717 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5750, loss[loss=0.07273, simple_loss=0.101, pruned_loss=0.01403, audio_tagging_loss=0.008196, over 15103.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09237, pruned_loss=0.01388, audio_tagging_loss=0.009205, over 3047209.35 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 15:55:14,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2443100.0, ans=0.1 2023-11-23 15:55:21,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2443100.0, ans=0.05 2023-11-23 15:55:27,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2443166.6666666665, ans=0.125 2023-11-23 15:55:36,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2443166.6666666665, ans=0.0 2023-11-23 15:55:56,265 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366500 2023-11-23 15:55:56,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2443300.0, ans=0.125 2023-11-23 15:56:00,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2443300.0, ans=0.0 2023-11-23 15:56:17,470 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5800, loss[loss=0.06777, simple_loss=0.085, pruned_loss=0.01123, audio_tagging_loss=0.01403, over 15478.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09258, pruned_loss=0.01384, audio_tagging_loss=0.009027, over 3051571.09 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:56:32,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2443500.0, ans=0.125 2023-11-23 15:56:35,401 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.26 vs. limit=15.0 2023-11-23 15:57:00,783 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366550 2023-11-23 15:57:14,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2443700.0, ans=0.0 2023-11-23 15:57:18,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.65 vs. limit=15.0 2023-11-23 15:57:18,854 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.421e+01 8.535e+01 9.028e+01 9.591e+01 1.185e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-23 15:57:21,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2443700.0, ans=0.0 2023-11-23 15:57:23,540 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5850, loss[loss=0.07747, simple_loss=0.1139, pruned_loss=0.01592, audio_tagging_loss=0.004583, over 16310.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09227, pruned_loss=0.01375, audio_tagging_loss=0.008985, over 3051315.55 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:57:50,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2443900.0, ans=0.125 2023-11-23 15:58:02,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2443966.6666666665, ans=0.2 2023-11-23 15:58:07,236 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366600 2023-11-23 15:58:12,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2443966.6666666665, ans=0.0 2023-11-23 15:58:30,445 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5900, loss[loss=0.06789, simple_loss=0.08989, pruned_loss=0.01481, audio_tagging_loss=0.008144, over 14835.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09263, pruned_loss=0.01374, audio_tagging_loss=0.008894, over 3046435.79 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:58:42,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0 2023-11-23 15:58:47,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2444166.6666666665, ans=0.2 2023-11-23 15:58:57,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.98 vs. limit=15.0 2023-11-23 15:59:14,261 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366650 2023-11-23 15:59:31,545 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.501e+01 9.185e+01 9.833e+01 1.392e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-23 15:59:33,305 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.78 vs. limit=12.0 2023-11-23 15:59:35,340 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 5950, loss[loss=0.08554, simple_loss=0.1173, pruned_loss=0.0207, audio_tagging_loss=0.006194, over 15082.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09408, pruned_loss=0.01401, audio_tagging_loss=0.008811, over 3055740.36 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 15:59:49,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2444500.0, ans=0.0 2023-11-23 16:00:19,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366700 2023-11-23 16:00:41,893 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6000, loss[loss=0.07138, simple_loss=0.09013, pruned_loss=0.01349, audio_tagging_loss=0.01282, over 15052.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09421, pruned_loss=0.01395, audio_tagging_loss=0.008804, over 3044508.71 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:00:41,894 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 16:01:26,551 INFO [train_asr.py:1253] (3/4) Epoch 31, validation: loss=0.05852, simple_loss=0.05109, pruned_loss=0.00511, audio_tagging_loss=0.02786, over 4681554.00 frames. 2023-11-23 16:01:26,553 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 16:01:46,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2444833.3333333335, ans=0.125 2023-11-23 16:01:48,150 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.50 vs. limit=12.0 2023-11-23 16:01:57,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2444900.0, ans=0.125 2023-11-23 16:02:08,870 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366750 2023-11-23 16:02:11,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2444966.6666666665, ans=0.0 2023-11-23 16:02:13,149 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 16:02:23,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2445033.3333333335, ans=0.1 2023-11-23 16:02:26,664 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.104e+01 8.225e+01 8.734e+01 9.674e+01 1.175e+02, threshold=1.747e+02, percent-clipped=0.0 2023-11-23 16:02:30,472 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6050, loss[loss=0.08427, simple_loss=0.1146, pruned_loss=0.01913, audio_tagging_loss=0.007825, over 17057.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09401, pruned_loss=0.01391, audio_tagging_loss=0.00883, over 3055392.06 frames. ], batch size: 62, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:03:13,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2445300.0, ans=0.1 2023-11-23 16:03:14,284 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366800 2023-11-23 16:03:22,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2445366.6666666665, ans=0.0 2023-11-23 16:03:27,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2445366.6666666665, ans=0.2 2023-11-23 16:03:31,476 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.34 vs. limit=15.0 2023-11-23 16:03:36,468 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6100, loss[loss=0.07066, simple_loss=0.09663, pruned_loss=0.015, audio_tagging_loss=0.007346, over 14928.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.0934, pruned_loss=0.01381, audio_tagging_loss=0.008864, over 3056960.10 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:03:49,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2445500.0, ans=0.1 2023-11-23 16:03:54,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.96 vs. limit=15.0 2023-11-23 16:04:09,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2445566.6666666665, ans=0.0 2023-11-23 16:04:10,128 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.52 vs. limit=12.0 2023-11-23 16:04:11,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2445566.6666666665, ans=0.125 2023-11-23 16:04:19,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366850 2023-11-23 16:04:37,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2445700.0, ans=0.2 2023-11-23 16:04:38,288 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.426e+01 8.983e+01 9.647e+01 1.287e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 16:04:41,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2445766.6666666665, ans=0.125 2023-11-23 16:04:42,610 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6150, loss[loss=0.05645, simple_loss=0.06739, pruned_loss=0.01112, audio_tagging_loss=0.01164, over 15167.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09388, pruned_loss=0.01392, audio_tagging_loss=0.008874, over 3056412.47 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:04:45,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2445766.6666666665, ans=0.2 2023-11-23 16:05:00,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=22.5 2023-11-23 16:05:01,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2445833.3333333335, ans=0.125 2023-11-23 16:05:02,850 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:05:24,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2445966.6666666665, ans=0.125 2023-11-23 16:05:25,604 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366900 2023-11-23 16:05:47,279 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6200, loss[loss=0.07146, simple_loss=0.1052, pruned_loss=0.01152, audio_tagging_loss=0.007341, over 14861.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.0922, pruned_loss=0.0137, audio_tagging_loss=0.009046, over 3057039.12 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:06:05,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2446166.6666666665, ans=0.125 2023-11-23 16:06:21,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2446233.3333333335, ans=0.125 2023-11-23 16:06:31,192 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 366950 2023-11-23 16:06:48,434 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.382e+01 9.012e+01 9.950e+01 2.061e+02, threshold=1.802e+02, percent-clipped=1.0 2023-11-23 16:06:49,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2446366.6666666665, ans=0.1 2023-11-23 16:06:52,762 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6250, loss[loss=0.08774, simple_loss=0.122, pruned_loss=0.02127, audio_tagging_loss=0.005482, over 16181.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09227, pruned_loss=0.01366, audio_tagging_loss=0.009093, over 3051170.87 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:06:53,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2446433.3333333335, ans=0.125 2023-11-23 16:06:53,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.50 vs. limit=15.0 2023-11-23 16:06:58,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2446433.3333333335, ans=0.125 2023-11-23 16:07:11,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2446500.0, ans=0.125 2023-11-23 16:07:15,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2446500.0, ans=0.125 2023-11-23 16:07:18,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2446566.6666666665, ans=0.0 2023-11-23 16:07:35,895 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367000 2023-11-23 16:07:40,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=2446633.3333333335, ans=15.0 2023-11-23 16:07:43,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.40 vs. limit=22.5 2023-11-23 16:07:45,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2446700.0, ans=0.95 2023-11-23 16:07:53,761 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.51 vs. limit=15.0 2023-11-23 16:07:58,540 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6300, loss[loss=0.0928, simple_loss=0.1174, pruned_loss=0.0244, audio_tagging_loss=0.00972, over 15010.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.0913, pruned_loss=0.01354, audio_tagging_loss=0.009279, over 3049324.24 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:08:34,879 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.64 vs. limit=15.0 2023-11-23 16:08:40,239 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367050 2023-11-23 16:08:40,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2446966.6666666665, ans=0.125 2023-11-23 16:08:41,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2446966.6666666665, ans=0.125 2023-11-23 16:08:56,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.48 vs. limit=15.0 2023-11-23 16:08:59,071 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.957e+01 8.355e+01 9.008e+01 9.768e+01 1.209e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-23 16:09:01,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2447100.0, ans=0.125 2023-11-23 16:09:02,813 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6350, loss[loss=0.06112, simple_loss=0.07771, pruned_loss=0.01113, audio_tagging_loss=0.01114, over 14783.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09132, pruned_loss=0.01374, audio_tagging_loss=0.009277, over 3046270.63 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:09:05,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2447100.0, ans=0.125 2023-11-23 16:09:20,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0 2023-11-23 16:09:30,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2447233.3333333335, ans=0.125 2023-11-23 16:09:44,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2447300.0, ans=0.1 2023-11-23 16:09:45,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367100 2023-11-23 16:09:53,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2447366.6666666665, ans=0.125 2023-11-23 16:09:59,316 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:10:06,309 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6400, loss[loss=0.07336, simple_loss=0.08762, pruned_loss=0.02086, audio_tagging_loss=0.008694, over 14707.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09098, pruned_loss=0.01378, audio_tagging_loss=0.009397, over 3053848.37 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:10:14,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2447433.3333333335, ans=0.1 2023-11-23 16:10:15,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2447433.3333333335, ans=0.125 2023-11-23 16:10:15,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2447433.3333333335, ans=0.0 2023-11-23 16:10:49,471 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367150 2023-11-23 16:10:59,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2447700.0, ans=0.125 2023-11-23 16:11:08,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.91 vs. limit=15.0 2023-11-23 16:11:08,616 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.384e+01 9.024e+01 9.742e+01 1.267e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 16:11:11,823 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6450, loss[loss=0.06805, simple_loss=0.09048, pruned_loss=0.01228, audio_tagging_loss=0.01053, over 15505.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09193, pruned_loss=0.01384, audio_tagging_loss=0.009417, over 3053741.92 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:11:20,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2447766.6666666665, ans=0.07 2023-11-23 16:11:32,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.60 vs. limit=10.0 2023-11-23 16:11:53,917 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367200 2023-11-23 16:11:58,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2447966.6666666665, ans=0.0 2023-11-23 16:12:02,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2448033.3333333335, ans=0.125 2023-11-23 16:12:09,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2448033.3333333335, ans=0.05 2023-11-23 16:12:17,099 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6500, loss[loss=0.07774, simple_loss=0.1032, pruned_loss=0.01685, audio_tagging_loss=0.009267, over 15111.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09079, pruned_loss=0.01361, audio_tagging_loss=0.009463, over 3051476.51 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:12:25,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2448100.0, ans=0.0 2023-11-23 16:12:39,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2448166.6666666665, ans=0.125 2023-11-23 16:13:00,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367250 2023-11-23 16:13:08,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.56 vs. limit=15.0 2023-11-23 16:13:09,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2448366.6666666665, ans=0.0 2023-11-23 16:13:17,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2448366.6666666665, ans=0.0 2023-11-23 16:13:18,765 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.647e+01 8.406e+01 9.076e+01 9.815e+01 1.354e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-23 16:13:19,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2448366.6666666665, ans=0.1 2023-11-23 16:13:21,315 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6550, loss[loss=0.05902, simple_loss=0.0808, pruned_loss=0.01025, audio_tagging_loss=0.008378, over 15006.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09042, pruned_loss=0.01343, audio_tagging_loss=0.00926, over 3053557.26 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:13:48,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2448566.6666666665, ans=0.125 2023-11-23 16:14:00,295 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0 2023-11-23 16:14:02,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2448633.3333333335, ans=0.1 2023-11-23 16:14:04,636 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367300 2023-11-23 16:14:15,380 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.30 vs. limit=15.0 2023-11-23 16:14:18,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2448700.0, ans=0.125 2023-11-23 16:14:25,945 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6600, loss[loss=0.06599, simple_loss=0.09004, pruned_loss=0.01275, audio_tagging_loss=0.008218, over 15295.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09056, pruned_loss=0.01342, audio_tagging_loss=0.009141, over 3044872.39 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:14:52,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2448900.0, ans=0.0 2023-11-23 16:14:52,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.79 vs. limit=22.5 2023-11-23 16:14:55,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2448900.0, ans=0.2 2023-11-23 16:14:57,842 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.89 vs. limit=12.0 2023-11-23 16:15:08,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367350 2023-11-23 16:15:13,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.55 vs. limit=15.0 2023-11-23 16:15:31,466 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.404e+01 9.056e+01 9.809e+01 1.686e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-23 16:15:31,520 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6650, loss[loss=0.0739, simple_loss=0.09871, pruned_loss=0.01451, audio_tagging_loss=0.01002, over 14696.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09015, pruned_loss=0.01338, audio_tagging_loss=0.009116, over 3046483.38 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:15:39,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2449100.0, ans=0.125 2023-11-23 16:15:55,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2449233.3333333335, ans=0.0 2023-11-23 16:16:03,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2449233.3333333335, ans=0.1 2023-11-23 16:16:14,117 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367400 2023-11-23 16:16:35,759 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6700, loss[loss=0.08153, simple_loss=0.1134, pruned_loss=0.01489, audio_tagging_loss=0.009953, over 14163.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09119, pruned_loss=0.01357, audio_tagging_loss=0.008977, over 3047453.48 frames. ], batch size: 51, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:16:52,058 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.55 vs. limit=15.0 2023-11-23 16:17:12,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2449566.6666666665, ans=0.1 2023-11-23 16:17:18,478 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.43 vs. limit=6.0 2023-11-23 16:17:19,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367450 2023-11-23 16:17:21,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2449633.3333333335, ans=0.125 2023-11-23 16:17:40,240 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.536e+01 8.542e+01 9.005e+01 9.931e+01 1.410e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 16:17:40,285 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6750, loss[loss=0.0873, simple_loss=0.112, pruned_loss=0.0211, audio_tagging_loss=0.01021, over 14730.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09071, pruned_loss=0.01353, audio_tagging_loss=0.009058, over 3038626.41 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:17:43,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2449766.6666666665, ans=0.0 2023-11-23 16:17:54,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.77 vs. limit=15.0 2023-11-23 16:18:09,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2449900.0, ans=0.125 2023-11-23 16:18:23,614 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367500 2023-11-23 16:18:26,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2449966.6666666665, ans=0.125 2023-11-23 16:18:26,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2449966.6666666665, ans=0.2 2023-11-23 16:18:29,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2449966.6666666665, ans=0.125 2023-11-23 16:18:31,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2450033.3333333335, ans=0.0 2023-11-23 16:18:43,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2450033.3333333335, ans=0.125 2023-11-23 16:18:45,569 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6800, loss[loss=0.07038, simple_loss=0.08702, pruned_loss=0.0137, audio_tagging_loss=0.01317, over 15950.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09125, pruned_loss=0.01357, audio_tagging_loss=0.009079, over 3036913.70 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:18:49,714 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.39 vs. limit=15.0 2023-11-23 16:18:56,888 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.77 vs. limit=15.0 2023-11-23 16:19:20,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2450233.3333333335, ans=0.1 2023-11-23 16:19:24,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2450300.0, ans=0.125 2023-11-23 16:19:27,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.64 vs. limit=15.0 2023-11-23 16:19:28,384 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367550 2023-11-23 16:19:31,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2450300.0, ans=0.125 2023-11-23 16:19:48,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2450366.6666666665, ans=0.2 2023-11-23 16:19:50,475 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6850, loss[loss=0.07501, simple_loss=0.1016, pruned_loss=0.01534, audio_tagging_loss=0.008895, over 15013.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09208, pruned_loss=0.01374, audio_tagging_loss=0.00898, over 3035095.13 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:19:51,630 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.211e+01 8.952e+01 9.815e+01 1.222e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-23 16:20:02,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2450500.0, ans=0.1 2023-11-23 16:20:02,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.86 vs. limit=15.0 2023-11-23 16:20:07,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2450500.0, ans=0.0 2023-11-23 16:20:33,202 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367600 2023-11-23 16:20:38,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2450633.3333333335, ans=0.125 2023-11-23 16:20:56,000 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6900, loss[loss=0.05508, simple_loss=0.07382, pruned_loss=0.01075, audio_tagging_loss=0.007421, over 14871.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09261, pruned_loss=0.01381, audio_tagging_loss=0.009023, over 3036292.61 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:21:01,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2450766.6666666665, ans=0.125 2023-11-23 16:21:14,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2450833.3333333335, ans=0.2 2023-11-23 16:21:24,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2450900.0, ans=0.2 2023-11-23 16:21:38,498 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367650 2023-11-23 16:21:46,577 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 16:21:53,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2451033.3333333335, ans=0.0 2023-11-23 16:22:01,294 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 6950, loss[loss=0.07494, simple_loss=0.1003, pruned_loss=0.01656, audio_tagging_loss=0.008249, over 15160.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09296, pruned_loss=0.01385, audio_tagging_loss=0.008976, over 3032236.71 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:22:02,446 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.751e+01 8.116e+01 8.969e+01 9.790e+01 1.259e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-23 16:22:11,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2451100.0, ans=0.125 2023-11-23 16:22:25,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2451233.3333333335, ans=0.1 2023-11-23 16:22:30,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2451233.3333333335, ans=0.125 2023-11-23 16:22:35,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2451233.3333333335, ans=0.04949747468305833 2023-11-23 16:22:35,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2451233.3333333335, ans=0.1 2023-11-23 16:22:36,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2451233.3333333335, ans=0.125 2023-11-23 16:22:40,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.80 vs. limit=15.0 2023-11-23 16:22:44,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367700 2023-11-23 16:22:57,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2451366.6666666665, ans=0.1 2023-11-23 16:23:05,244 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7000, loss[loss=0.0735, simple_loss=0.09878, pruned_loss=0.01325, audio_tagging_loss=0.01086, over 15254.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09278, pruned_loss=0.0138, audio_tagging_loss=0.009026, over 3034964.89 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:23:13,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2451433.3333333335, ans=0.125 2023-11-23 16:23:21,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2451500.0, ans=0.125 2023-11-23 16:23:23,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2451500.0, ans=0.125 2023-11-23 16:23:41,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2451566.6666666665, ans=0.125 2023-11-23 16:23:47,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2451633.3333333335, ans=0.0 2023-11-23 16:23:48,452 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367750 2023-11-23 16:24:10,636 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7050, loss[loss=0.06211, simple_loss=0.08018, pruned_loss=0.0124, audio_tagging_loss=0.009623, over 14407.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09233, pruned_loss=0.01387, audio_tagging_loss=0.00912, over 3038597.99 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:24:11,789 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.295e+01 8.920e+01 9.938e+01 1.396e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 16:24:17,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2451766.6666666665, ans=0.125 2023-11-23 16:24:30,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2451833.3333333335, ans=0.125 2023-11-23 16:24:33,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2451833.3333333335, ans=0.1 2023-11-23 16:24:53,015 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367800 2023-11-23 16:24:57,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2451966.6666666665, ans=0.95 2023-11-23 16:25:15,569 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7100, loss[loss=0.08036, simple_loss=0.1078, pruned_loss=0.01641, audio_tagging_loss=0.01005, over 15620.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09268, pruned_loss=0.01392, audio_tagging_loss=0.009162, over 3038104.67 frames. ], batch size: 62, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:25:17,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2452100.0, ans=0.125 2023-11-23 16:25:20,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2452100.0, ans=0.125 2023-11-23 16:25:53,470 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:25:58,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367850 2023-11-23 16:26:05,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2452300.0, ans=0.125 2023-11-23 16:26:20,284 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7150, loss[loss=0.09695, simple_loss=0.1373, pruned_loss=0.02257, audio_tagging_loss=0.005704, over 15774.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09361, pruned_loss=0.01405, audio_tagging_loss=0.009194, over 3044362.49 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:26:21,452 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.184e+01 8.537e+01 9.022e+01 9.919e+01 1.379e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 16:26:30,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2452433.3333333335, ans=0.125 2023-11-23 16:26:45,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2452566.6666666665, ans=0.125 2023-11-23 16:26:48,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2452566.6666666665, ans=0.05 2023-11-23 16:27:03,252 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367900 2023-11-23 16:27:03,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2452633.3333333335, ans=0.125 2023-11-23 16:27:07,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2452633.3333333335, ans=0.125 2023-11-23 16:27:17,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2452700.0, ans=0.0 2023-11-23 16:27:18,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2452700.0, ans=0.125 2023-11-23 16:27:24,442 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7200, loss[loss=0.05839, simple_loss=0.08389, pruned_loss=0.009148, audio_tagging_loss=0.007295, over 15468.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09362, pruned_loss=0.01398, audio_tagging_loss=0.009226, over 3046755.20 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:27:24,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2452766.6666666665, ans=0.125 2023-11-23 16:27:36,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2452833.3333333335, ans=0.0 2023-11-23 16:27:47,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2452833.3333333335, ans=0.0 2023-11-23 16:27:59,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2452900.0, ans=0.125 2023-11-23 16:28:03,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2452966.6666666665, ans=0.125 2023-11-23 16:28:06,758 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 367950 2023-11-23 16:28:21,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.84 vs. limit=15.0 2023-11-23 16:28:21,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.18 vs. limit=22.5 2023-11-23 16:28:21,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2453033.3333333335, ans=0.125 2023-11-23 16:28:29,953 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7250, loss[loss=0.06412, simple_loss=0.08675, pruned_loss=0.01258, audio_tagging_loss=0.008165, over 13433.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09259, pruned_loss=0.01381, audio_tagging_loss=0.009362, over 3054776.96 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:28:31,185 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.513e+01 9.044e+01 9.572e+01 1.281e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 16:29:10,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2453300.0, ans=0.0 2023-11-23 16:29:11,046 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.30 vs. limit=15.0 2023-11-23 16:29:12,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368000 2023-11-23 16:29:29,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2453366.6666666665, ans=0.0 2023-11-23 16:29:32,596 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.35 vs. limit=15.0 2023-11-23 16:29:38,296 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7300, loss[loss=0.08198, simple_loss=0.1135, pruned_loss=0.01797, audio_tagging_loss=0.007281, over 14476.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09304, pruned_loss=0.01387, audio_tagging_loss=0.00922, over 3058947.08 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:29:47,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2453433.3333333335, ans=0.0 2023-11-23 16:29:59,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2453500.0, ans=0.1 2023-11-23 16:30:02,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2453500.0, ans=0.0 2023-11-23 16:30:22,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368050 2023-11-23 16:30:33,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2453700.0, ans=0.0 2023-11-23 16:30:35,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2453700.0, ans=0.1 2023-11-23 16:30:42,881 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7350, loss[loss=0.05346, simple_loss=0.07345, pruned_loss=0.008135, audio_tagging_loss=0.008598, over 15889.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09193, pruned_loss=0.01366, audio_tagging_loss=0.009109, over 3055128.54 frames. ], batch size: 64, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:30:44,147 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.269e+01 8.908e+01 9.511e+01 1.304e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 16:30:57,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2453833.3333333335, ans=0.1 2023-11-23 16:31:03,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2453833.3333333335, ans=0.035 2023-11-23 16:31:15,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2453900.0, ans=0.125 2023-11-23 16:31:26,596 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368100 2023-11-23 16:31:28,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.56 vs. limit=12.0 2023-11-23 16:31:49,072 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7400, loss[loss=0.07138, simple_loss=0.09986, pruned_loss=0.01344, audio_tagging_loss=0.008013, over 14936.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09307, pruned_loss=0.01394, audio_tagging_loss=0.008937, over 3049980.56 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:31:54,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2454100.0, ans=0.0 2023-11-23 16:32:08,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.76 vs. limit=15.0 2023-11-23 16:32:32,541 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368150 2023-11-23 16:32:46,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2454366.6666666665, ans=0.125 2023-11-23 16:32:55,333 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7450, loss[loss=0.07999, simple_loss=0.1041, pruned_loss=0.01972, audio_tagging_loss=0.008228, over 14969.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09258, pruned_loss=0.01381, audio_tagging_loss=0.008859, over 3048770.98 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:32:56,564 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.753e+01 8.234e+01 8.919e+01 9.567e+01 1.256e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 16:33:10,914 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.66 vs. limit=15.0 2023-11-23 16:33:22,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0 2023-11-23 16:33:29,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2454566.6666666665, ans=0.1 2023-11-23 16:33:38,302 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368200 2023-11-23 16:33:42,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2454633.3333333335, ans=0.09899494936611666 2023-11-23 16:33:46,450 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.64 vs. limit=15.0 2023-11-23 16:33:53,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.97 vs. limit=15.0 2023-11-23 16:33:59,405 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7500, loss[loss=0.06184, simple_loss=0.0867, pruned_loss=0.01148, audio_tagging_loss=0.007007, over 16568.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09176, pruned_loss=0.01365, audio_tagging_loss=0.008837, over 3050330.52 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:33:59,689 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:34:11,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2454833.3333333335, ans=0.125 2023-11-23 16:34:42,530 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368250 2023-11-23 16:35:00,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2455033.3333333335, ans=0.0 2023-11-23 16:35:04,298 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7550, loss[loss=0.06218, simple_loss=0.08378, pruned_loss=0.01204, audio_tagging_loss=0.008247, over 14498.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09183, pruned_loss=0.01375, audio_tagging_loss=0.00873, over 3051919.41 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 16.0 2023-11-23 16:35:05,440 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.880e+01 8.421e+01 8.983e+01 9.712e+01 1.318e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 16:35:15,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2455100.0, ans=0.0 2023-11-23 16:35:43,267 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.40 vs. limit=6.0 2023-11-23 16:35:48,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368300 2023-11-23 16:36:10,453 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7600, loss[loss=0.06074, simple_loss=0.07958, pruned_loss=0.01195, audio_tagging_loss=0.008993, over 15292.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09106, pruned_loss=0.01379, audio_tagging_loss=0.008874, over 3052916.48 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:36:21,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2455500.0, ans=0.125 2023-11-23 16:36:23,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2455500.0, ans=0.125 2023-11-23 16:36:51,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2455633.3333333335, ans=0.09899494936611666 2023-11-23 16:36:52,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2455633.3333333335, ans=0.1 2023-11-23 16:36:53,346 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368350 2023-11-23 16:36:59,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2455633.3333333335, ans=0.125 2023-11-23 16:37:10,847 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.65 vs. limit=22.5 2023-11-23 16:37:15,076 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7650, loss[loss=0.06551, simple_loss=0.08648, pruned_loss=0.0145, audio_tagging_loss=0.007769, over 15383.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09081, pruned_loss=0.01371, audio_tagging_loss=0.008834, over 3045226.90 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0 2023-11-23 16:37:16,300 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 8.296e+01 8.999e+01 9.815e+01 1.332e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-23 16:37:36,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2455833.3333333335, ans=0.0 2023-11-23 16:37:45,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2455900.0, ans=0.0 2023-11-23 16:37:46,819 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.44 vs. limit=12.0 2023-11-23 16:37:54,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2455966.6666666665, ans=0.0 2023-11-23 16:37:55,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2455966.6666666665, ans=0.2 2023-11-23 16:37:58,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368400 2023-11-23 16:38:20,880 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7700, loss[loss=0.07519, simple_loss=0.1001, pruned_loss=0.01518, audio_tagging_loss=0.009981, over 16120.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09194, pruned_loss=0.01384, audio_tagging_loss=0.008855, over 3049808.92 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:39:01,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2456300.0, ans=0.125 2023-11-23 16:39:02,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2456300.0, ans=0.1 2023-11-23 16:39:04,001 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368450 2023-11-23 16:39:04,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2456300.0, ans=0.125 2023-11-23 16:39:04,848 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.34 vs. limit=15.0 2023-11-23 16:39:13,228 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.38 vs. limit=10.0 2023-11-23 16:39:20,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2456366.6666666665, ans=0.125 2023-11-23 16:39:26,195 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7750, loss[loss=0.06694, simple_loss=0.09391, pruned_loss=0.01326, audio_tagging_loss=0.006731, over 15472.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09158, pruned_loss=0.01368, audio_tagging_loss=0.008986, over 3050661.85 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:39:29,813 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.166e+01 8.983e+01 1.017e+02 1.300e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 16:39:39,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2456500.0, ans=0.125 2023-11-23 16:39:41,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2456500.0, ans=0.0 2023-11-23 16:39:51,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2456566.6666666665, ans=0.125 2023-11-23 16:39:59,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2456566.6666666665, ans=0.125 2023-11-23 16:40:07,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2456633.3333333335, ans=0.0 2023-11-23 16:40:08,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368500 2023-11-23 16:40:08,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2456633.3333333335, ans=0.04949747468305833 2023-11-23 16:40:17,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2456700.0, ans=0.0 2023-11-23 16:40:30,627 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7800, loss[loss=0.03921, simple_loss=0.05443, pruned_loss=0.00208, audio_tagging_loss=0.009917, over 16004.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.0913, pruned_loss=0.01368, audio_tagging_loss=0.009062, over 3046700.47 frames. ], batch size: 63, lr: 2.19e-03, grad_scale: 8.0 2023-11-23 16:40:35,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2456766.6666666665, ans=0.05 2023-11-23 16:40:53,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2456833.3333333335, ans=0.125 2023-11-23 16:40:58,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2456900.0, ans=0.125 2023-11-23 16:41:10,649 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.30 vs. limit=15.0 2023-11-23 16:41:13,779 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368550 2023-11-23 16:41:28,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2457033.3333333335, ans=0.0 2023-11-23 16:41:35,465 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7850, loss[loss=0.07278, simple_loss=0.09925, pruned_loss=0.01361, audio_tagging_loss=0.009537, over 15632.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09181, pruned_loss=0.01396, audio_tagging_loss=0.009092, over 3040873.90 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:41:39,080 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 8.382e+01 9.207e+01 1.008e+02 1.649e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-23 16:41:55,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2457166.6666666665, ans=0.2 2023-11-23 16:41:56,952 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.34 vs. limit=15.0 2023-11-23 16:42:00,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2457233.3333333335, ans=0.1 2023-11-23 16:42:05,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2457233.3333333335, ans=0.125 2023-11-23 16:42:07,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.10 vs. limit=15.0 2023-11-23 16:42:12,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2457233.3333333335, ans=0.125 2023-11-23 16:42:15,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2457300.0, ans=0.125 2023-11-23 16:42:17,999 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368600 2023-11-23 16:42:29,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2457366.6666666665, ans=0.0 2023-11-23 16:42:40,591 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7900, loss[loss=0.05674, simple_loss=0.06346, pruned_loss=0.0129, audio_tagging_loss=0.01211, over 15337.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09217, pruned_loss=0.01396, audio_tagging_loss=0.009169, over 3050859.90 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:42:40,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2457433.3333333335, ans=10.0 2023-11-23 16:42:47,923 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.21 vs. limit=15.0 2023-11-23 16:42:53,316 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:42:54,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2457500.0, ans=0.1 2023-11-23 16:43:23,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368650 2023-11-23 16:43:33,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2457700.0, ans=0.1 2023-11-23 16:43:35,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2457700.0, ans=0.125 2023-11-23 16:43:45,889 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 7950, loss[loss=0.0733, simple_loss=0.1029, pruned_loss=0.01462, audio_tagging_loss=0.007223, over 15470.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.092, pruned_loss=0.01383, audio_tagging_loss=0.009305, over 3050824.95 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:43:49,525 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.234e+01 8.856e+01 9.622e+01 1.359e+02, threshold=1.771e+02, percent-clipped=0.0 2023-11-23 16:43:52,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2457766.6666666665, ans=0.125 2023-11-23 16:44:01,287 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 16:44:03,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2457833.3333333335, ans=0.0 2023-11-23 16:44:10,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2457900.0, ans=0.2 2023-11-23 16:44:21,512 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0 2023-11-23 16:44:23,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2457966.6666666665, ans=0.125 2023-11-23 16:44:28,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368700 2023-11-23 16:44:50,930 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8000, loss[loss=0.06628, simple_loss=0.09048, pruned_loss=0.01228, audio_tagging_loss=0.008755, over 14096.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09218, pruned_loss=0.01389, audio_tagging_loss=0.009343, over 3050201.74 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:44:58,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2458100.0, ans=0.05 2023-11-23 16:45:01,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2458100.0, ans=0.125 2023-11-23 16:45:23,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2458233.3333333335, ans=0.1 2023-11-23 16:45:32,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2458300.0, ans=0.0 2023-11-23 16:45:33,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368750 2023-11-23 16:45:54,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2458433.3333333335, ans=0.2 2023-11-23 16:45:55,519 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8050, loss[loss=0.07737, simple_loss=0.101, pruned_loss=0.01613, audio_tagging_loss=0.01072, over 15306.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09128, pruned_loss=0.01381, audio_tagging_loss=0.009477, over 3053835.98 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:46:00,364 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.362e+01 8.796e+01 9.330e+01 1.321e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-23 16:46:04,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2458433.3333333335, ans=0.125 2023-11-23 16:46:18,174 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:46:25,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.91 vs. limit=22.5 2023-11-23 16:46:31,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2458566.6666666665, ans=0.125 2023-11-23 16:46:38,203 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368800 2023-11-23 16:46:56,148 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.58 vs. limit=15.0 2023-11-23 16:47:00,267 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8100, loss[loss=0.07031, simple_loss=0.08828, pruned_loss=0.01498, audio_tagging_loss=0.01118, over 14848.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09234, pruned_loss=0.01395, audio_tagging_loss=0.009373, over 3047908.93 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:47:36,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2458900.0, ans=0.0 2023-11-23 16:47:43,056 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368850 2023-11-23 16:47:58,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2459033.3333333335, ans=0.1 2023-11-23 16:48:01,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=15.0 2023-11-23 16:48:04,214 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8150, loss[loss=0.09155, simple_loss=0.1163, pruned_loss=0.02493, audio_tagging_loss=0.008458, over 16005.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09201, pruned_loss=0.01397, audio_tagging_loss=0.009232, over 3043665.06 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:48:09,583 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.557e+01 9.149e+01 9.819e+01 1.226e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-23 16:48:30,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2459233.3333333335, ans=0.0 2023-11-23 16:48:32,214 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.87 vs. limit=12.0 2023-11-23 16:48:46,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368900 2023-11-23 16:49:08,925 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8200, loss[loss=0.0631, simple_loss=0.09077, pruned_loss=0.01073, audio_tagging_loss=0.006984, over 15711.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.0927, pruned_loss=0.01392, audio_tagging_loss=0.009047, over 3046906.07 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:49:09,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2459433.3333333335, ans=0.1 2023-11-23 16:49:10,257 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 16:49:10,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.27 vs. limit=10.0 2023-11-23 16:49:43,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2459566.6666666665, ans=0.125 2023-11-23 16:49:45,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2459566.6666666665, ans=0.125 2023-11-23 16:49:45,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2459566.6666666665, ans=0.1 2023-11-23 16:49:51,307 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 368950 2023-11-23 16:50:13,325 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8250, loss[loss=0.0795, simple_loss=0.1032, pruned_loss=0.01876, audio_tagging_loss=0.009161, over 15860.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09264, pruned_loss=0.01391, audio_tagging_loss=0.009015, over 3049553.25 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:50:18,103 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.120e+01 8.966e+01 9.710e+01 1.268e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-23 16:50:56,558 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369000 2023-11-23 16:51:18,624 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8300, loss[loss=0.06684, simple_loss=0.09048, pruned_loss=0.01265, audio_tagging_loss=0.008947, over 14643.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09245, pruned_loss=0.01378, audio_tagging_loss=0.009053, over 3045745.04 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:51:40,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2460166.6666666665, ans=0.1 2023-11-23 16:51:43,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2460233.3333333335, ans=0.0 2023-11-23 16:51:56,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2460300.0, ans=0.0 2023-11-23 16:51:59,563 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.20 vs. limit=15.0 2023-11-23 16:52:01,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369050 2023-11-23 16:52:04,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2460300.0, ans=0.0 2023-11-23 16:52:23,780 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8350, loss[loss=0.04895, simple_loss=0.06007, pruned_loss=0.006594, audio_tagging_loss=0.01231, over 15559.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09209, pruned_loss=0.01373, audio_tagging_loss=0.00904, over 3046162.49 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 16:52:29,184 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.409e+01 8.427e+01 9.150e+01 9.760e+01 1.195e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-23 16:52:29,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2460433.3333333335, ans=0.1 2023-11-23 16:52:30,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2460433.3333333335, ans=0.07 2023-11-23 16:52:51,232 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.07 vs. limit=15.0 2023-11-23 16:53:06,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369100 2023-11-23 16:53:10,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_na.min_abs, batch_count=2460633.3333333335, ans=0.02 2023-11-23 16:53:11,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2460633.3333333335, ans=0.125 2023-11-23 16:53:26,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2460700.0, ans=0.0 2023-11-23 16:53:28,827 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8400, loss[loss=0.07134, simple_loss=0.1006, pruned_loss=0.01457, audio_tagging_loss=0.006443, over 15416.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09062, pruned_loss=0.01343, audio_tagging_loss=0.009048, over 3040246.02 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:53:40,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2460833.3333333335, ans=0.2 2023-11-23 16:53:48,216 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.25 vs. limit=15.0 2023-11-23 16:54:10,967 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369150 2023-11-23 16:54:11,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2460966.6666666665, ans=0.125 2023-11-23 16:54:13,309 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0 2023-11-23 16:54:30,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2461033.3333333335, ans=0.0 2023-11-23 16:54:32,573 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8450, loss[loss=0.0803, simple_loss=0.1103, pruned_loss=0.01762, audio_tagging_loss=0.007515, over 13951.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.0908, pruned_loss=0.01353, audio_tagging_loss=0.009049, over 3034015.15 frames. ], batch size: 52, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:54:34,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0 2023-11-23 16:54:37,476 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.445e+01 8.837e+01 9.573e+01 1.326e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 16:54:51,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2461166.6666666665, ans=0.2 2023-11-23 16:55:04,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2461233.3333333335, ans=0.125 2023-11-23 16:55:16,222 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369200 2023-11-23 16:55:16,887 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.09 vs. limit=15.0 2023-11-23 16:55:36,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2461366.6666666665, ans=0.125 2023-11-23 16:55:36,796 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:55:38,842 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8500, loss[loss=0.07263, simple_loss=0.0945, pruned_loss=0.01665, audio_tagging_loss=0.008735, over 14571.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09084, pruned_loss=0.01345, audio_tagging_loss=0.009083, over 3040588.99 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:55:46,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2461433.3333333335, ans=0.125 2023-11-23 16:55:51,152 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.62 vs. limit=10.0 2023-11-23 16:55:52,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.01 vs. limit=15.0 2023-11-23 16:55:58,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2461500.0, ans=0.2 2023-11-23 16:56:01,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.72 vs. limit=15.0 2023-11-23 16:56:08,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2461566.6666666665, ans=0.1 2023-11-23 16:56:10,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2461566.6666666665, ans=0.125 2023-11-23 16:56:21,371 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369250 2023-11-23 16:56:44,059 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8550, loss[loss=0.07439, simple_loss=0.1016, pruned_loss=0.01282, audio_tagging_loss=0.01079, over 15806.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09121, pruned_loss=0.01356, audio_tagging_loss=0.009126, over 3041774.33 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:56:47,236 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.39 vs. limit=6.0 2023-11-23 16:56:49,053 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.318e+01 8.758e+01 9.491e+01 1.282e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-23 16:57:01,907 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 16:57:18,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2461900.0, ans=0.125 2023-11-23 16:57:21,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2461966.6666666665, ans=0.1 2023-11-23 16:57:26,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369300 2023-11-23 16:57:48,084 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8600, loss[loss=0.07952, simple_loss=0.1071, pruned_loss=0.01538, audio_tagging_loss=0.01058, over 15375.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09186, pruned_loss=0.01338, audio_tagging_loss=0.00908, over 3045619.93 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:57:58,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2462100.0, ans=0.0 2023-11-23 16:58:09,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0 2023-11-23 16:58:11,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2462166.6666666665, ans=0.125 2023-11-23 16:58:20,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.76 vs. limit=6.0 2023-11-23 16:58:30,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2462300.0, ans=0.125 2023-11-23 16:58:31,386 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369350 2023-11-23 16:58:52,588 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8650, loss[loss=0.05443, simple_loss=0.06555, pruned_loss=0.0121, audio_tagging_loss=0.009561, over 16247.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09204, pruned_loss=0.01359, audio_tagging_loss=0.009072, over 3047742.51 frames. ], batch size: 63, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:58:53,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2462433.3333333335, ans=0.125 2023-11-23 16:58:57,966 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.075e+01 8.582e+01 9.201e+01 9.858e+01 1.215e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-23 16:59:03,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2462433.3333333335, ans=0.125 2023-11-23 16:59:05,356 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.38 vs. limit=10.0 2023-11-23 16:59:15,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2462500.0, ans=0.125 2023-11-23 16:59:20,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.48 vs. limit=15.0 2023-11-23 16:59:31,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2462633.3333333335, ans=0.125 2023-11-23 16:59:35,344 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369400 2023-11-23 16:59:37,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.65 vs. limit=15.0 2023-11-23 16:59:50,352 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.58 vs. limit=12.0 2023-11-23 16:59:57,846 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8700, loss[loss=0.07131, simple_loss=0.09008, pruned_loss=0.01102, audio_tagging_loss=0.01525, over 14952.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09238, pruned_loss=0.01364, audio_tagging_loss=0.009136, over 3050175.70 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 16:59:59,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2462766.6666666665, ans=0.125 2023-11-23 17:00:03,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2462766.6666666665, ans=0.1 2023-11-23 17:00:27,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2462900.0, ans=0.0 2023-11-23 17:00:41,114 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369450 2023-11-23 17:00:48,652 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.84 vs. limit=15.0 2023-11-23 17:00:54,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.48 vs. limit=15.0 2023-11-23 17:01:02,640 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8750, loss[loss=0.07796, simple_loss=0.1089, pruned_loss=0.01486, audio_tagging_loss=0.008638, over 15903.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09294, pruned_loss=0.01383, audio_tagging_loss=0.009132, over 3054266.50 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:01:05,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2463100.0, ans=0.125 2023-11-23 17:01:07,438 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.270e+01 8.973e+01 9.908e+01 1.550e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-23 17:01:10,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2463100.0, ans=0.125 2023-11-23 17:01:45,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369500 2023-11-23 17:01:50,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2463300.0, ans=0.2 2023-11-23 17:01:52,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0 2023-11-23 17:02:00,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2463366.6666666665, ans=0.0 2023-11-23 17:02:01,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.56 vs. limit=22.5 2023-11-23 17:02:06,830 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8800, loss[loss=0.05906, simple_loss=0.06689, pruned_loss=0.01328, audio_tagging_loss=0.01233, over 15268.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09272, pruned_loss=0.0139, audio_tagging_loss=0.009212, over 3056078.56 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:02:06,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2463433.3333333335, ans=0.0 2023-11-23 17:02:28,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2463500.0, ans=0.125 2023-11-23 17:02:43,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2463566.6666666665, ans=0.0 2023-11-23 17:02:45,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2463633.3333333335, ans=0.0 2023-11-23 17:02:49,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369550 2023-11-23 17:03:03,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2463700.0, ans=0.0 2023-11-23 17:03:12,309 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8850, loss[loss=0.07122, simple_loss=0.09736, pruned_loss=0.01416, audio_tagging_loss=0.008382, over 15230.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09321, pruned_loss=0.01402, audio_tagging_loss=0.009265, over 3053781.02 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:03:17,847 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.515e+01 9.142e+01 9.954e+01 1.396e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-23 17:03:20,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2463766.6666666665, ans=0.5 2023-11-23 17:03:24,197 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:03:30,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2463833.3333333335, ans=0.125 2023-11-23 17:03:52,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2463966.6666666665, ans=0.2 2023-11-23 17:03:54,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369600 2023-11-23 17:03:55,084 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.39 vs. limit=22.5 2023-11-23 17:03:57,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2463966.6666666665, ans=0.125 2023-11-23 17:04:06,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2464033.3333333335, ans=0.1 2023-11-23 17:04:15,438 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.08 vs. limit=22.5 2023-11-23 17:04:17,301 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8900, loss[loss=0.09075, simple_loss=0.1166, pruned_loss=0.02398, audio_tagging_loss=0.008456, over 15738.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09393, pruned_loss=0.01411, audio_tagging_loss=0.009115, over 3049773.48 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:04:24,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2464100.0, ans=0.125 2023-11-23 17:04:31,639 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.25 vs. limit=10.0 2023-11-23 17:04:41,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2464233.3333333335, ans=0.125 2023-11-23 17:05:00,855 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369650 2023-11-23 17:05:12,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2464366.6666666665, ans=0.5 2023-11-23 17:05:22,409 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 8950, loss[loss=0.07184, simple_loss=0.1037, pruned_loss=0.0121, audio_tagging_loss=0.007867, over 15216.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09371, pruned_loss=0.01402, audio_tagging_loss=0.008971, over 3045560.30 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:05:27,243 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.154e+01 8.323e+01 9.092e+01 9.814e+01 1.259e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-23 17:05:54,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2464566.6666666665, ans=10.0 2023-11-23 17:06:04,937 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369700 2023-11-23 17:06:06,724 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.72 vs. limit=6.0 2023-11-23 17:06:11,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2464633.3333333335, ans=0.0 2023-11-23 17:06:27,171 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9000, loss[loss=0.08416, simple_loss=0.1086, pruned_loss=0.02109, audio_tagging_loss=0.008743, over 15214.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09396, pruned_loss=0.01401, audio_tagging_loss=0.008796, over 3047041.62 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:06:27,172 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 17:06:59,289 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2590, 4.2765, 4.4932, 4.4581], device='cuda:3') 2023-11-23 17:07:07,404 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4534, 3.8190, 2.8417, 3.7252], device='cuda:3') 2023-11-23 17:07:11,615 INFO [train_asr.py:1253] (3/4) Epoch 31, validation: loss=0.05916, simple_loss=0.05108, pruned_loss=0.005166, audio_tagging_loss=0.02845, over 4681554.00 frames. 2023-11-23 17:07:11,616 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 17:07:32,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2464833.3333333335, ans=0.1 2023-11-23 17:07:54,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369750 2023-11-23 17:07:56,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2464966.6666666665, ans=0.0 2023-11-23 17:08:02,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2465033.3333333335, ans=0.125 2023-11-23 17:08:11,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2465033.3333333335, ans=0.0 2023-11-23 17:08:15,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2465100.0, ans=0.125 2023-11-23 17:08:16,243 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9050, loss[loss=0.05245, simple_loss=0.07359, pruned_loss=0.005599, audio_tagging_loss=0.01005, over 16436.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09467, pruned_loss=0.01414, audio_tagging_loss=0.008737, over 3055723.14 frames. ], batch size: 63, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:08:21,067 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.342e+01 8.456e+01 8.970e+01 9.727e+01 1.359e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-23 17:08:33,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2465166.6666666665, ans=0.125 2023-11-23 17:08:43,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=12.0 2023-11-23 17:08:58,829 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369800 2023-11-23 17:09:14,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2465366.6666666665, ans=0.125 2023-11-23 17:09:15,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2465366.6666666665, ans=0.0 2023-11-23 17:09:19,869 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 17:09:20,918 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9100, loss[loss=0.07734, simple_loss=0.1039, pruned_loss=0.01824, audio_tagging_loss=0.007164, over 14380.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09417, pruned_loss=0.01408, audio_tagging_loss=0.008684, over 3061974.11 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:09:32,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2465500.0, ans=0.125 2023-11-23 17:09:37,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2465500.0, ans=0.125 2023-11-23 17:09:52,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2465566.6666666665, ans=0.95 2023-11-23 17:09:54,874 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 17:10:03,293 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369850 2023-11-23 17:10:25,956 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9150, loss[loss=0.05965, simple_loss=0.07667, pruned_loss=0.01201, audio_tagging_loss=0.009308, over 15410.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09391, pruned_loss=0.01394, audio_tagging_loss=0.008713, over 3054434.97 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:10:30,943 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.474e+01 8.198e+01 8.578e+01 9.347e+01 1.177e+02, threshold=1.716e+02, percent-clipped=0.0 2023-11-23 17:10:31,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2465766.6666666665, ans=0.09899494936611666 2023-11-23 17:10:33,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2465766.6666666665, ans=0.04949747468305833 2023-11-23 17:10:34,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2465766.6666666665, ans=0.0 2023-11-23 17:10:56,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2465900.0, ans=0.125 2023-11-23 17:11:08,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369900 2023-11-23 17:11:30,967 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9200, loss[loss=0.07397, simple_loss=0.1008, pruned_loss=0.01483, audio_tagging_loss=0.008738, over 15892.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.0928, pruned_loss=0.01366, audio_tagging_loss=0.008767, over 3053821.92 frames. ], batch size: 61, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:11:32,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2466100.0, ans=0.125 2023-11-23 17:11:41,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2466100.0, ans=0.0 2023-11-23 17:11:42,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2466166.6666666665, ans=0.1 2023-11-23 17:11:59,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2466233.3333333335, ans=0.125 2023-11-23 17:12:09,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2466300.0, ans=0.125 2023-11-23 17:12:09,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2466300.0, ans=0.125 2023-11-23 17:12:13,807 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 369950 2023-11-23 17:12:35,939 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9250, loss[loss=0.05743, simple_loss=0.07136, pruned_loss=0.01202, audio_tagging_loss=0.009724, over 14633.00 frames. ], tot_loss[loss=0.069, simple_loss=0.0932, pruned_loss=0.01369, audio_tagging_loss=0.008701, over 3053098.10 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:12:40,851 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.301e+01 8.925e+01 9.795e+01 1.226e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 17:12:44,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2466433.3333333335, ans=0.125 2023-11-23 17:13:06,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2466566.6666666665, ans=0.0 2023-11-23 17:13:17,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2466633.3333333335, ans=0.0 2023-11-23 17:13:18,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370000 2023-11-23 17:13:42,038 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9300, loss[loss=0.05993, simple_loss=0.08105, pruned_loss=0.01208, audio_tagging_loss=0.007323, over 15091.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09294, pruned_loss=0.01379, audio_tagging_loss=0.008728, over 3057331.22 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:13:54,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2466833.3333333335, ans=0.125 2023-11-23 17:13:55,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.74 vs. limit=15.0 2023-11-23 17:14:02,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2466833.3333333335, ans=0.0 2023-11-23 17:14:08,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2466900.0, ans=0.125 2023-11-23 17:14:25,619 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370050 2023-11-23 17:14:47,296 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9350, loss[loss=0.06953, simple_loss=0.08902, pruned_loss=0.01556, audio_tagging_loss=0.009462, over 16107.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09285, pruned_loss=0.01395, audio_tagging_loss=0.008838, over 3055578.96 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:14:52,707 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.132e+01 8.275e+01 9.065e+01 9.778e+01 1.188e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-23 17:15:18,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2467233.3333333335, ans=0.1 2023-11-23 17:15:29,554 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370100 2023-11-23 17:15:38,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2467366.6666666665, ans=0.125 2023-11-23 17:15:50,781 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9400, loss[loss=0.0825, simple_loss=0.1216, pruned_loss=0.01482, audio_tagging_loss=0.006858, over 15027.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09245, pruned_loss=0.01376, audio_tagging_loss=0.008963, over 3050864.28 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:15:55,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2467433.3333333335, ans=0.125 2023-11-23 17:16:01,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2467433.3333333335, ans=0.0 2023-11-23 17:16:10,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2467500.0, ans=0.2 2023-11-23 17:16:15,476 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.14 vs. limit=15.0 2023-11-23 17:16:19,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2467566.6666666665, ans=0.0 2023-11-23 17:16:20,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2467566.6666666665, ans=0.09899494936611666 2023-11-23 17:16:25,811 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.99 vs. limit=10.0 2023-11-23 17:16:33,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370150 2023-11-23 17:16:53,395 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:16:54,590 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9450, loss[loss=0.06496, simple_loss=0.08807, pruned_loss=0.01374, audio_tagging_loss=0.007188, over 15398.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09302, pruned_loss=0.0138, audio_tagging_loss=0.008918, over 3062294.33 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:16:55,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2467766.6666666665, ans=0.125 2023-11-23 17:16:58,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2467766.6666666665, ans=0.125 2023-11-23 17:16:59,340 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.446e+01 8.380e+01 9.034e+01 1.002e+02 1.394e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-23 17:17:32,296 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.36 vs. limit=15.0 2023-11-23 17:17:36,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370200 2023-11-23 17:17:37,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2467966.6666666665, ans=0.09899494936611666 2023-11-23 17:17:37,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2467966.6666666665, ans=0.125 2023-11-23 17:17:58,773 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9500, loss[loss=0.06017, simple_loss=0.07984, pruned_loss=0.01077, audio_tagging_loss=0.009485, over 14629.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09268, pruned_loss=0.01382, audio_tagging_loss=0.009085, over 3060426.15 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:18:40,443 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370250 2023-11-23 17:18:54,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2468366.6666666665, ans=0.0 2023-11-23 17:19:02,044 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9550, loss[loss=0.06659, simple_loss=0.09171, pruned_loss=0.01085, audio_tagging_loss=0.009885, over 14925.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09288, pruned_loss=0.01376, audio_tagging_loss=0.00913, over 3060923.80 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:19:07,295 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.559e+01 9.097e+01 9.877e+01 1.537e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-23 17:19:18,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2468500.0, ans=0.125 2023-11-23 17:19:21,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2468500.0, ans=0.125 2023-11-23 17:19:24,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2468500.0, ans=0.125 2023-11-23 17:19:28,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2468566.6666666665, ans=0.125 2023-11-23 17:19:43,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370300 2023-11-23 17:20:05,143 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9600, loss[loss=0.05317, simple_loss=0.06169, pruned_loss=0.01043, audio_tagging_loss=0.0119, over 14787.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09225, pruned_loss=0.01361, audio_tagging_loss=0.009252, over 3057790.49 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:20:15,475 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.95 vs. limit=15.0 2023-11-23 17:20:46,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370350 2023-11-23 17:21:06,830 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9650, loss[loss=0.05306, simple_loss=0.07072, pruned_loss=0.009133, audio_tagging_loss=0.008563, over 15472.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09208, pruned_loss=0.01363, audio_tagging_loss=0.00922, over 3054318.29 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:21:09,906 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.89 vs. limit=12.0 2023-11-23 17:21:11,546 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.236e+01 8.814e+01 9.557e+01 1.553e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-23 17:21:25,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2469166.6666666665, ans=0.125 2023-11-23 17:21:48,763 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370400 2023-11-23 17:21:53,446 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.18 vs. limit=15.0 2023-11-23 17:22:04,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2469366.6666666665, ans=0.125 2023-11-23 17:22:10,464 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9700, loss[loss=0.07017, simple_loss=0.0925, pruned_loss=0.01382, audio_tagging_loss=0.0101, over 15056.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09231, pruned_loss=0.01361, audio_tagging_loss=0.009141, over 3057302.06 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:22:15,858 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.68 vs. limit=15.0 2023-11-23 17:22:23,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2469500.0, ans=0.125 2023-11-23 17:22:30,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.67 vs. limit=12.0 2023-11-23 17:22:51,992 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370450 2023-11-23 17:23:13,580 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9750, loss[loss=0.06694, simple_loss=0.09348, pruned_loss=0.01318, audio_tagging_loss=0.007023, over 14407.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09109, pruned_loss=0.0135, audio_tagging_loss=0.009026, over 3051064.37 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:23:15,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2469766.6666666665, ans=10.0 2023-11-23 17:23:20,157 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.276e+01 8.363e+01 8.980e+01 9.912e+01 1.548e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 17:23:51,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2469966.6666666665, ans=0.125 2023-11-23 17:23:55,044 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370500 2023-11-23 17:24:16,043 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9800, loss[loss=0.07237, simple_loss=0.09672, pruned_loss=0.01474, audio_tagging_loss=0.009274, over 15079.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09174, pruned_loss=0.01368, audio_tagging_loss=0.009023, over 3051686.38 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:24:21,337 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.05 vs. limit=12.0 2023-11-23 17:24:40,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2470233.3333333335, ans=0.0 2023-11-23 17:24:43,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2470233.3333333335, ans=0.0 2023-11-23 17:24:47,952 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.05 vs. limit=15.0 2023-11-23 17:24:51,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2470233.3333333335, ans=0.125 2023-11-23 17:24:54,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2470300.0, ans=0.2 2023-11-23 17:24:54,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2470300.0, ans=0.125 2023-11-23 17:24:57,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370550 2023-11-23 17:25:10,338 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:25:18,008 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9850, loss[loss=0.07668, simple_loss=0.09101, pruned_loss=0.02031, audio_tagging_loss=0.01086, over 13911.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09145, pruned_loss=0.01386, audio_tagging_loss=0.009015, over 3050495.73 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:25:24,368 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.399e+01 8.482e+01 9.119e+01 9.855e+01 1.403e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-23 17:25:30,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2470500.0, ans=0.05 2023-11-23 17:25:39,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2470500.0, ans=0.125 2023-11-23 17:25:40,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2470500.0, ans=0.125 2023-11-23 17:25:58,191 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370600 2023-11-23 17:26:18,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2470700.0, ans=0.1 2023-11-23 17:26:20,139 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9900, loss[loss=0.06634, simple_loss=0.09005, pruned_loss=0.01314, audio_tagging_loss=0.008175, over 14471.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09094, pruned_loss=0.01372, audio_tagging_loss=0.009021, over 3048635.16 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:26:21,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2470766.6666666665, ans=0.5 2023-11-23 17:26:33,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.30 vs. limit=15.0 2023-11-23 17:26:35,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2470833.3333333335, ans=0.125 2023-11-23 17:26:43,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2470900.0, ans=0.125 2023-11-23 17:26:50,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2470900.0, ans=0.125 2023-11-23 17:26:57,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2470966.6666666665, ans=0.125 2023-11-23 17:27:00,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370650 2023-11-23 17:27:22,253 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 9950, loss[loss=0.05617, simple_loss=0.07332, pruned_loss=0.01137, audio_tagging_loss=0.008144, over 15150.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09069, pruned_loss=0.01358, audio_tagging_loss=0.008986, over 3053529.39 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:27:28,090 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.973e+01 8.205e+01 8.691e+01 9.411e+01 1.200e+02, threshold=1.738e+02, percent-clipped=0.0 2023-11-23 17:27:28,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0 2023-11-23 17:27:49,831 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0 2023-11-23 17:28:03,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370700 2023-11-23 17:28:24,121 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10000, loss[loss=0.05732, simple_loss=0.07646, pruned_loss=0.009933, audio_tagging_loss=0.009157, over 14539.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09012, pruned_loss=0.01336, audio_tagging_loss=0.008978, over 3058679.18 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:28:30,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.00 vs. limit=22.5 2023-11-23 17:28:36,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2471500.0, ans=0.125 2023-11-23 17:28:45,646 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.66 vs. limit=15.0 2023-11-23 17:29:05,016 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370750 2023-11-23 17:29:26,669 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10050, loss[loss=0.05549, simple_loss=0.07045, pruned_loss=0.009892, audio_tagging_loss=0.01037, over 15050.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09096, pruned_loss=0.01344, audio_tagging_loss=0.008974, over 3062259.70 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:29:33,089 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.306e+01 9.076e+01 9.554e+01 1.175e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-23 17:29:45,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.04 vs. limit=10.0 2023-11-23 17:29:55,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2471900.0, ans=0.125 2023-11-23 17:30:03,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2471966.6666666665, ans=0.125 2023-11-23 17:30:06,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370800 2023-11-23 17:30:21,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2472033.3333333335, ans=0.0 2023-11-23 17:30:28,782 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10100, loss[loss=0.04968, simple_loss=0.0645, pruned_loss=0.006219, audio_tagging_loss=0.01121, over 14705.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09066, pruned_loss=0.01337, audio_tagging_loss=0.008994, over 3060348.22 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:30:31,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2472100.0, ans=0.125 2023-11-23 17:30:39,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2472166.6666666665, ans=0.5 2023-11-23 17:31:01,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.67 vs. limit=22.5 2023-11-23 17:31:04,672 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 17:31:09,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370850 2023-11-23 17:31:15,329 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.32 vs. limit=10.0 2023-11-23 17:31:18,150 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:31:18,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=12.0 2023-11-23 17:31:20,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2472366.6666666665, ans=0.0 2023-11-23 17:31:27,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2472366.6666666665, ans=0.125 2023-11-23 17:31:29,919 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10150, loss[loss=0.07594, simple_loss=0.104, pruned_loss=0.01761, audio_tagging_loss=0.006331, over 13676.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09132, pruned_loss=0.01353, audio_tagging_loss=0.009074, over 3063030.38 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:31:30,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2472433.3333333335, ans=0.125 2023-11-23 17:31:36,348 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.103e+01 8.204e+01 9.033e+01 9.611e+01 1.160e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-23 17:31:47,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2472500.0, ans=0.125 2023-11-23 17:31:53,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2472500.0, ans=0.95 2023-11-23 17:31:54,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2472566.6666666665, ans=0.125 2023-11-23 17:31:57,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2472566.6666666665, ans=0.125 2023-11-23 17:31:58,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2472566.6666666665, ans=0.125 2023-11-23 17:31:59,360 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:32:11,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370900 2023-11-23 17:32:31,958 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10200, loss[loss=0.06605, simple_loss=0.08006, pruned_loss=0.01297, audio_tagging_loss=0.01306, over 14610.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09106, pruned_loss=0.01349, audio_tagging_loss=0.009157, over 3052014.37 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:32:34,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0 2023-11-23 17:32:49,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2472833.3333333335, ans=0.0 2023-11-23 17:32:55,240 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:33:13,523 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 370950 2023-11-23 17:33:27,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2473033.3333333335, ans=0.125 2023-11-23 17:33:35,208 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10250, loss[loss=0.05627, simple_loss=0.0734, pruned_loss=0.01, audio_tagging_loss=0.00957, over 15181.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.0906, pruned_loss=0.01345, audio_tagging_loss=0.009241, over 3051425.07 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:33:41,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2473100.0, ans=0.125 2023-11-23 17:33:41,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2473100.0, ans=0.0 2023-11-23 17:33:42,409 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.633e+01 9.319e+01 1.005e+02 1.656e+02, threshold=1.864e+02, percent-clipped=0.0 2023-11-23 17:34:16,137 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371000 2023-11-23 17:34:18,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2473300.0, ans=0.125 2023-11-23 17:34:23,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2473366.6666666665, ans=0.09899494936611666 2023-11-23 17:34:36,435 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10300, loss[loss=0.09081, simple_loss=0.1301, pruned_loss=0.0202, audio_tagging_loss=0.005571, over 13852.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09154, pruned_loss=0.01359, audio_tagging_loss=0.009294, over 3055861.76 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:34:41,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2473433.3333333335, ans=0.0 2023-11-23 17:34:44,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2473433.3333333335, ans=0.0 2023-11-23 17:34:46,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2473433.3333333335, ans=0.125 2023-11-23 17:34:49,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2473500.0, ans=0.125 2023-11-23 17:34:53,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2473500.0, ans=0.125 2023-11-23 17:34:59,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2473500.0, ans=0.0 2023-11-23 17:35:04,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2473566.6666666665, ans=0.125 2023-11-23 17:35:08,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2473566.6666666665, ans=0.125 2023-11-23 17:35:17,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371050 2023-11-23 17:35:38,910 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10350, loss[loss=0.06967, simple_loss=0.08846, pruned_loss=0.01411, audio_tagging_loss=0.01133, over 16185.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09043, pruned_loss=0.01342, audio_tagging_loss=0.009482, over 3050985.46 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:35:46,711 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.382e+01 9.122e+01 9.955e+01 1.360e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-23 17:35:52,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2473833.3333333335, ans=0.1 2023-11-23 17:36:04,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2473900.0, ans=0.0 2023-11-23 17:36:14,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2473900.0, ans=0.125 2023-11-23 17:36:19,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2473966.6666666665, ans=0.04949747468305833 2023-11-23 17:36:20,709 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371100 2023-11-23 17:36:26,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2473966.6666666665, ans=0.1 2023-11-23 17:36:41,597 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10400, loss[loss=0.09876, simple_loss=0.1413, pruned_loss=0.02225, audio_tagging_loss=0.005883, over 14877.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09065, pruned_loss=0.01361, audio_tagging_loss=0.00942, over 3049585.04 frames. ], batch size: 52, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:36:56,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2474166.6666666665, ans=0.125 2023-11-23 17:37:03,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2474166.6666666665, ans=0.125 2023-11-23 17:37:13,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2474233.3333333335, ans=0.125 2023-11-23 17:37:22,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371150 2023-11-23 17:37:25,951 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.60 vs. limit=15.0 2023-11-23 17:37:43,937 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10450, loss[loss=0.06137, simple_loss=0.08496, pruned_loss=0.01095, audio_tagging_loss=0.007947, over 14187.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09022, pruned_loss=0.01346, audio_tagging_loss=0.009422, over 3047634.29 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:37:51,534 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.218e+01 8.367e+01 9.085e+01 9.738e+01 1.469e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-23 17:38:24,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2474633.3333333335, ans=0.125 2023-11-23 17:38:25,412 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371200 2023-11-23 17:38:40,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2474700.0, ans=0.95 2023-11-23 17:38:44,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2474700.0, ans=0.125 2023-11-23 17:38:46,220 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10500, loss[loss=0.08823, simple_loss=0.1233, pruned_loss=0.01992, audio_tagging_loss=0.006664, over 14667.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09067, pruned_loss=0.01361, audio_tagging_loss=0.009281, over 3041406.45 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:39:25,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2474966.6666666665, ans=0.1 2023-11-23 17:39:27,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371250 2023-11-23 17:39:48,787 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10550, loss[loss=0.06348, simple_loss=0.08165, pruned_loss=0.01021, audio_tagging_loss=0.01245, over 14455.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09043, pruned_loss=0.01341, audio_tagging_loss=0.009212, over 3030754.50 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:39:49,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2475100.0, ans=0.1 2023-11-23 17:39:53,051 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.17 vs. limit=15.0 2023-11-23 17:39:55,926 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.253e+01 8.407e+01 8.928e+01 9.553e+01 1.582e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-23 17:40:30,001 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371300 2023-11-23 17:40:48,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2475366.6666666665, ans=0.0 2023-11-23 17:40:51,000 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10600, loss[loss=0.09551, simple_loss=0.1322, pruned_loss=0.02507, audio_tagging_loss=0.004342, over 15778.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09048, pruned_loss=0.01346, audio_tagging_loss=0.009155, over 3037836.15 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:40:53,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2475433.3333333335, ans=0.125 2023-11-23 17:41:01,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2475433.3333333335, ans=0.0 2023-11-23 17:41:07,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2475500.0, ans=0.0 2023-11-23 17:41:31,698 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371350 2023-11-23 17:41:34,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2475633.3333333335, ans=0.125 2023-11-23 17:41:53,201 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10650, loss[loss=0.08149, simple_loss=0.1105, pruned_loss=0.01807, audio_tagging_loss=0.008174, over 14334.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09117, pruned_loss=0.01355, audio_tagging_loss=0.008978, over 3033843.98 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:42:00,371 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.195e+01 8.706e+01 9.399e+01 1.158e+02, threshold=1.741e+02, percent-clipped=0.0 2023-11-23 17:42:00,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2475766.6666666665, ans=0.125 2023-11-23 17:42:00,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2475766.6666666665, ans=0.0 2023-11-23 17:42:07,796 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.29 vs. limit=10.0 2023-11-23 17:42:11,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2475833.3333333335, ans=0.125 2023-11-23 17:42:19,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2475900.0, ans=0.1 2023-11-23 17:42:21,402 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.20 vs. limit=15.0 2023-11-23 17:42:23,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2475900.0, ans=0.0 2023-11-23 17:42:25,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2475900.0, ans=0.035 2023-11-23 17:42:34,844 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371400 2023-11-23 17:42:37,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2475966.6666666665, ans=0.1 2023-11-23 17:42:55,552 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10700, loss[loss=0.09319, simple_loss=0.1249, pruned_loss=0.02231, audio_tagging_loss=0.008442, over 15250.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09119, pruned_loss=0.01352, audio_tagging_loss=0.009002, over 3028507.98 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:43:35,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2476300.0, ans=0.1 2023-11-23 17:43:36,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371450 2023-11-23 17:43:58,062 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10750, loss[loss=0.07404, simple_loss=0.1096, pruned_loss=0.01421, audio_tagging_loss=0.005035, over 13875.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09155, pruned_loss=0.01335, audio_tagging_loss=0.008904, over 3036471.93 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:44:05,720 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.362e+01 8.891e+01 9.533e+01 1.093e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-23 17:44:18,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2476500.0, ans=0.2 2023-11-23 17:44:19,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2476500.0, ans=0.07 2023-11-23 17:44:39,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371500 2023-11-23 17:44:43,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2476633.3333333335, ans=0.0 2023-11-23 17:44:44,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2476633.3333333335, ans=0.125 2023-11-23 17:45:01,410 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10800, loss[loss=0.06822, simple_loss=0.08677, pruned_loss=0.01443, audio_tagging_loss=0.01041, over 14355.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09167, pruned_loss=0.01338, audio_tagging_loss=0.0089, over 3042121.95 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0 2023-11-23 17:45:05,787 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0 2023-11-23 17:45:07,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2476766.6666666665, ans=0.125 2023-11-23 17:45:18,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2476833.3333333335, ans=0.125 2023-11-23 17:45:18,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2476833.3333333335, ans=0.04949747468305833 2023-11-23 17:45:27,503 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 17:45:41,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371550 2023-11-23 17:45:52,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2477033.3333333335, ans=0.1 2023-11-23 17:45:52,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2477033.3333333335, ans=0.125 2023-11-23 17:45:54,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2477033.3333333335, ans=0.0 2023-11-23 17:45:57,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2477033.3333333335, ans=0.125 2023-11-23 17:46:03,195 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10850, loss[loss=0.08663, simple_loss=0.1126, pruned_loss=0.02127, audio_tagging_loss=0.009053, over 15224.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09029, pruned_loss=0.0132, audio_tagging_loss=0.008935, over 3035324.06 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 17:46:09,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2477100.0, ans=0.05 2023-11-23 17:46:14,351 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.439e+01 8.913e+01 9.439e+01 1.225e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 17:46:14,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2477166.6666666665, ans=0.125 2023-11-23 17:46:40,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2477300.0, ans=0.2 2023-11-23 17:46:43,585 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371600 2023-11-23 17:46:52,962 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 17:47:01,445 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:47:04,941 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10900, loss[loss=0.05443, simple_loss=0.07075, pruned_loss=0.00954, audio_tagging_loss=0.009513, over 14914.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09066, pruned_loss=0.01341, audio_tagging_loss=0.008965, over 3036758.38 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 17:47:12,186 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 17:47:16,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2477500.0, ans=0.1 2023-11-23 17:47:27,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2477500.0, ans=0.2 2023-11-23 17:47:42,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2477633.3333333335, ans=0.125 2023-11-23 17:47:46,039 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371650 2023-11-23 17:48:06,481 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 10950, loss[loss=0.06774, simple_loss=0.101, pruned_loss=0.008421, audio_tagging_loss=0.00883, over 15013.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09094, pruned_loss=0.01353, audio_tagging_loss=0.009017, over 3034717.10 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 17:48:17,613 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.169e+01 8.798e+01 9.612e+01 4.443e+02, threshold=1.760e+02, percent-clipped=1.0 2023-11-23 17:48:35,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2477900.0, ans=0.125 2023-11-23 17:48:47,244 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371700 2023-11-23 17:49:08,787 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11000, loss[loss=0.07809, simple_loss=0.1055, pruned_loss=0.0179, audio_tagging_loss=0.007451, over 14967.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09121, pruned_loss=0.01346, audio_tagging_loss=0.009017, over 3033582.35 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 17:49:18,891 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 17:49:22,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2478166.6666666665, ans=0.1 2023-11-23 17:49:31,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2478233.3333333335, ans=0.015 2023-11-23 17:49:31,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2478233.3333333335, ans=0.125 2023-11-23 17:49:40,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2478233.3333333335, ans=0.2 2023-11-23 17:49:44,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2478300.0, ans=0.2 2023-11-23 17:49:49,422 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371750 2023-11-23 17:50:11,112 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11050, loss[loss=0.05824, simple_loss=0.06915, pruned_loss=0.01301, audio_tagging_loss=0.01065, over 15114.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09142, pruned_loss=0.01352, audio_tagging_loss=0.008997, over 3043778.65 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 17:50:20,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.79 vs. limit=15.0 2023-11-23 17:50:21,702 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.456e+01 8.354e+01 9.034e+01 9.901e+01 1.113e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-23 17:50:28,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2478500.0, ans=0.0 2023-11-23 17:50:29,982 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.85 vs. limit=12.0 2023-11-23 17:50:42,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2478566.6666666665, ans=0.0 2023-11-23 17:50:44,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.00 vs. limit=15.0 2023-11-23 17:50:48,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2478633.3333333335, ans=0.0 2023-11-23 17:50:52,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371800 2023-11-23 17:50:58,290 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.71 vs. limit=15.0 2023-11-23 17:51:01,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2478700.0, ans=0.125 2023-11-23 17:51:07,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2478700.0, ans=0.0 2023-11-23 17:51:13,277 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11100, loss[loss=0.08634, simple_loss=0.1281, pruned_loss=0.01646, audio_tagging_loss=0.00581, over 15688.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09245, pruned_loss=0.01374, audio_tagging_loss=0.009096, over 3042995.13 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 17:51:14,960 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.07 vs. limit=10.0 2023-11-23 17:51:17,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.33 vs. limit=15.0 2023-11-23 17:51:32,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2478833.3333333335, ans=0.125 2023-11-23 17:51:34,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.65 vs. limit=10.0 2023-11-23 17:51:51,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2478966.6666666665, ans=0.125 2023-11-23 17:51:54,212 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371850 2023-11-23 17:51:55,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2478966.6666666665, ans=0.0 2023-11-23 17:52:03,455 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.23 vs. limit=22.5 2023-11-23 17:52:15,291 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11150, loss[loss=0.07344, simple_loss=0.1044, pruned_loss=0.01338, audio_tagging_loss=0.007878, over 15347.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09231, pruned_loss=0.01383, audio_tagging_loss=0.009142, over 3052187.06 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 8.0 2023-11-23 17:52:19,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2479100.0, ans=0.07 2023-11-23 17:52:26,735 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 8.417e+01 9.306e+01 9.835e+01 1.557e+02, threshold=1.861e+02, percent-clipped=0.0 2023-11-23 17:52:30,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2479166.6666666665, ans=0.125 2023-11-23 17:52:33,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.69 vs. limit=12.0 2023-11-23 17:52:34,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2479166.6666666665, ans=0.0 2023-11-23 17:52:55,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371900 2023-11-23 17:52:56,522 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0 2023-11-23 17:52:57,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2479300.0, ans=0.125 2023-11-23 17:53:00,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2479300.0, ans=0.0 2023-11-23 17:53:01,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2479300.0, ans=0.1 2023-11-23 17:53:05,987 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0 2023-11-23 17:53:17,760 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11200, loss[loss=0.05736, simple_loss=0.07306, pruned_loss=0.01018, audio_tagging_loss=0.01065, over 16550.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09089, pruned_loss=0.01357, audio_tagging_loss=0.009297, over 3053175.19 frames. ], batch size: 65, lr: 2.18e-03, grad_scale: 16.0 2023-11-23 17:53:29,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2479500.0, ans=0.125 2023-11-23 17:53:31,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.70 vs. limit=12.0 2023-11-23 17:53:45,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2479566.6666666665, ans=0.0 2023-11-23 17:53:58,923 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 371950 2023-11-23 17:54:00,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2479633.3333333335, ans=0.1 2023-11-23 17:54:08,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2479700.0, ans=0.0 2023-11-23 17:54:13,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2479700.0, ans=0.04949747468305833 2023-11-23 17:54:19,192 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11250, loss[loss=0.05629, simple_loss=0.06776, pruned_loss=0.01273, audio_tagging_loss=0.009679, over 15059.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09004, pruned_loss=0.01367, audio_tagging_loss=0.009316, over 3043393.36 frames. ], batch size: 58, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 17:54:30,511 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.457e+01 8.293e+01 8.818e+01 9.873e+01 1.266e+02, threshold=1.764e+02, percent-clipped=0.0 2023-11-23 17:54:38,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.77 vs. limit=15.0 2023-11-23 17:54:44,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2479900.0, ans=0.125 2023-11-23 17:54:50,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.71 vs. limit=22.5 2023-11-23 17:55:00,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.06 vs. limit=15.0 2023-11-23 17:55:00,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372000 2023-11-23 17:55:24,135 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11300, loss[loss=0.09236, simple_loss=0.1216, pruned_loss=0.0235, audio_tagging_loss=0.00805, over 15900.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09061, pruned_loss=0.01379, audio_tagging_loss=0.009148, over 3046650.91 frames. ], batch size: 59, lr: 2.17e-03, grad_scale: 8.0 2023-11-23 17:56:05,845 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372050 2023-11-23 17:56:15,276 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.45 vs. limit=15.0 2023-11-23 17:56:24,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2480366.6666666665, ans=0.09899494936611666 2023-11-23 17:56:27,518 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11350, loss[loss=0.05918, simple_loss=0.08067, pruned_loss=0.01081, audio_tagging_loss=0.008028, over 15708.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09098, pruned_loss=0.01397, audio_tagging_loss=0.009062, over 3047077.63 frames. ], batch size: 61, lr: 2.17e-03, grad_scale: 8.0 2023-11-23 17:56:39,395 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.419e+01 8.973e+01 9.627e+01 1.152e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-23 17:56:44,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.89 vs. limit=15.0 2023-11-23 17:56:46,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.40 vs. limit=15.0 2023-11-23 17:56:56,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2480566.6666666665, ans=0.125 2023-11-23 17:57:08,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372100 2023-11-23 17:57:27,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2480766.6666666665, ans=0.1 2023-11-23 17:57:28,755 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11400, loss[loss=0.05557, simple_loss=0.07142, pruned_loss=0.01127, audio_tagging_loss=0.008595, over 15292.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09065, pruned_loss=0.01386, audio_tagging_loss=0.009001, over 3050046.52 frames. ], batch size: 60, lr: 2.17e-03, grad_scale: 8.0 2023-11-23 17:57:47,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2480833.3333333335, ans=0.0 2023-11-23 17:58:02,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2480900.0, ans=0.2 2023-11-23 17:58:10,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372150 2023-11-23 17:58:31,098 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11450, loss[loss=0.05525, simple_loss=0.07408, pruned_loss=0.007115, audio_tagging_loss=0.01109, over 14443.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09019, pruned_loss=0.01352, audio_tagging_loss=0.00905, over 3048943.74 frames. ], batch size: 53, lr: 2.17e-03, grad_scale: 8.0 2023-11-23 17:58:32,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2481100.0, ans=0.0 2023-11-23 17:58:32,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2481100.0, ans=0.025 2023-11-23 17:58:43,920 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.130e+01 8.677e+01 9.441e+01 1.245e+02, threshold=1.735e+02, percent-clipped=0.0 2023-11-23 17:59:01,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2481233.3333333335, ans=0.0 2023-11-23 17:59:12,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372200 2023-11-23 17:59:33,999 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11500, loss[loss=0.06034, simple_loss=0.08069, pruned_loss=0.01186, audio_tagging_loss=0.00813, over 15052.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09013, pruned_loss=0.01345, audio_tagging_loss=0.009038, over 3046044.98 frames. ], batch size: 58, lr: 2.17e-03, grad_scale: 8.0 2023-11-23 17:59:52,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2481500.0, ans=0.125 2023-11-23 17:59:54,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2481500.0, ans=0.0 2023-11-23 18:00:14,409 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372250 2023-11-23 18:00:35,656 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11550, loss[loss=0.05071, simple_loss=0.05614, pruned_loss=0.01196, audio_tagging_loss=0.01068, over 15286.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09044, pruned_loss=0.01366, audio_tagging_loss=0.009003, over 3039557.15 frames. ], batch size: 59, lr: 2.17e-03, grad_scale: 8.0 2023-11-23 18:00:48,169 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.328e+01 8.420e+01 9.006e+01 9.672e+01 1.345e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 18:00:50,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2481833.3333333335, ans=0.125 2023-11-23 18:01:07,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2481900.0, ans=0.1 2023-11-23 18:01:13,619 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 18:01:17,176 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372300 2023-11-23 18:01:30,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2482033.3333333335, ans=0.125 2023-11-23 18:01:35,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2482033.3333333335, ans=0.125 2023-11-23 18:01:37,301 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11600, loss[loss=0.07189, simple_loss=0.09565, pruned_loss=0.01337, audio_tagging_loss=0.0107, over 15668.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09071, pruned_loss=0.01357, audio_tagging_loss=0.009016, over 3037144.23 frames. ], batch size: 58, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:01:42,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2482100.0, ans=0.125 2023-11-23 18:01:45,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2482100.0, ans=0.0 2023-11-23 18:01:46,385 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.45 vs. limit=22.5 2023-11-23 18:02:06,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2482233.3333333335, ans=0.125 2023-11-23 18:02:17,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372350 2023-11-23 18:02:19,983 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.77 vs. limit=10.0 2023-11-23 18:02:28,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2482366.6666666665, ans=0.2 2023-11-23 18:02:36,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2482366.6666666665, ans=0.0 2023-11-23 18:02:36,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2482366.6666666665, ans=0.125 2023-11-23 18:02:38,544 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11650, loss[loss=0.05449, simple_loss=0.07288, pruned_loss=0.008983, audio_tagging_loss=0.00907, over 14906.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09029, pruned_loss=0.01368, audio_tagging_loss=0.009077, over 3034530.21 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:02:50,779 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.302e+01 8.966e+01 9.575e+01 1.248e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-23 18:02:57,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.43 vs. limit=10.0 2023-11-23 18:03:07,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2482566.6666666665, ans=0.2 2023-11-23 18:03:08,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2482566.6666666665, ans=0.2 2023-11-23 18:03:09,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2482566.6666666665, ans=0.2 2023-11-23 18:03:18,838 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372400 2023-11-23 18:03:40,991 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11700, loss[loss=0.03847, simple_loss=0.04577, pruned_loss=0.003934, audio_tagging_loss=0.01165, over 14361.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09027, pruned_loss=0.01376, audio_tagging_loss=0.009103, over 3038334.64 frames. ], batch size: 57, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:04:22,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372450 2023-11-23 18:04:42,739 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11750, loss[loss=0.09069, simple_loss=0.1201, pruned_loss=0.02155, audio_tagging_loss=0.00909, over 16053.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.08984, pruned_loss=0.01369, audio_tagging_loss=0.009121, over 3036936.87 frames. ], batch size: 58, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:04:46,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2483100.0, ans=0.1 2023-11-23 18:04:53,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2483166.6666666665, ans=0.0 2023-11-23 18:04:54,928 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.352e+01 9.061e+01 9.652e+01 1.174e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-23 18:05:04,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2483166.6666666665, ans=0.1 2023-11-23 18:05:18,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.77 vs. limit=22.5 2023-11-23 18:05:23,293 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372500 2023-11-23 18:05:44,257 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11800, loss[loss=0.07353, simple_loss=0.09607, pruned_loss=0.01537, audio_tagging_loss=0.01013, over 15116.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09052, pruned_loss=0.01369, audio_tagging_loss=0.009145, over 3038269.79 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:05:45,680 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:05:55,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2483500.0, ans=10.0 2023-11-23 18:05:56,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.77 vs. limit=15.0 2023-11-23 18:05:59,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2483500.0, ans=0.125 2023-11-23 18:06:06,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2483500.0, ans=0.125 2023-11-23 18:06:21,479 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.54 vs. limit=22.5 2023-11-23 18:06:25,664 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372550 2023-11-23 18:06:32,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2483633.3333333335, ans=0.125 2023-11-23 18:06:44,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2483700.0, ans=0.125 2023-11-23 18:06:46,836 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11850, loss[loss=0.05093, simple_loss=0.06958, pruned_loss=0.00659, audio_tagging_loss=0.009554, over 15695.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09, pruned_loss=0.01365, audio_tagging_loss=0.009205, over 3039132.40 frames. ], batch size: 59, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:06:54,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2483766.6666666665, ans=0.1 2023-11-23 18:06:59,891 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.390e+01 9.057e+01 9.896e+01 1.267e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-23 18:07:01,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2483833.3333333335, ans=0.125 2023-11-23 18:07:03,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2483833.3333333335, ans=0.0 2023-11-23 18:07:12,074 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:07:17,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2483900.0, ans=0.1 2023-11-23 18:07:28,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372600 2023-11-23 18:07:43,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2484033.3333333335, ans=0.125 2023-11-23 18:07:44,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2484033.3333333335, ans=0.1 2023-11-23 18:07:49,522 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11900, loss[loss=0.05481, simple_loss=0.06939, pruned_loss=0.01009, audio_tagging_loss=0.01002, over 13743.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.0907, pruned_loss=0.01373, audio_tagging_loss=0.009227, over 3039897.67 frames. ], batch size: 55, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:07:55,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2484100.0, ans=0.125 2023-11-23 18:07:57,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2484100.0, ans=0.0 2023-11-23 18:08:01,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2484166.6666666665, ans=0.125 2023-11-23 18:08:20,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.57 vs. limit=10.0 2023-11-23 18:08:20,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2484233.3333333335, ans=0.0 2023-11-23 18:08:29,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2484300.0, ans=0.0 2023-11-23 18:08:30,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372650 2023-11-23 18:08:41,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.91 vs. limit=15.0 2023-11-23 18:08:51,643 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 11950, loss[loss=0.05677, simple_loss=0.07483, pruned_loss=0.008333, audio_tagging_loss=0.01102, over 14955.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09111, pruned_loss=0.01389, audio_tagging_loss=0.009294, over 3046079.42 frames. ], batch size: 57, lr: 2.17e-03, grad_scale: 16.0 2023-11-23 18:08:53,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2484433.3333333335, ans=0.2 2023-11-23 18:08:56,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2484433.3333333335, ans=0.0 2023-11-23 18:09:04,020 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.358e+01 9.149e+01 9.907e+01 1.513e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-23 18:09:11,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2484500.0, ans=0.125 2023-11-23 18:09:20,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2484566.6666666665, ans=0.0 2023-11-23 18:09:31,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372700 2023-11-23 18:09:38,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2484700.0, ans=0.1 2023-11-23 18:09:50,720 INFO [train_asr.py:1221] (3/4) Epoch 31, batch 12000, loss[loss=0.08068, simple_loss=0.1017, pruned_loss=0.01977, audio_tagging_loss=0.01007, over 14865.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09096, pruned_loss=0.01388, audio_tagging_loss=0.009436, over 3044393.88 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 32.0 2023-11-23 18:09:50,721 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 18:10:31,547 INFO [train_asr.py:1253] (3/4) Epoch 31, validation: loss=0.05782, simple_loss=0.05111, pruned_loss=0.00519, audio_tagging_loss=0.02708, over 4681554.00 frames. 2023-11-23 18:10:31,548 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 18:10:33,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2484766.6666666665, ans=0.125 2023-11-23 18:10:46,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2484833.3333333335, ans=0.125 2023-11-23 18:10:46,364 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0 2023-11-23 18:10:55,902 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.30 vs. limit=22.5 2023-11-23 18:10:56,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2484900.0, ans=0.125 2023-11-23 18:11:32,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2484926.6666666665, ans=0.1 2023-11-23 18:11:33,928 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 0, loss[loss=0.0779, simple_loss=0.08625, pruned_loss=0.01755, audio_tagging_loss=0.01722, over 15382.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.08625, pruned_loss=0.01755, audio_tagging_loss=0.01722, over 15382.00 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:11:33,929 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 18:12:06,977 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9760, 3.8143, 4.9152, 4.4217], device='cuda:3') 2023-11-23 18:12:09,581 INFO [train_asr.py:1253] (3/4) Epoch 32, validation: loss=0.05806, simple_loss=0.0511, pruned_loss=0.005184, audio_tagging_loss=0.02732, over 4681554.00 frames. 2023-11-23 18:12:09,582 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 18:12:20,792 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372750 2023-11-23 18:12:27,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2484993.3333333335, ans=0.125 2023-11-23 18:12:32,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2484993.3333333335, ans=0.0 2023-11-23 18:12:39,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2485060.0, ans=10.0 2023-11-23 18:12:40,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2485060.0, ans=0.0 2023-11-23 18:12:49,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2485126.6666666665, ans=0.1 2023-11-23 18:12:54,212 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.631e+01 9.416e+01 1.070e+02 1.612e+02, threshold=1.883e+02, percent-clipped=0.0 2023-11-23 18:13:11,371 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 50, loss[loss=0.06027, simple_loss=0.06915, pruned_loss=0.007817, audio_tagging_loss=0.01787, over 14929.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09114, pruned_loss=0.01356, audio_tagging_loss=0.01738, over 696435.64 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:13:19,580 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=15.0 2023-11-23 18:13:22,512 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372800 2023-11-23 18:13:29,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2485326.6666666665, ans=0.2 2023-11-23 18:13:35,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2485393.3333333335, ans=0.125 2023-11-23 18:13:52,155 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0 2023-11-23 18:14:07,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.82 vs. limit=15.0 2023-11-23 18:14:10,746 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.91 vs. limit=15.0 2023-11-23 18:14:13,831 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 100, loss[loss=0.07431, simple_loss=0.08626, pruned_loss=0.01765, audio_tagging_loss=0.01353, over 15351.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.0933, pruned_loss=0.01414, audio_tagging_loss=0.01666, over 1210141.21 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:14:24,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2485593.3333333335, ans=0.0 2023-11-23 18:14:25,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372850 2023-11-23 18:14:44,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2485726.6666666665, ans=0.2 2023-11-23 18:14:53,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2485793.3333333335, ans=0.035 2023-11-23 18:14:59,365 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.698e+01 8.892e+01 9.428e+01 1.026e+02 1.429e+02, threshold=1.886e+02, percent-clipped=0.0 2023-11-23 18:14:59,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2485793.3333333335, ans=0.0 2023-11-23 18:14:59,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2485793.3333333335, ans=0.125 2023-11-23 18:15:17,204 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 150, loss[loss=0.07104, simple_loss=0.08801, pruned_loss=0.01511, audio_tagging_loss=0.01192, over 15558.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.0915, pruned_loss=0.01355, audio_tagging_loss=0.01508, over 1616006.95 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:15:28,451 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372900 2023-11-23 18:15:54,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2486126.6666666665, ans=0.125 2023-11-23 18:15:56,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2486126.6666666665, ans=0.2 2023-11-23 18:16:04,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2486126.6666666665, ans=0.1 2023-11-23 18:16:18,936 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 200, loss[loss=0.06967, simple_loss=0.09493, pruned_loss=0.01266, audio_tagging_loss=0.009544, over 15498.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.092, pruned_loss=0.01376, audio_tagging_loss=0.01334, over 1941822.22 frames. ], batch size: 59, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:16:21,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2486260.0, ans=0.0 2023-11-23 18:16:21,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2486260.0, ans=0.125 2023-11-23 18:16:25,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2486260.0, ans=0.125 2023-11-23 18:16:30,284 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 372950 2023-11-23 18:16:39,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2486326.6666666665, ans=0.2 2023-11-23 18:16:46,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2486393.3333333335, ans=0.95 2023-11-23 18:16:47,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2486393.3333333335, ans=0.2 2023-11-23 18:16:47,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2486393.3333333335, ans=0.125 2023-11-23 18:16:50,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2486393.3333333335, ans=0.0 2023-11-23 18:16:50,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2486393.3333333335, ans=0.0 2023-11-23 18:16:55,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.00 vs. limit=15.0 2023-11-23 18:17:04,693 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.397e+01 8.534e+01 9.090e+01 9.831e+01 1.209e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-23 18:17:04,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2486460.0, ans=0.125 2023-11-23 18:17:20,650 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 250, loss[loss=0.06632, simple_loss=0.09939, pruned_loss=0.01175, audio_tagging_loss=0.004878, over 15648.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09311, pruned_loss=0.01397, audio_tagging_loss=0.01191, over 2189382.64 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:17:31,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373000 2023-11-23 18:18:05,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2486793.3333333335, ans=0.0 2023-11-23 18:18:22,659 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 300, loss[loss=0.06286, simple_loss=0.07979, pruned_loss=0.01193, audio_tagging_loss=0.01104, over 16924.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09339, pruned_loss=0.0139, audio_tagging_loss=0.01101, over 2381165.17 frames. ], batch size: 65, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:18:34,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373050 2023-11-23 18:18:58,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2487126.6666666665, ans=0.0 2023-11-23 18:19:08,697 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.767e+01 9.315e+01 9.987e+01 1.181e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-23 18:19:24,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0 2023-11-23 18:19:24,683 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 350, loss[loss=0.07399, simple_loss=0.09834, pruned_loss=0.01709, audio_tagging_loss=0.007727, over 15345.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09327, pruned_loss=0.01395, audio_tagging_loss=0.01041, over 2531454.04 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:19:36,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373100 2023-11-23 18:19:39,134 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.73 vs. limit=10.0 2023-11-23 18:20:00,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=2487460.0, ans=0.02 2023-11-23 18:20:13,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2487526.6666666665, ans=0.0 2023-11-23 18:20:17,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2487526.6666666665, ans=0.0 2023-11-23 18:20:26,831 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 400, loss[loss=0.07171, simple_loss=0.09885, pruned_loss=0.01422, audio_tagging_loss=0.008065, over 14420.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09369, pruned_loss=0.01398, audio_tagging_loss=0.009997, over 2647040.89 frames. ], batch size: 54, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:20:37,971 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373150 2023-11-23 18:20:43,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2487660.0, ans=0.09899494936611666 2023-11-23 18:20:58,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2487726.6666666665, ans=0.1 2023-11-23 18:21:11,955 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.48 vs. limit=15.0 2023-11-23 18:21:12,520 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.510e+01 9.091e+01 9.716e+01 1.253e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-23 18:21:20,402 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0 2023-11-23 18:21:28,546 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 450, loss[loss=0.07946, simple_loss=0.106, pruned_loss=0.01631, audio_tagging_loss=0.01014, over 15505.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09329, pruned_loss=0.01399, audio_tagging_loss=0.009673, over 2732460.93 frames. ], batch size: 56, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:21:40,003 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373200 2023-11-23 18:21:48,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.81 vs. limit=15.0 2023-11-23 18:21:50,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2487993.3333333335, ans=0.125 2023-11-23 18:22:11,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2488126.6666666665, ans=0.125 2023-11-23 18:22:12,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2488126.6666666665, ans=0.125 2023-11-23 18:22:13,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2488126.6666666665, ans=0.5 2023-11-23 18:22:22,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2488193.3333333335, ans=0.1 2023-11-23 18:22:31,185 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 500, loss[loss=0.05166, simple_loss=0.06146, pruned_loss=0.0121, audio_tagging_loss=0.008833, over 13477.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09183, pruned_loss=0.01378, audio_tagging_loss=0.009491, over 2791759.55 frames. ], batch size: 53, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:22:31,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2488260.0, ans=0.2 2023-11-23 18:22:42,948 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373250 2023-11-23 18:22:57,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2488393.3333333335, ans=0.07 2023-11-23 18:23:01,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2488393.3333333335, ans=0.125 2023-11-23 18:23:14,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2488460.0, ans=0.125 2023-11-23 18:23:17,398 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.516e+01 9.137e+01 9.960e+01 1.270e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-23 18:23:34,141 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 550, loss[loss=0.05554, simple_loss=0.0715, pruned_loss=0.01161, audio_tagging_loss=0.008178, over 14831.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.091, pruned_loss=0.01355, audio_tagging_loss=0.009519, over 2845498.56 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 32.0 2023-11-23 18:23:35,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2488593.3333333335, ans=0.1 2023-11-23 18:23:44,853 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373300 2023-11-23 18:24:09,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2488793.3333333335, ans=0.1 2023-11-23 18:24:14,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff3.min_abs, batch_count=2488793.3333333335, ans=0.2 2023-11-23 18:24:24,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2488860.0, ans=0.1 2023-11-23 18:24:36,001 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 600, loss[loss=0.06533, simple_loss=0.09074, pruned_loss=0.01084, audio_tagging_loss=0.009114, over 13868.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09117, pruned_loss=0.01365, audio_tagging_loss=0.009579, over 2891152.70 frames. ], batch size: 53, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:24:46,729 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373350 2023-11-23 18:24:50,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2488993.3333333335, ans=0.0 2023-11-23 18:24:58,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2488993.3333333335, ans=0.125 2023-11-23 18:25:01,529 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.20 vs. limit=22.5 2023-11-23 18:25:06,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2489060.0, ans=0.125 2023-11-23 18:25:16,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.64 vs. limit=15.0 2023-11-23 18:25:23,234 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.713e+01 9.332e+01 1.014e+02 1.298e+02, threshold=1.866e+02, percent-clipped=0.0 2023-11-23 18:25:38,133 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 650, loss[loss=0.08187, simple_loss=0.1195, pruned_loss=0.01576, audio_tagging_loss=0.00637, over 14864.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09155, pruned_loss=0.01362, audio_tagging_loss=0.009551, over 2918542.81 frames. ], batch size: 56, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:25:46,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2489260.0, ans=0.125 2023-11-23 18:25:50,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373400 2023-11-23 18:26:01,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2489326.6666666665, ans=0.0 2023-11-23 18:26:09,554 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.54 vs. limit=15.0 2023-11-23 18:26:13,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2489393.3333333335, ans=0.0 2023-11-23 18:26:39,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2489526.6666666665, ans=0.125 2023-11-23 18:26:41,069 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 700, loss[loss=0.06619, simple_loss=0.09233, pruned_loss=0.01175, audio_tagging_loss=0.008274, over 16040.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.0921, pruned_loss=0.01359, audio_tagging_loss=0.009367, over 2943188.47 frames. ], batch size: 59, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:26:42,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2489593.3333333335, ans=0.125 2023-11-23 18:26:51,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373450 2023-11-23 18:27:03,814 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.26 vs. limit=6.0 2023-11-23 18:27:13,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2489726.6666666665, ans=0.2 2023-11-23 18:27:17,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2489793.3333333335, ans=0.125 2023-11-23 18:27:21,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2489793.3333333335, ans=0.1 2023-11-23 18:27:29,303 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.089e+01 8.956e+01 9.594e+01 1.307e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 18:27:42,831 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 750, loss[loss=0.07934, simple_loss=0.1005, pruned_loss=0.01649, audio_tagging_loss=0.0126, over 14883.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09193, pruned_loss=0.01348, audio_tagging_loss=0.009377, over 2968802.32 frames. ], batch size: 56, lr: 2.14e-03, grad_scale: 8.0 2023-11-23 18:27:53,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373500 2023-11-23 18:28:03,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2489993.3333333335, ans=0.125 2023-11-23 18:28:10,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.93 vs. limit=10.0 2023-11-23 18:28:12,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2490060.0, ans=0.2 2023-11-23 18:28:26,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2490126.6666666665, ans=0.07 2023-11-23 18:28:45,050 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 800, loss[loss=0.06933, simple_loss=0.1008, pruned_loss=0.01025, audio_tagging_loss=0.008675, over 15913.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09306, pruned_loss=0.01367, audio_tagging_loss=0.009354, over 2985538.89 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:28:55,836 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373550 2023-11-23 18:29:05,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2490326.6666666665, ans=0.035 2023-11-23 18:29:07,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2490326.6666666665, ans=0.04949747468305833 2023-11-23 18:29:12,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2490393.3333333335, ans=0.1 2023-11-23 18:29:33,501 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.422e+01 8.967e+01 9.780e+01 1.208e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-23 18:29:46,999 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 850, loss[loss=0.06172, simple_loss=0.07658, pruned_loss=0.01242, audio_tagging_loss=0.011, over 14590.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09363, pruned_loss=0.01393, audio_tagging_loss=0.009357, over 2998759.03 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:29:56,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2490593.3333333335, ans=0.1 2023-11-23 18:29:57,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2490593.3333333335, ans=0.1 2023-11-23 18:29:58,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373600 2023-11-23 18:30:23,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2490793.3333333335, ans=0.0 2023-11-23 18:30:33,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2490793.3333333335, ans=0.0 2023-11-23 18:30:48,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2490926.6666666665, ans=0.05 2023-11-23 18:30:49,687 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 900, loss[loss=0.08479, simple_loss=0.1234, pruned_loss=0.01741, audio_tagging_loss=0.005679, over 15520.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09328, pruned_loss=0.01389, audio_tagging_loss=0.009416, over 3009341.13 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:30:57,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2490926.6666666665, ans=0.125 2023-11-23 18:31:01,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373650 2023-11-23 18:31:08,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.62 vs. limit=15.0 2023-11-23 18:31:33,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2491126.6666666665, ans=0.2 2023-11-23 18:31:38,019 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.169e+01 8.291e+01 8.989e+01 9.605e+01 1.191e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-23 18:31:47,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2491193.3333333335, ans=0.125 2023-11-23 18:31:51,786 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 950, loss[loss=0.07127, simple_loss=0.09806, pruned_loss=0.01462, audio_tagging_loss=0.007612, over 15929.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09354, pruned_loss=0.01394, audio_tagging_loss=0.009311, over 3024876.45 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 16.0 2023-11-23 18:31:57,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2491260.0, ans=0.0 2023-11-23 18:32:03,021 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373700 2023-11-23 18:32:07,128 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.53 vs. limit=10.0 2023-11-23 18:32:16,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2491393.3333333335, ans=0.125 2023-11-23 18:32:20,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2491393.3333333335, ans=0.125 2023-11-23 18:32:41,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2491526.6666666665, ans=0.0 2023-11-23 18:32:44,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2491526.6666666665, ans=0.0 2023-11-23 18:32:44,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2491526.6666666665, ans=0.125 2023-11-23 18:32:53,759 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1000, loss[loss=0.07097, simple_loss=0.09653, pruned_loss=0.01374, audio_tagging_loss=0.008962, over 15933.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09324, pruned_loss=0.0139, audio_tagging_loss=0.009083, over 3027678.81 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:32:54,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.02 vs. limit=15.0 2023-11-23 18:32:56,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2491593.3333333335, ans=0.125 2023-11-23 18:32:57,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2491593.3333333335, ans=0.125 2023-11-23 18:32:59,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2491593.3333333335, ans=22.5 2023-11-23 18:33:05,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373750 2023-11-23 18:33:18,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2491726.6666666665, ans=0.125 2023-11-23 18:33:20,538 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 18:33:28,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2491726.6666666665, ans=0.125 2023-11-23 18:33:42,323 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.226e+01 8.316e+01 8.954e+01 9.512e+01 1.152e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 18:33:55,925 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1050, loss[loss=0.09203, simple_loss=0.1187, pruned_loss=0.0235, audio_tagging_loss=0.009154, over 15341.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09324, pruned_loss=0.01396, audio_tagging_loss=0.008956, over 3029047.36 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:34:04,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2491926.6666666665, ans=0.2 2023-11-23 18:34:04,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2491926.6666666665, ans=0.0 2023-11-23 18:34:07,879 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373800 2023-11-23 18:34:19,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2491993.3333333335, ans=0.125 2023-11-23 18:34:30,866 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:34:30,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2492060.0, ans=0.05 2023-11-23 18:34:32,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2492060.0, ans=0.0 2023-11-23 18:34:41,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2492126.6666666665, ans=0.2 2023-11-23 18:34:59,575 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1100, loss[loss=0.04751, simple_loss=0.06485, pruned_loss=0.006484, audio_tagging_loss=0.008605, over 15126.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09177, pruned_loss=0.01373, audio_tagging_loss=0.009007, over 3032802.85 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:35:02,540 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 18:35:10,854 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373850 2023-11-23 18:35:24,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2492393.3333333335, ans=0.125 2023-11-23 18:35:36,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2492460.0, ans=0.2 2023-11-23 18:35:48,090 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.110e+01 8.335e+01 9.013e+01 9.723e+01 1.742e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-23 18:35:54,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2492526.6666666665, ans=0.125 2023-11-23 18:36:02,010 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1150, loss[loss=0.04985, simple_loss=0.07109, pruned_loss=0.008294, audio_tagging_loss=0.006009, over 16470.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09153, pruned_loss=0.01366, audio_tagging_loss=0.009079, over 3044113.43 frames. ], batch size: 65, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:36:13,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373900 2023-11-23 18:36:24,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2492660.0, ans=0.0 2023-11-23 18:36:39,854 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:36:43,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2492793.3333333335, ans=0.0 2023-11-23 18:36:49,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2492793.3333333335, ans=0.5 2023-11-23 18:36:50,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2492860.0, ans=0.1 2023-11-23 18:36:55,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.57 vs. limit=15.0 2023-11-23 18:37:04,191 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1200, loss[loss=0.0659, simple_loss=0.08512, pruned_loss=0.01352, audio_tagging_loss=0.009823, over 13775.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09192, pruned_loss=0.01374, audio_tagging_loss=0.009051, over 3043745.93 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:37:16,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 373950 2023-11-23 18:37:28,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2493060.0, ans=0.0 2023-11-23 18:37:52,689 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.465e+01 8.983e+01 9.576e+01 1.363e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 18:38:03,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2493193.3333333335, ans=0.125 2023-11-23 18:38:06,964 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1250, loss[loss=0.06779, simple_loss=0.0877, pruned_loss=0.01379, audio_tagging_loss=0.01015, over 15139.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09136, pruned_loss=0.01361, audio_tagging_loss=0.009023, over 3040085.78 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:38:18,457 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374000 2023-11-23 18:38:23,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2493326.6666666665, ans=0.1 2023-11-23 18:38:47,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2493460.0, ans=0.125 2023-11-23 18:39:00,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2493526.6666666665, ans=0.125 2023-11-23 18:39:02,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2493526.6666666665, ans=0.0 2023-11-23 18:39:09,329 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1300, loss[loss=0.07354, simple_loss=0.1037, pruned_loss=0.01503, audio_tagging_loss=0.00666, over 16386.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09087, pruned_loss=0.01351, audio_tagging_loss=0.008957, over 3037343.60 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:39:10,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2493593.3333333335, ans=0.125 2023-11-23 18:39:10,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2493593.3333333335, ans=0.125 2023-11-23 18:39:20,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374050 2023-11-23 18:39:22,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2493660.0, ans=0.1 2023-11-23 18:39:24,405 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:39:38,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0 2023-11-23 18:39:59,215 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.307e+01 8.471e+01 8.997e+01 9.654e+01 1.238e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 18:40:04,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2493860.0, ans=0.0 2023-11-23 18:40:10,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2493926.6666666665, ans=0.125 2023-11-23 18:40:11,611 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1350, loss[loss=0.04288, simple_loss=0.05006, pruned_loss=0.006772, audio_tagging_loss=0.01108, over 13234.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09052, pruned_loss=0.01332, audio_tagging_loss=0.009019, over 3042545.52 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:40:18,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2493926.6666666665, ans=0.125 2023-11-23 18:40:22,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374100 2023-11-23 18:40:23,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2493993.3333333335, ans=0.0 2023-11-23 18:40:25,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2493993.3333333335, ans=0.1 2023-11-23 18:40:34,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2493993.3333333335, ans=0.025 2023-11-23 18:40:41,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2494060.0, ans=0.0 2023-11-23 18:40:44,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2494060.0, ans=0.0 2023-11-23 18:40:45,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2494060.0, ans=0.125 2023-11-23 18:40:55,043 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 18:41:13,475 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1400, loss[loss=0.09889, simple_loss=0.1292, pruned_loss=0.02605, audio_tagging_loss=0.00823, over 13543.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09084, pruned_loss=0.01338, audio_tagging_loss=0.009054, over 3048373.91 frames. ], batch size: 50, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:41:14,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2494260.0, ans=0.125 2023-11-23 18:41:15,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2494260.0, ans=0.125 2023-11-23 18:41:22,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2494260.0, ans=0.0 2023-11-23 18:41:25,514 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374150 2023-11-23 18:41:33,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2494326.6666666665, ans=15.0 2023-11-23 18:41:39,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2494393.3333333335, ans=0.2 2023-11-23 18:41:43,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2494393.3333333335, ans=0.125 2023-11-23 18:41:45,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2494393.3333333335, ans=0.125 2023-11-23 18:42:01,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2494460.0, ans=0.0 2023-11-23 18:42:03,223 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.059e+01 8.437e+01 8.832e+01 9.783e+01 1.644e+02, threshold=1.766e+02, percent-clipped=0.0 2023-11-23 18:42:09,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.50 vs. limit=12.0 2023-11-23 18:42:13,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2494526.6666666665, ans=0.1 2023-11-23 18:42:13,980 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.59 vs. limit=22.5 2023-11-23 18:42:15,818 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1450, loss[loss=0.05505, simple_loss=0.07082, pruned_loss=0.01018, audio_tagging_loss=0.009452, over 15642.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09145, pruned_loss=0.01349, audio_tagging_loss=0.009078, over 3045114.50 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:42:24,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2494593.3333333335, ans=0.1 2023-11-23 18:42:27,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374200 2023-11-23 18:42:31,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2494660.0, ans=0.125 2023-11-23 18:42:33,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2494660.0, ans=0.1 2023-11-23 18:42:51,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2494726.6666666665, ans=0.0 2023-11-23 18:42:57,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2494793.3333333335, ans=0.125 2023-11-23 18:43:08,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=2494860.0, ans=15.0 2023-11-23 18:43:17,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2494926.6666666665, ans=0.1 2023-11-23 18:43:18,569 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1500, loss[loss=0.08334, simple_loss=0.1122, pruned_loss=0.01702, audio_tagging_loss=0.0102, over 15628.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09217, pruned_loss=0.01366, audio_tagging_loss=0.009142, over 3045077.14 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:43:24,651 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:43:27,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2494926.6666666665, ans=0.2 2023-11-23 18:43:29,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374250 2023-11-23 18:43:37,359 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:44:00,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=15.0 2023-11-23 18:44:02,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2495126.6666666665, ans=0.2 2023-11-23 18:44:06,848 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.50 vs. limit=15.0 2023-11-23 18:44:08,638 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.534e+01 8.386e+01 9.077e+01 9.732e+01 1.149e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-23 18:44:21,222 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1550, loss[loss=0.06511, simple_loss=0.08615, pruned_loss=0.012, audio_tagging_loss=0.01003, over 14945.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09285, pruned_loss=0.01386, audio_tagging_loss=0.009205, over 3043337.47 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 18:44:21,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2495260.0, ans=0.0 2023-11-23 18:44:32,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374300 2023-11-23 18:44:36,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2495326.6666666665, ans=0.125 2023-11-23 18:44:48,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2495393.3333333335, ans=0.0 2023-11-23 18:44:58,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2495460.0, ans=0.125 2023-11-23 18:45:04,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0 2023-11-23 18:45:21,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2495526.6666666665, ans=0.1 2023-11-23 18:45:23,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2495593.3333333335, ans=0.07 2023-11-23 18:45:24,453 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1600, loss[loss=0.05924, simple_loss=0.07408, pruned_loss=0.01246, audio_tagging_loss=0.009751, over 15462.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09345, pruned_loss=0.01386, audio_tagging_loss=0.009134, over 3045181.69 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:45:27,437 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.03 vs. limit=22.5 2023-11-23 18:45:35,057 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374350 2023-11-23 18:45:35,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.52 vs. limit=10.0 2023-11-23 18:45:37,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2495660.0, ans=0.125 2023-11-23 18:45:45,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2495660.0, ans=0.125 2023-11-23 18:45:46,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2495660.0, ans=0.125 2023-11-23 18:45:50,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2495726.6666666665, ans=0.1 2023-11-23 18:46:01,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2495793.3333333335, ans=0.2 2023-11-23 18:46:14,057 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.412e+01 9.043e+01 9.774e+01 1.173e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 18:46:25,940 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1650, loss[loss=0.07533, simple_loss=0.1043, pruned_loss=0.01661, audio_tagging_loss=0.006586, over 14874.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09283, pruned_loss=0.01373, audio_tagging_loss=0.009156, over 3044737.07 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:46:37,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374400 2023-11-23 18:46:46,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2495993.3333333335, ans=0.125 2023-11-23 18:46:55,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2496060.0, ans=0.2 2023-11-23 18:47:22,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2496193.3333333335, ans=0.125 2023-11-23 18:47:25,554 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.35 vs. limit=15.0 2023-11-23 18:47:28,987 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1700, loss[loss=0.06878, simple_loss=0.09863, pruned_loss=0.01293, audio_tagging_loss=0.006537, over 16417.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09251, pruned_loss=0.01365, audio_tagging_loss=0.009092, over 3043494.56 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:47:39,717 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374450 2023-11-23 18:48:06,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2496460.0, ans=0.125 2023-11-23 18:48:18,895 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.196e+01 8.417e+01 9.193e+01 9.971e+01 1.264e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-23 18:48:19,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2496526.6666666665, ans=0.0 2023-11-23 18:48:20,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2496526.6666666665, ans=0.05 2023-11-23 18:48:31,083 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1750, loss[loss=0.07721, simple_loss=0.09803, pruned_loss=0.01433, audio_tagging_loss=0.01387, over 13879.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09245, pruned_loss=0.01367, audio_tagging_loss=0.009147, over 3039814.46 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:48:35,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2496593.3333333335, ans=0.125 2023-11-23 18:48:42,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374500 2023-11-23 18:48:51,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2496660.0, ans=0.1 2023-11-23 18:48:57,617 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:49:01,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2496726.6666666665, ans=0.0 2023-11-23 18:49:14,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2496793.3333333335, ans=0.0 2023-11-23 18:49:33,575 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1800, loss[loss=0.07199, simple_loss=0.1021, pruned_loss=0.01203, audio_tagging_loss=0.008925, over 14954.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09275, pruned_loss=0.01371, audio_tagging_loss=0.00914, over 3042065.82 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:49:38,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2496926.6666666665, ans=0.1 2023-11-23 18:49:45,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374550 2023-11-23 18:50:08,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2497060.0, ans=0.125 2023-11-23 18:50:17,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2497126.6666666665, ans=0.04949747468305833 2023-11-23 18:50:21,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2497126.6666666665, ans=0.125 2023-11-23 18:50:23,587 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.228e+01 8.763e+01 9.519e+01 1.190e+02, threshold=1.753e+02, percent-clipped=0.0 2023-11-23 18:50:26,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.32 vs. limit=15.0 2023-11-23 18:50:30,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.89 vs. limit=22.5 2023-11-23 18:50:35,991 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1850, loss[loss=0.07069, simple_loss=0.1051, pruned_loss=0.01125, audio_tagging_loss=0.006906, over 15716.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09302, pruned_loss=0.01381, audio_tagging_loss=0.009018, over 3042020.76 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:50:47,281 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374600 2023-11-23 18:50:47,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2497326.6666666665, ans=0.125 2023-11-23 18:50:53,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2497326.6666666665, ans=0.125 2023-11-23 18:50:54,376 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 18:51:29,624 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.14 vs. limit=6.0 2023-11-23 18:51:38,705 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1900, loss[loss=0.07366, simple_loss=0.1028, pruned_loss=0.01526, audio_tagging_loss=0.007026, over 16165.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09312, pruned_loss=0.01386, audio_tagging_loss=0.008909, over 3045973.21 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:51:44,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2497593.3333333335, ans=0.0 2023-11-23 18:51:50,217 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374650 2023-11-23 18:51:54,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2497660.0, ans=0.0 2023-11-23 18:52:29,654 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.323e+01 8.306e+01 8.793e+01 9.456e+01 1.356e+02, threshold=1.759e+02, percent-clipped=0.0 2023-11-23 18:52:41,806 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 1950, loss[loss=0.05289, simple_loss=0.06931, pruned_loss=0.008693, audio_tagging_loss=0.009544, over 15151.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09125, pruned_loss=0.01356, audio_tagging_loss=0.008977, over 3043622.45 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:52:45,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2497926.6666666665, ans=0.0 2023-11-23 18:52:46,051 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.40 vs. limit=15.0 2023-11-23 18:52:48,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2497926.6666666665, ans=0.125 2023-11-23 18:52:51,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2497926.6666666665, ans=0.1 2023-11-23 18:52:53,911 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374700 2023-11-23 18:53:06,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2498060.0, ans=0.125 2023-11-23 18:53:39,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2498193.3333333335, ans=0.0 2023-11-23 18:53:45,421 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2000, loss[loss=0.05721, simple_loss=0.08226, pruned_loss=0.01021, audio_tagging_loss=0.005878, over 14564.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09102, pruned_loss=0.0134, audio_tagging_loss=0.008964, over 3042959.30 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:53:46,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2498260.0, ans=0.125 2023-11-23 18:53:56,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374750 2023-11-23 18:53:58,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2498326.6666666665, ans=0.0 2023-11-23 18:54:11,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2498393.3333333335, ans=0.0 2023-11-23 18:54:27,431 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.96 vs. limit=22.5 2023-11-23 18:54:35,234 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.533e+01 8.441e+01 9.066e+01 9.690e+01 1.160e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-23 18:54:44,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2498526.6666666665, ans=0.1 2023-11-23 18:54:48,068 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2050, loss[loss=0.08064, simple_loss=0.0979, pruned_loss=0.01968, audio_tagging_loss=0.01201, over 14405.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09079, pruned_loss=0.01337, audio_tagging_loss=0.008937, over 3044351.67 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:54:49,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2498593.3333333335, ans=0.125 2023-11-23 18:54:58,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2498593.3333333335, ans=0.0 2023-11-23 18:54:59,445 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374800 2023-11-23 18:55:12,891 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.08 vs. limit=6.0 2023-11-23 18:55:23,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2498726.6666666665, ans=0.125 2023-11-23 18:55:25,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.23 vs. limit=15.0 2023-11-23 18:55:50,648 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2100, loss[loss=0.05356, simple_loss=0.07139, pruned_loss=0.0087, audio_tagging_loss=0.009161, over 14508.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.0904, pruned_loss=0.0132, audio_tagging_loss=0.008934, over 3048078.46 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:55:54,893 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.67 vs. limit=15.0 2023-11-23 18:56:01,928 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374850 2023-11-23 18:56:18,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2499060.0, ans=0.0 2023-11-23 18:56:27,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0 2023-11-23 18:56:39,982 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.878e+01 8.622e+01 9.048e+01 9.766e+01 1.225e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 18:56:40,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2499193.3333333335, ans=0.2 2023-11-23 18:56:52,568 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2150, loss[loss=0.07757, simple_loss=0.1025, pruned_loss=0.01617, audio_tagging_loss=0.01017, over 16689.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09098, pruned_loss=0.0133, audio_tagging_loss=0.008926, over 3055602.84 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:56:56,219 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0 2023-11-23 18:57:04,356 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374900 2023-11-23 18:57:20,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2499393.3333333335, ans=0.1 2023-11-23 18:57:24,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2499393.3333333335, ans=0.0 2023-11-23 18:57:24,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.99 vs. limit=22.5 2023-11-23 18:57:26,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2499393.3333333335, ans=0.0 2023-11-23 18:57:27,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2499393.3333333335, ans=0.2 2023-11-23 18:57:29,799 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 18:57:44,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2499526.6666666665, ans=0.125 2023-11-23 18:57:46,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2499526.6666666665, ans=0.125 2023-11-23 18:57:54,954 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2200, loss[loss=0.06365, simple_loss=0.08916, pruned_loss=0.009156, audio_tagging_loss=0.009908, over 14883.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09152, pruned_loss=0.01357, audio_tagging_loss=0.008923, over 3059003.79 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:58:05,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2499593.3333333335, ans=0.1 2023-11-23 18:58:06,036 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 374950 2023-11-23 18:58:09,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2499660.0, ans=0.2 2023-11-23 18:58:37,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2499793.3333333335, ans=0.125 2023-11-23 18:58:44,341 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.234e+01 8.395e+01 9.074e+01 9.645e+01 1.441e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-23 18:58:49,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2499860.0, ans=0.125 2023-11-23 18:58:50,254 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.17 vs. limit=6.0 2023-11-23 18:58:56,710 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2250, loss[loss=0.07205, simple_loss=0.08868, pruned_loss=0.0172, audio_tagging_loss=0.01052, over 15569.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09082, pruned_loss=0.01347, audio_tagging_loss=0.009061, over 3047339.46 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 18:58:56,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2499926.6666666665, ans=0.125 2023-11-23 18:59:07,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375000 2023-11-23 18:59:40,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2500126.6666666665, ans=0.0 2023-11-23 18:59:44,421 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.47 vs. limit=12.0 2023-11-23 18:59:48,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2500193.3333333335, ans=0.125 2023-11-23 18:59:58,410 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2300, loss[loss=0.06936, simple_loss=0.09369, pruned_loss=0.01293, audio_tagging_loss=0.00959, over 14576.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09061, pruned_loss=0.01329, audio_tagging_loss=0.009045, over 3045264.39 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:00:10,247 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375050 2023-11-23 19:00:11,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2500326.6666666665, ans=0.1 2023-11-23 19:00:34,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2500460.0, ans=0.0 2023-11-23 19:00:43,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2500460.0, ans=0.0 2023-11-23 19:00:43,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2500460.0, ans=0.125 2023-11-23 19:00:47,718 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.606e+01 9.166e+01 9.861e+01 1.207e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-23 19:00:52,510 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:01:00,770 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2350, loss[loss=0.05484, simple_loss=0.06521, pruned_loss=0.01004, audio_tagging_loss=0.01219, over 13965.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09105, pruned_loss=0.01329, audio_tagging_loss=0.009134, over 3050077.76 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:01:05,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.44 vs. limit=15.0 2023-11-23 19:01:06,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2500593.3333333335, ans=0.125 2023-11-23 19:01:12,037 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375100 2023-11-23 19:01:14,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2500660.0, ans=0.2 2023-11-23 19:01:35,886 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.05 vs. limit=15.0 2023-11-23 19:01:43,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2500793.3333333335, ans=0.125 2023-11-23 19:01:57,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2500860.0, ans=0.0 2023-11-23 19:02:03,154 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2400, loss[loss=0.05643, simple_loss=0.06439, pruned_loss=0.01179, audio_tagging_loss=0.01245, over 14812.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09099, pruned_loss=0.01335, audio_tagging_loss=0.009225, over 3051050.51 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:02:13,894 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375150 2023-11-23 19:02:17,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2500993.3333333335, ans=0.125 2023-11-23 19:02:20,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2500993.3333333335, ans=0.125 2023-11-23 19:02:40,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2501126.6666666665, ans=0.0 2023-11-23 19:02:49,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2501126.6666666665, ans=0.2 2023-11-23 19:02:54,391 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.466e+01 8.908e+01 9.614e+01 2.076e+02, threshold=1.782e+02, percent-clipped=1.0 2023-11-23 19:03:05,706 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2450, loss[loss=0.06822, simple_loss=0.09086, pruned_loss=0.01465, audio_tagging_loss=0.008133, over 15562.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09076, pruned_loss=0.01334, audio_tagging_loss=0.00928, over 3049759.73 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:03:16,928 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375200 2023-11-23 19:03:55,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2501526.6666666665, ans=0.0 2023-11-23 19:03:57,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2501526.6666666665, ans=0.1 2023-11-23 19:04:07,618 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2500, loss[loss=0.07274, simple_loss=0.09247, pruned_loss=0.01699, audio_tagging_loss=0.009514, over 14275.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09087, pruned_loss=0.01342, audio_tagging_loss=0.009324, over 3048141.71 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:04:19,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375250 2023-11-23 19:04:19,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2501660.0, ans=0.125 2023-11-23 19:04:39,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2501726.6666666665, ans=0.2 2023-11-23 19:04:42,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2501726.6666666665, ans=0.1 2023-11-23 19:04:43,075 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.25 vs. limit=15.0 2023-11-23 19:04:55,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.23 vs. limit=12.0 2023-11-23 19:05:01,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.221e+01 8.437e+01 9.142e+01 9.814e+01 1.303e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-23 19:05:06,213 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.06 vs. limit=15.0 2023-11-23 19:05:10,393 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2550, loss[loss=0.05547, simple_loss=0.07174, pruned_loss=0.009413, audio_tagging_loss=0.01018, over 15515.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09106, pruned_loss=0.01355, audio_tagging_loss=0.009276, over 3050054.88 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 8.0 2023-11-23 19:05:13,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2501926.6666666665, ans=0.125 2023-11-23 19:05:16,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2501926.6666666665, ans=0.05 2023-11-23 19:05:19,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2501926.6666666665, ans=0.125 2023-11-23 19:05:21,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375300 2023-11-23 19:05:50,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2502126.6666666665, ans=0.2 2023-11-23 19:05:55,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.96 vs. limit=15.0 2023-11-23 19:06:06,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2502193.3333333335, ans=0.1 2023-11-23 19:06:12,339 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2600, loss[loss=0.07295, simple_loss=0.1007, pruned_loss=0.01598, audio_tagging_loss=0.006615, over 15413.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09091, pruned_loss=0.01353, audio_tagging_loss=0.009058, over 3048298.75 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 8.0 2023-11-23 19:06:14,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.85 vs. limit=15.0 2023-11-23 19:06:15,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2502260.0, ans=0.125 2023-11-23 19:06:23,667 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375350 2023-11-23 19:06:27,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2502326.6666666665, ans=0.07 2023-11-23 19:06:39,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2502393.3333333335, ans=0.125 2023-11-23 19:06:51,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.10 vs. limit=15.0 2023-11-23 19:07:06,771 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.389e+01 8.376e+01 8.898e+01 9.967e+01 2.098e+02, threshold=1.780e+02, percent-clipped=2.0 2023-11-23 19:07:14,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2502593.3333333335, ans=0.07 2023-11-23 19:07:15,696 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2650, loss[loss=0.05413, simple_loss=0.07226, pruned_loss=0.009885, audio_tagging_loss=0.008108, over 15020.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09052, pruned_loss=0.01337, audio_tagging_loss=0.009045, over 3047639.85 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 8.0 2023-11-23 19:07:26,380 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375400 2023-11-23 19:07:42,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2502726.6666666665, ans=0.125 2023-11-23 19:07:46,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2502726.6666666665, ans=0.0 2023-11-23 19:07:53,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.40 vs. limit=22.5 2023-11-23 19:07:55,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2502793.3333333335, ans=0.125 2023-11-23 19:07:56,884 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.77 vs. limit=15.0 2023-11-23 19:08:02,298 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0 2023-11-23 19:08:03,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2502793.3333333335, ans=0.125 2023-11-23 19:08:13,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2502860.0, ans=0.2 2023-11-23 19:08:17,546 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2700, loss[loss=0.05689, simple_loss=0.07177, pruned_loss=0.0112, audio_tagging_loss=0.009811, over 15674.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09081, pruned_loss=0.0135, audio_tagging_loss=0.008986, over 3051366.19 frames. ], batch size: 63, lr: 2.13e-03, grad_scale: 8.0 2023-11-23 19:08:19,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.22 vs. limit=10.0 2023-11-23 19:08:28,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375450 2023-11-23 19:08:28,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2502993.3333333335, ans=0.125 2023-11-23 19:08:37,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2502993.3333333335, ans=0.1 2023-11-23 19:08:41,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.44 vs. limit=15.0 2023-11-23 19:08:47,252 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:08:54,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2503126.6666666665, ans=0.125 2023-11-23 19:08:55,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2503126.6666666665, ans=0.0 2023-11-23 19:09:07,913 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:09:10,591 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.343e+01 9.045e+01 9.938e+01 1.315e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 19:09:14,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2503193.3333333335, ans=0.125 2023-11-23 19:09:18,826 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2750, loss[loss=0.07031, simple_loss=0.09436, pruned_loss=0.01186, audio_tagging_loss=0.01126, over 15122.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09164, pruned_loss=0.0136, audio_tagging_loss=0.008883, over 3056663.81 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 8.0 2023-11-23 19:09:20,556 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.45 vs. limit=15.0 2023-11-23 19:09:30,066 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375500 2023-11-23 19:10:11,506 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:10:16,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2503526.6666666665, ans=15.0 2023-11-23 19:10:20,918 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2800, loss[loss=0.07576, simple_loss=0.1002, pruned_loss=0.01558, audio_tagging_loss=0.01009, over 16770.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09106, pruned_loss=0.01344, audio_tagging_loss=0.008869, over 3058181.59 frames. ], batch size: 63, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:10:22,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2503593.3333333335, ans=0.125 2023-11-23 19:10:31,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2503593.3333333335, ans=0.0 2023-11-23 19:10:32,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375550 2023-11-23 19:10:45,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn1.whiten.whitening_limit, batch_count=2503726.6666666665, ans=22.5 2023-11-23 19:10:52,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2503726.6666666665, ans=0.0 2023-11-23 19:11:04,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2503793.3333333335, ans=0.0 2023-11-23 19:11:04,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2503793.3333333335, ans=0.125 2023-11-23 19:11:04,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0 2023-11-23 19:11:14,365 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.252e+01 8.866e+01 9.555e+01 1.297e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-23 19:11:15,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2503860.0, ans=0.125 2023-11-23 19:11:19,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2503860.0, ans=0.125 2023-11-23 19:11:22,782 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2850, loss[loss=0.05931, simple_loss=0.07996, pruned_loss=0.01107, audio_tagging_loss=0.008263, over 15290.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.0911, pruned_loss=0.01345, audio_tagging_loss=0.008804, over 3056129.09 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:11:24,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2503926.6666666665, ans=0.0 2023-11-23 19:11:24,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2503926.6666666665, ans=0.125 2023-11-23 19:11:26,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2503926.6666666665, ans=0.125 2023-11-23 19:11:29,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2503926.6666666665, ans=0.0 2023-11-23 19:11:34,022 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375600 2023-11-23 19:12:06,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2504126.6666666665, ans=0.125 2023-11-23 19:12:26,392 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2900, loss[loss=0.07366, simple_loss=0.09783, pruned_loss=0.01454, audio_tagging_loss=0.01021, over 16147.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09135, pruned_loss=0.0135, audio_tagging_loss=0.008922, over 3050797.10 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:12:33,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2504260.0, ans=0.125 2023-11-23 19:12:34,604 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.45 vs. limit=15.0 2023-11-23 19:12:37,690 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375650 2023-11-23 19:13:12,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2504460.0, ans=0.125 2023-11-23 19:13:19,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2504526.6666666665, ans=0.125 2023-11-23 19:13:20,157 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.540e+01 9.133e+01 9.789e+01 1.546e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-23 19:13:28,627 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 2950, loss[loss=0.06697, simple_loss=0.09803, pruned_loss=0.009348, audio_tagging_loss=0.0086, over 16007.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09148, pruned_loss=0.01367, audio_tagging_loss=0.009073, over 3050373.45 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:13:34,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2504593.3333333335, ans=0.0 2023-11-23 19:13:40,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375700 2023-11-23 19:13:41,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2504660.0, ans=0.2 2023-11-23 19:13:46,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2504660.0, ans=0.0 2023-11-23 19:14:04,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2504726.6666666665, ans=0.125 2023-11-23 19:14:06,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2504793.3333333335, ans=0.125 2023-11-23 19:14:16,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2504793.3333333335, ans=0.125 2023-11-23 19:14:17,916 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:14:19,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2504860.0, ans=0.125 2023-11-23 19:14:19,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2504860.0, ans=0.1 2023-11-23 19:14:31,241 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3000, loss[loss=0.06615, simple_loss=0.08703, pruned_loss=0.01412, audio_tagging_loss=0.008516, over 15446.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09187, pruned_loss=0.01371, audio_tagging_loss=0.009029, over 3052342.75 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:14:31,242 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 19:14:59,615 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.1683, 3.0724, 3.3182, 2.9099, 3.7776, 3.7507, 3.2498, 3.1284], device='cuda:3') 2023-11-23 19:15:09,799 INFO [train_asr.py:1253] (3/4) Epoch 32, validation: loss=0.05818, simple_loss=0.05104, pruned_loss=0.005158, audio_tagging_loss=0.0275, over 4681554.00 frames. 2023-11-23 19:15:09,799 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 19:15:11,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2504926.6666666665, ans=0.0 2023-11-23 19:15:21,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375750 2023-11-23 19:15:32,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2504993.3333333335, ans=0.125 2023-11-23 19:15:43,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2505060.0, ans=0.125 2023-11-23 19:15:49,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2505126.6666666665, ans=0.125 2023-11-23 19:15:52,367 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.97 vs. limit=12.0 2023-11-23 19:16:01,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2505193.3333333335, ans=0.04949747468305833 2023-11-23 19:16:03,445 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.656e+01 9.141e+01 9.887e+01 1.235e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-23 19:16:07,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2505193.3333333335, ans=0.1 2023-11-23 19:16:11,782 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3050, loss[loss=0.07639, simple_loss=0.1035, pruned_loss=0.01646, audio_tagging_loss=0.008154, over 14458.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09264, pruned_loss=0.01377, audio_tagging_loss=0.009089, over 3052205.05 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:16:23,126 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375800 2023-11-23 19:16:37,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2505393.3333333335, ans=0.2 2023-11-23 19:16:48,178 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:16:50,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2505460.0, ans=0.1 2023-11-23 19:17:13,721 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3100, loss[loss=0.1031, simple_loss=0.1344, pruned_loss=0.02805, audio_tagging_loss=0.007814, over 14582.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09322, pruned_loss=0.01397, audio_tagging_loss=0.009074, over 3041929.95 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:17:15,687 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.86 vs. limit=22.5 2023-11-23 19:17:20,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2505593.3333333335, ans=0.1 2023-11-23 19:17:25,015 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375850 2023-11-23 19:17:26,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2505660.0, ans=15.0 2023-11-23 19:17:42,110 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.23 vs. limit=15.0 2023-11-23 19:18:07,373 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.481e+01 9.058e+01 9.503e+01 1.492e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-23 19:18:08,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2505860.0, ans=0.1 2023-11-23 19:18:15,796 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3150, loss[loss=0.07142, simple_loss=0.09321, pruned_loss=0.01485, audio_tagging_loss=0.009969, over 14383.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09334, pruned_loss=0.01387, audio_tagging_loss=0.009153, over 3042562.51 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:18:17,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2505926.6666666665, ans=0.1 2023-11-23 19:18:23,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2505926.6666666665, ans=0.09899494936611666 2023-11-23 19:18:27,741 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375900 2023-11-23 19:18:37,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2505993.3333333335, ans=0.1 2023-11-23 19:18:45,403 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.53 vs. limit=15.0 2023-11-23 19:19:07,017 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:19:18,397 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3200, loss[loss=0.05176, simple_loss=0.06251, pruned_loss=0.009056, audio_tagging_loss=0.01144, over 14928.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09224, pruned_loss=0.01383, audio_tagging_loss=0.009341, over 3044251.18 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:19:25,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.24 vs. limit=15.0 2023-11-23 19:19:27,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2506260.0, ans=0.0 2023-11-23 19:19:29,719 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 375950 2023-11-23 19:19:33,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2506326.6666666665, ans=0.125 2023-11-23 19:19:53,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2506460.0, ans=0.125 2023-11-23 19:19:57,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2506460.0, ans=0.2 2023-11-23 19:20:06,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2506526.6666666665, ans=0.1 2023-11-23 19:20:13,188 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.152e+01 8.777e+01 9.495e+01 2.540e+02, threshold=1.755e+02, percent-clipped=1.0 2023-11-23 19:20:20,387 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3250, loss[loss=0.06332, simple_loss=0.08192, pruned_loss=0.01021, audio_tagging_loss=0.01215, over 14778.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09217, pruned_loss=0.01386, audio_tagging_loss=0.009347, over 3040847.67 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:20:28,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2506593.3333333335, ans=0.95 2023-11-23 19:20:31,849 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376000 2023-11-23 19:20:37,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2506660.0, ans=0.1 2023-11-23 19:20:50,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2506726.6666666665, ans=0.1 2023-11-23 19:20:57,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2506726.6666666665, ans=0.0 2023-11-23 19:21:01,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2506793.3333333335, ans=0.125 2023-11-23 19:21:20,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2506860.0, ans=0.125 2023-11-23 19:21:24,045 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:21:26,147 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3300, loss[loss=0.08524, simple_loss=0.1099, pruned_loss=0.02356, audio_tagging_loss=0.006721, over 15077.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09264, pruned_loss=0.01399, audio_tagging_loss=0.009394, over 3043029.64 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:21:37,361 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376050 2023-11-23 19:21:37,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2506993.3333333335, ans=0.1 2023-11-23 19:21:48,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2506993.3333333335, ans=0.2 2023-11-23 19:21:53,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.61 vs. limit=6.0 2023-11-23 19:21:54,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.60 vs. limit=15.0 2023-11-23 19:22:12,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2507126.6666666665, ans=0.125 2023-11-23 19:22:20,309 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.632e+01 8.685e+01 9.326e+01 1.032e+02 1.222e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-23 19:22:20,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2507193.3333333335, ans=0.125 2023-11-23 19:22:28,030 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3350, loss[loss=0.06041, simple_loss=0.07942, pruned_loss=0.009798, audio_tagging_loss=0.0109, over 15854.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09266, pruned_loss=0.01396, audio_tagging_loss=0.009276, over 3044474.81 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:22:39,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376100 2023-11-23 19:22:58,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2507393.3333333335, ans=0.125 2023-11-23 19:23:00,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.14 vs. limit=15.0 2023-11-23 19:23:00,501 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.29 vs. limit=8.0 2023-11-23 19:23:13,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2507460.0, ans=0.125 2023-11-23 19:23:19,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2507526.6666666665, ans=0.0 2023-11-23 19:23:30,757 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3400, loss[loss=0.07557, simple_loss=0.09284, pruned_loss=0.01853, audio_tagging_loss=0.01062, over 15404.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09287, pruned_loss=0.01406, audio_tagging_loss=0.009095, over 3045411.84 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:23:41,850 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376150 2023-11-23 19:23:44,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.49 vs. limit=15.0 2023-11-23 19:23:48,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2507660.0, ans=0.125 2023-11-23 19:23:59,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2507726.6666666665, ans=15.0 2023-11-23 19:24:02,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2507726.6666666665, ans=0.07 2023-11-23 19:24:12,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2507793.3333333335, ans=0.0 2023-11-23 19:24:22,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2507860.0, ans=0.0 2023-11-23 19:24:24,965 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.340e+01 8.680e+01 9.645e+01 1.165e+02, threshold=1.736e+02, percent-clipped=0.0 2023-11-23 19:24:25,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2507860.0, ans=0.125 2023-11-23 19:24:32,684 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3450, loss[loss=0.06976, simple_loss=0.08988, pruned_loss=0.0146, audio_tagging_loss=0.01022, over 16208.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09299, pruned_loss=0.01398, audio_tagging_loss=0.009046, over 3047817.40 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:24:32,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2507926.6666666665, ans=0.1 2023-11-23 19:24:44,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376200 2023-11-23 19:24:46,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2507993.3333333335, ans=0.0 2023-11-23 19:24:54,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.77 vs. limit=22.5 2023-11-23 19:24:54,300 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.90 vs. limit=15.0 2023-11-23 19:24:59,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2508060.0, ans=0.1 2023-11-23 19:25:00,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.73 vs. limit=15.0 2023-11-23 19:25:28,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2508193.3333333335, ans=0.125 2023-11-23 19:25:35,036 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3500, loss[loss=0.06886, simple_loss=0.09548, pruned_loss=0.01099, audio_tagging_loss=0.01013, over 14418.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09306, pruned_loss=0.01392, audio_tagging_loss=0.008943, over 3044368.19 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:25:46,586 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376250 2023-11-23 19:25:52,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2508326.6666666665, ans=0.0 2023-11-23 19:25:59,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.87 vs. limit=15.0 2023-11-23 19:26:06,578 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:26:10,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2508393.3333333335, ans=10.0 2023-11-23 19:26:29,755 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.248e+01 9.005e+01 9.777e+01 1.238e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 19:26:38,183 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3550, loss[loss=0.06444, simple_loss=0.08657, pruned_loss=0.01275, audio_tagging_loss=0.008407, over 15967.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.092, pruned_loss=0.01379, audio_tagging_loss=0.008918, over 3043104.20 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:26:40,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2508593.3333333335, ans=0.1 2023-11-23 19:26:45,009 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.63 vs. limit=15.0 2023-11-23 19:26:49,799 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376300 2023-11-23 19:27:15,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2508793.3333333335, ans=0.125 2023-11-23 19:27:23,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2508793.3333333335, ans=0.1 2023-11-23 19:27:31,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2508860.0, ans=0.125 2023-11-23 19:27:32,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2508860.0, ans=0.2 2023-11-23 19:27:38,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2508860.0, ans=0.125 2023-11-23 19:27:40,973 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3600, loss[loss=0.06661, simple_loss=0.09018, pruned_loss=0.01136, audio_tagging_loss=0.01016, over 15482.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09214, pruned_loss=0.01376, audio_tagging_loss=0.008942, over 3045964.15 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:27:43,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2508926.6666666665, ans=0.125 2023-11-23 19:27:48,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2508926.6666666665, ans=0.0 2023-11-23 19:27:51,796 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376350 2023-11-23 19:28:31,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.46 vs. limit=15.0 2023-11-23 19:28:35,123 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.359e+01 9.090e+01 9.890e+01 1.396e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-23 19:28:39,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.01 vs. limit=15.0 2023-11-23 19:28:42,874 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3650, loss[loss=0.05351, simple_loss=0.07215, pruned_loss=0.009866, audio_tagging_loss=0.00757, over 15534.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09228, pruned_loss=0.01385, audio_tagging_loss=0.008935, over 3046439.46 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:28:54,288 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376400 2023-11-23 19:28:57,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2509326.6666666665, ans=0.125 2023-11-23 19:29:03,204 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:29:08,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2509393.3333333335, ans=0.125 2023-11-23 19:29:08,838 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.43 vs. limit=15.0 2023-11-23 19:29:40,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.79 vs. limit=15.0 2023-11-23 19:29:44,986 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3700, loss[loss=0.05894, simple_loss=0.07473, pruned_loss=0.01169, audio_tagging_loss=0.009891, over 14609.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09352, pruned_loss=0.01417, audio_tagging_loss=0.008837, over 3050504.39 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:29:56,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376450 2023-11-23 19:30:07,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2509660.0, ans=0.0 2023-11-23 19:30:19,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2509726.6666666665, ans=0.125 2023-11-23 19:30:40,599 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.564e+01 8.395e+01 9.094e+01 9.680e+01 1.215e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-23 19:30:47,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2509926.6666666665, ans=0.07 2023-11-23 19:30:48,259 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3750, loss[loss=0.08171, simple_loss=0.1049, pruned_loss=0.01579, audio_tagging_loss=0.01349, over 14397.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09412, pruned_loss=0.0143, audio_tagging_loss=0.008848, over 3055502.52 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:30:58,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.13 vs. limit=22.5 2023-11-23 19:30:59,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376500 2023-11-23 19:31:00,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2509993.3333333335, ans=0.0 2023-11-23 19:31:01,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2509993.3333333335, ans=0.125 2023-11-23 19:31:14,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.33 vs. limit=15.0 2023-11-23 19:31:30,740 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:31:45,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.45 vs. limit=12.0 2023-11-23 19:31:50,362 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3800, loss[loss=0.08399, simple_loss=0.1167, pruned_loss=0.01813, audio_tagging_loss=0.007527, over 15583.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09417, pruned_loss=0.01424, audio_tagging_loss=0.00889, over 3059801.14 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:31:59,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2510260.0, ans=0.0 2023-11-23 19:32:01,470 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376550 2023-11-23 19:32:15,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2510393.3333333335, ans=0.2 2023-11-23 19:32:19,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2510393.3333333335, ans=0.0 2023-11-23 19:32:41,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2510526.6666666665, ans=15.0 2023-11-23 19:32:45,285 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.658e+01 9.226e+01 1.003e+02 1.868e+02, threshold=1.845e+02, percent-clipped=1.0 2023-11-23 19:32:47,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.05 vs. limit=22.5 2023-11-23 19:32:52,908 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3850, loss[loss=0.06824, simple_loss=0.09053, pruned_loss=0.01356, audio_tagging_loss=0.009415, over 15555.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09368, pruned_loss=0.01418, audio_tagging_loss=0.009008, over 3051257.49 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:32:55,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2510593.3333333335, ans=0.0 2023-11-23 19:33:02,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2510593.3333333335, ans=0.05 2023-11-23 19:33:03,574 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376600 2023-11-23 19:33:06,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2510660.0, ans=0.1 2023-11-23 19:33:17,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.74 vs. limit=15.0 2023-11-23 19:33:46,070 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.08 vs. limit=15.0 2023-11-23 19:33:54,698 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3900, loss[loss=0.04851, simple_loss=0.06298, pruned_loss=0.008115, audio_tagging_loss=0.008903, over 14561.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09249, pruned_loss=0.01382, audio_tagging_loss=0.009087, over 3048896.97 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:34:05,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376650 2023-11-23 19:34:06,554 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.16 vs. limit=15.0 2023-11-23 19:34:21,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2511060.0, ans=0.2 2023-11-23 19:34:46,500 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=12.0 2023-11-23 19:34:49,479 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.099e+01 8.305e+01 8.896e+01 9.680e+01 1.276e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 19:34:55,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2511260.0, ans=0.125 2023-11-23 19:34:56,883 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 3950, loss[loss=0.07828, simple_loss=0.1043, pruned_loss=0.01679, audio_tagging_loss=0.009324, over 16852.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09141, pruned_loss=0.01366, audio_tagging_loss=0.009202, over 3050401.51 frames. ], batch size: 63, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:35:08,287 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376700 2023-11-23 19:35:26,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2511393.3333333335, ans=0.125 2023-11-23 19:35:37,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2511460.0, ans=0.0 2023-11-23 19:35:56,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2511526.6666666665, ans=0.125 2023-11-23 19:35:59,035 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4000, loss[loss=0.07332, simple_loss=0.1057, pruned_loss=0.01316, audio_tagging_loss=0.007294, over 16267.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09185, pruned_loss=0.01372, audio_tagging_loss=0.009254, over 3052035.31 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:36:10,335 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376750 2023-11-23 19:36:26,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2511726.6666666665, ans=0.125 2023-11-23 19:36:29,497 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.75 vs. limit=15.0 2023-11-23 19:36:46,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2511793.3333333335, ans=0.125 2023-11-23 19:36:53,799 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.509e+01 8.978e+01 9.607e+01 1.233e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-23 19:36:56,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2511860.0, ans=0.0 2023-11-23 19:37:00,848 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4050, loss[loss=0.07981, simple_loss=0.1163, pruned_loss=0.01401, audio_tagging_loss=0.007659, over 15654.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09278, pruned_loss=0.01392, audio_tagging_loss=0.009284, over 3060860.28 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:37:03,265 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:37:03,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.13 vs. limit=15.0 2023-11-23 19:37:11,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2511926.6666666665, ans=0.0 2023-11-23 19:37:12,618 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376800 2023-11-23 19:37:44,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2512126.6666666665, ans=0.09899494936611666 2023-11-23 19:37:47,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2512126.6666666665, ans=10.0 2023-11-23 19:38:02,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2512260.0, ans=0.125 2023-11-23 19:38:03,258 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4100, loss[loss=0.07389, simple_loss=0.09674, pruned_loss=0.01357, audio_tagging_loss=0.01195, over 15533.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09317, pruned_loss=0.01396, audio_tagging_loss=0.009213, over 3059896.88 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:38:07,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2512260.0, ans=0.125 2023-11-23 19:38:11,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2512260.0, ans=0.125 2023-11-23 19:38:14,702 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376850 2023-11-23 19:38:36,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2512393.3333333335, ans=0.1 2023-11-23 19:38:44,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2512460.0, ans=0.0 2023-11-23 19:38:59,476 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.223e+01 8.590e+01 9.258e+01 1.002e+02 1.554e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-23 19:39:05,551 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4150, loss[loss=0.06715, simple_loss=0.09136, pruned_loss=0.01152, audio_tagging_loss=0.009949, over 16186.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09241, pruned_loss=0.01386, audio_tagging_loss=0.009209, over 3055468.34 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:39:12,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2512593.3333333335, ans=0.125 2023-11-23 19:39:16,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376900 2023-11-23 19:39:23,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2512660.0, ans=0.125 2023-11-23 19:39:34,702 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.50 vs. limit=12.0 2023-11-23 19:39:49,391 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:39:49,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2512793.3333333335, ans=0.125 2023-11-23 19:39:56,609 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:40:07,872 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4200, loss[loss=0.07132, simple_loss=0.09277, pruned_loss=0.01716, audio_tagging_loss=0.007783, over 15147.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09264, pruned_loss=0.01371, audio_tagging_loss=0.009081, over 3054942.40 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:40:14,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2512926.6666666665, ans=0.1 2023-11-23 19:40:19,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 376950 2023-11-23 19:40:23,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2512993.3333333335, ans=0.0 2023-11-23 19:40:56,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2513193.3333333335, ans=0.125 2023-11-23 19:41:04,172 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.429e+01 8.916e+01 9.542e+01 1.145e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 19:41:05,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2513193.3333333335, ans=0.1 2023-11-23 19:41:06,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2513193.3333333335, ans=0.0 2023-11-23 19:41:10,208 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4250, loss[loss=0.08723, simple_loss=0.1285, pruned_loss=0.01848, audio_tagging_loss=0.00451, over 15655.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09272, pruned_loss=0.01371, audio_tagging_loss=0.009034, over 3052577.52 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:41:22,246 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377000 2023-11-23 19:41:32,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2513326.6666666665, ans=0.125 2023-11-23 19:41:37,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2513393.3333333335, ans=0.125 2023-11-23 19:41:52,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2513460.0, ans=0.1 2023-11-23 19:41:53,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2513460.0, ans=0.1 2023-11-23 19:42:13,232 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4300, loss[loss=0.04912, simple_loss=0.05426, pruned_loss=0.009748, audio_tagging_loss=0.01224, over 15296.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09126, pruned_loss=0.01351, audio_tagging_loss=0.009068, over 3046511.56 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:42:24,754 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377050 2023-11-23 19:42:41,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2513726.6666666665, ans=0.09899494936611666 2023-11-23 19:42:46,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2513726.6666666665, ans=0.125 2023-11-23 19:43:01,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2513860.0, ans=0.125 2023-11-23 19:43:09,396 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.731e+01 8.254e+01 8.867e+01 9.585e+01 1.161e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-23 19:43:15,474 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4350, loss[loss=0.05169, simple_loss=0.06518, pruned_loss=0.007325, audio_tagging_loss=0.01178, over 16567.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09173, pruned_loss=0.01355, audio_tagging_loss=0.009009, over 3048002.18 frames. ], batch size: 63, lr: 2.13e-03, grad_scale: 16.0 2023-11-23 19:43:19,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2513926.6666666665, ans=0.125 2023-11-23 19:43:26,691 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377100 2023-11-23 19:43:45,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2514060.0, ans=15.0 2023-11-23 19:43:56,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2514126.6666666665, ans=0.2 2023-11-23 19:43:56,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2514126.6666666665, ans=0.2 2023-11-23 19:44:01,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2514126.6666666665, ans=0.125 2023-11-23 19:44:02,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=2514126.6666666665, ans=0.1 2023-11-23 19:44:02,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2514126.6666666665, ans=0.125 2023-11-23 19:44:16,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2514260.0, ans=0.125 2023-11-23 19:44:16,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2514260.0, ans=0.125 2023-11-23 19:44:17,163 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4400, loss[loss=0.07269, simple_loss=0.091, pruned_loss=0.01544, audio_tagging_loss=0.01174, over 14088.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.0923, pruned_loss=0.0138, audio_tagging_loss=0.008933, over 3049229.91 frames. ], batch size: 52, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:44:19,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2514260.0, ans=0.125 2023-11-23 19:44:25,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.26 vs. limit=15.0 2023-11-23 19:44:28,440 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377150 2023-11-23 19:44:32,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2514326.6666666665, ans=0.0 2023-11-23 19:44:38,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2514326.6666666665, ans=0.0 2023-11-23 19:44:45,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2514393.3333333335, ans=0.09899494936611666 2023-11-23 19:45:13,009 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.470e+01 9.052e+01 9.834e+01 1.276e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 19:45:13,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2514526.6666666665, ans=0.125 2023-11-23 19:45:20,345 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4450, loss[loss=0.06025, simple_loss=0.08089, pruned_loss=0.0113, audio_tagging_loss=0.008505, over 15221.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09277, pruned_loss=0.01395, audio_tagging_loss=0.008889, over 3040520.42 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:45:21,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2514593.3333333335, ans=0.125 2023-11-23 19:45:31,745 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377200 2023-11-23 19:45:45,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2514726.6666666665, ans=0.125 2023-11-23 19:46:23,441 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4500, loss[loss=0.06585, simple_loss=0.08585, pruned_loss=0.01325, audio_tagging_loss=0.009676, over 15085.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09165, pruned_loss=0.01358, audio_tagging_loss=0.008951, over 3041420.40 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0 2023-11-23 19:46:24,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2514926.6666666665, ans=0.0 2023-11-23 19:46:34,098 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377250 2023-11-23 19:46:47,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2515060.0, ans=0.125 2023-11-23 19:47:04,606 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.80 vs. limit=6.0 2023-11-23 19:47:10,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2515126.6666666665, ans=0.125 2023-11-23 19:47:18,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2515193.3333333335, ans=0.05 2023-11-23 19:47:18,871 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.414e+01 9.190e+01 9.821e+01 1.226e+02, threshold=1.838e+02, percent-clipped=0.0 2023-11-23 19:47:25,478 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4550, loss[loss=0.05863, simple_loss=0.08619, pruned_loss=0.009446, audio_tagging_loss=0.006086, over 16077.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09174, pruned_loss=0.01365, audio_tagging_loss=0.008976, over 3045521.91 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 19:47:36,321 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377300 2023-11-23 19:47:36,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2515326.6666666665, ans=0.2 2023-11-23 19:47:41,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2515326.6666666665, ans=0.125 2023-11-23 19:47:44,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=2515326.6666666665, ans=0.02 2023-11-23 19:47:46,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2515326.6666666665, ans=0.2 2023-11-23 19:47:49,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2515393.3333333335, ans=0.0 2023-11-23 19:47:51,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2515393.3333333335, ans=0.125 2023-11-23 19:47:58,967 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.32 vs. limit=15.0 2023-11-23 19:48:11,894 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 19:48:28,003 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4600, loss[loss=0.08541, simple_loss=0.112, pruned_loss=0.01885, audio_tagging_loss=0.01055, over 14637.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09154, pruned_loss=0.01368, audio_tagging_loss=0.009146, over 3040882.76 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 19:48:30,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2515593.3333333335, ans=0.125 2023-11-23 19:48:30,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2515593.3333333335, ans=0.07 2023-11-23 19:48:35,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2515593.3333333335, ans=0.125 2023-11-23 19:48:39,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377350 2023-11-23 19:48:46,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2515660.0, ans=0.2 2023-11-23 19:48:55,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.85 vs. limit=12.0 2023-11-23 19:49:10,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2515793.3333333335, ans=0.1 2023-11-23 19:49:16,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2515860.0, ans=0.0 2023-11-23 19:49:23,551 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.494e+01 9.080e+01 9.829e+01 1.199e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-23 19:49:26,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2515860.0, ans=0.2 2023-11-23 19:49:30,639 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4650, loss[loss=0.06367, simple_loss=0.08833, pruned_loss=0.01057, audio_tagging_loss=0.00894, over 15465.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09119, pruned_loss=0.0135, audio_tagging_loss=0.009134, over 3048158.20 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 19:49:34,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2515926.6666666665, ans=0.125 2023-11-23 19:49:41,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377400 2023-11-23 19:49:54,238 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:50:00,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2516060.0, ans=0.0 2023-11-23 19:50:11,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2516126.6666666665, ans=0.2 2023-11-23 19:50:30,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2516193.3333333335, ans=0.0 2023-11-23 19:50:33,075 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4700, loss[loss=0.07971, simple_loss=0.09505, pruned_loss=0.02131, audio_tagging_loss=0.01087, over 14471.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09072, pruned_loss=0.0134, audio_tagging_loss=0.009319, over 3045952.13 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 19:50:43,764 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377450 2023-11-23 19:50:48,618 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=15.24 vs. limit=15.0 2023-11-23 19:50:58,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.06 vs. limit=15.0 2023-11-23 19:51:00,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2516393.3333333335, ans=0.0 2023-11-23 19:51:12,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.20 vs. limit=15.0 2023-11-23 19:51:20,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2516460.0, ans=0.125 2023-11-23 19:51:23,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2516526.6666666665, ans=0.0 2023-11-23 19:51:28,162 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.521e+01 9.088e+01 9.578e+01 1.245e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-23 19:51:34,066 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4750, loss[loss=0.07736, simple_loss=0.1094, pruned_loss=0.01513, audio_tagging_loss=0.00755, over 15401.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.0907, pruned_loss=0.0135, audio_tagging_loss=0.009295, over 3041899.91 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 19:51:45,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377500 2023-11-23 19:51:48,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2516660.0, ans=0.1 2023-11-23 19:51:49,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2516660.0, ans=0.0 2023-11-23 19:51:56,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2516660.0, ans=0.2 2023-11-23 19:52:08,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2516726.6666666665, ans=0.0 2023-11-23 19:52:09,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2516726.6666666665, ans=0.2 2023-11-23 19:52:20,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2516793.3333333335, ans=0.0 2023-11-23 19:52:28,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2516860.0, ans=0.125 2023-11-23 19:52:36,090 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4800, loss[loss=0.05417, simple_loss=0.06744, pruned_loss=0.008696, audio_tagging_loss=0.01175, over 16342.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09147, pruned_loss=0.01359, audio_tagging_loss=0.009363, over 3041550.23 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 19:52:47,940 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377550 2023-11-23 19:52:56,802 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:52:59,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2516993.3333333335, ans=0.125 2023-11-23 19:53:21,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2517126.6666666665, ans=0.125 2023-11-23 19:53:26,911 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.36 vs. limit=10.0 2023-11-23 19:53:27,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2517193.3333333335, ans=0.125 2023-11-23 19:53:33,630 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.424e+01 8.895e+01 9.657e+01 1.306e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 19:53:36,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2517193.3333333335, ans=0.0 2023-11-23 19:53:38,429 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4850, loss[loss=0.06846, simple_loss=0.08542, pruned_loss=0.0166, audio_tagging_loss=0.009155, over 15142.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.0922, pruned_loss=0.01365, audio_tagging_loss=0.009387, over 3046014.90 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 19:53:40,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.94 vs. limit=10.0 2023-11-23 19:53:45,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2517260.0, ans=0.125 2023-11-23 19:53:49,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377600 2023-11-23 19:53:51,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2517326.6666666665, ans=0.0 2023-11-23 19:53:53,106 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.13 vs. limit=15.0 2023-11-23 19:54:40,503 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4900, loss[loss=0.08142, simple_loss=0.1127, pruned_loss=0.01806, audio_tagging_loss=0.006996, over 15253.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09284, pruned_loss=0.01373, audio_tagging_loss=0.009287, over 3050296.95 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 19:54:51,788 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377650 2023-11-23 19:54:55,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2517660.0, ans=0.125 2023-11-23 19:55:07,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2517726.6666666665, ans=0.0 2023-11-23 19:55:13,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2517726.6666666665, ans=0.0 2023-11-23 19:55:17,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2517726.6666666665, ans=0.125 2023-11-23 19:55:29,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2517860.0, ans=0.1 2023-11-23 19:55:33,001 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:55:35,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.03 vs. limit=15.0 2023-11-23 19:55:38,619 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.557e+01 8.293e+01 8.753e+01 9.560e+01 1.963e+02, threshold=1.751e+02, percent-clipped=1.0 2023-11-23 19:55:43,402 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 4950, loss[loss=0.07757, simple_loss=0.1046, pruned_loss=0.01646, audio_tagging_loss=0.008828, over 16757.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09363, pruned_loss=0.01393, audio_tagging_loss=0.009078, over 3051825.56 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 19:55:54,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377700 2023-11-23 19:55:57,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2517993.3333333335, ans=0.1 2023-11-23 19:56:01,953 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.84 vs. limit=15.0 2023-11-23 19:56:04,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2517993.3333333335, ans=0.125 2023-11-23 19:56:09,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2518060.0, ans=0.1 2023-11-23 19:56:21,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2518126.6666666665, ans=0.125 2023-11-23 19:56:40,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2518193.3333333335, ans=0.0 2023-11-23 19:56:45,965 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5000, loss[loss=0.0697, simple_loss=0.09854, pruned_loss=0.0109, audio_tagging_loss=0.009528, over 14978.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09293, pruned_loss=0.01386, audio_tagging_loss=0.008969, over 3050844.35 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 19:56:58,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377750 2023-11-23 19:56:59,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2518326.6666666665, ans=0.2 2023-11-23 19:56:59,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2518326.6666666665, ans=0.2 2023-11-23 19:57:17,816 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:57:17,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2518393.3333333335, ans=0.0 2023-11-23 19:57:18,942 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:57:25,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2518460.0, ans=0.125 2023-11-23 19:57:32,839 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.37 vs. limit=15.0 2023-11-23 19:57:34,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2.whitening_limit, batch_count=2518460.0, ans=15.0 2023-11-23 19:57:40,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2518526.6666666665, ans=0.2 2023-11-23 19:57:43,976 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.445e+01 8.514e+01 9.045e+01 9.813e+01 1.165e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 19:57:48,794 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5050, loss[loss=0.06008, simple_loss=0.07604, pruned_loss=0.0143, audio_tagging_loss=0.007762, over 14562.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09197, pruned_loss=0.01374, audio_tagging_loss=0.008886, over 3046486.92 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 19:57:51,920 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.49 vs. limit=12.0 2023-11-23 19:58:00,106 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377800 2023-11-23 19:58:24,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2518793.3333333335, ans=0.0 2023-11-23 19:58:37,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2518860.0, ans=0.125 2023-11-23 19:58:50,380 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5100, loss[loss=0.09031, simple_loss=0.1303, pruned_loss=0.01764, audio_tagging_loss=0.007534, over 14688.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09213, pruned_loss=0.01372, audio_tagging_loss=0.008832, over 3048461.64 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 19:58:53,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.57 vs. limit=15.0 2023-11-23 19:59:00,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2518926.6666666665, ans=0.0 2023-11-23 19:59:01,759 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377850 2023-11-23 19:59:16,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2519060.0, ans=0.125 2023-11-23 19:59:24,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2519060.0, ans=0.0 2023-11-23 19:59:25,456 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 19:59:38,440 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0 2023-11-23 19:59:47,574 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.550e+01 9.235e+01 1.014e+02 1.258e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-23 19:59:52,265 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5150, loss[loss=0.05317, simple_loss=0.06788, pruned_loss=0.008318, audio_tagging_loss=0.01091, over 13410.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09198, pruned_loss=0.01363, audio_tagging_loss=0.008776, over 3049078.14 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 19:59:53,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2519260.0, ans=0.1 2023-11-23 20:00:02,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2519260.0, ans=0.125 2023-11-23 20:00:03,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377900 2023-11-23 20:00:09,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.73 vs. limit=15.0 2023-11-23 20:00:28,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2519460.0, ans=0.125 2023-11-23 20:00:54,664 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5200, loss[loss=0.05532, simple_loss=0.07397, pruned_loss=0.00737, audio_tagging_loss=0.01096, over 16200.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09218, pruned_loss=0.01364, audio_tagging_loss=0.008807, over 3042299.10 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:01:05,943 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 377950 2023-11-23 20:01:15,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2519660.0, ans=0.125 2023-11-23 20:01:21,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2519726.6666666665, ans=0.1 2023-11-23 20:01:23,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0 2023-11-23 20:01:34,631 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.56 vs. limit=12.0 2023-11-23 20:01:51,891 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.115e+01 8.438e+01 9.195e+01 9.999e+01 1.195e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-23 20:01:53,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2519860.0, ans=0.125 2023-11-23 20:01:56,764 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5250, loss[loss=0.07822, simple_loss=0.112, pruned_loss=0.01536, audio_tagging_loss=0.006853, over 15638.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09254, pruned_loss=0.01365, audio_tagging_loss=0.008754, over 3045760.37 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:01:57,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2519926.6666666665, ans=0.125 2023-11-23 20:02:07,995 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378000 2023-11-23 20:02:12,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2519993.3333333335, ans=0.0 2023-11-23 20:02:19,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2519993.3333333335, ans=0.125 2023-11-23 20:02:54,531 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.38 vs. limit=22.5 2023-11-23 20:02:59,100 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5300, loss[loss=0.06488, simple_loss=0.07441, pruned_loss=0.01661, audio_tagging_loss=0.01107, over 15126.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09234, pruned_loss=0.01359, audio_tagging_loss=0.00879, over 3046870.23 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:02:59,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2520260.0, ans=0.125 2023-11-23 20:03:10,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378050 2023-11-23 20:03:18,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2520326.6666666665, ans=0.07 2023-11-23 20:03:32,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2520393.3333333335, ans=0.125 2023-11-23 20:03:35,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.69 vs. limit=22.5 2023-11-23 20:03:37,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2520460.0, ans=0.015 2023-11-23 20:03:56,742 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.515e+01 8.458e+01 9.027e+01 9.985e+01 1.662e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 20:04:01,556 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5350, loss[loss=0.06244, simple_loss=0.08278, pruned_loss=0.01137, audio_tagging_loss=0.009686, over 14924.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09121, pruned_loss=0.01342, audio_tagging_loss=0.008813, over 3047271.94 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:04:13,478 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378100 2023-11-23 20:04:47,425 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.12 vs. limit=15.0 2023-11-23 20:05:04,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2520926.6666666665, ans=0.0 2023-11-23 20:05:04,835 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5400, loss[loss=0.05919, simple_loss=0.0786, pruned_loss=0.01035, audio_tagging_loss=0.009545, over 15274.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09154, pruned_loss=0.01342, audio_tagging_loss=0.008852, over 3053419.70 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:05:12,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2520926.6666666665, ans=0.0 2023-11-23 20:05:13,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.67 vs. limit=10.0 2023-11-23 20:05:15,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.50 vs. limit=12.0 2023-11-23 20:05:15,559 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378150 2023-11-23 20:05:53,639 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.60 vs. limit=15.0 2023-11-23 20:06:01,387 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.400e+01 9.032e+01 9.850e+01 1.216e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-23 20:06:06,754 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5450, loss[loss=0.06082, simple_loss=0.07757, pruned_loss=0.01316, audio_tagging_loss=0.008878, over 16147.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09195, pruned_loss=0.01352, audio_tagging_loss=0.008936, over 3044816.04 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:06:17,531 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378200 2023-11-23 20:06:17,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2521326.6666666665, ans=0.125 2023-11-23 20:06:17,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2521326.6666666665, ans=0.2 2023-11-23 20:06:30,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2521393.3333333335, ans=0.0 2023-11-23 20:06:47,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2521460.0, ans=0.0 2023-11-23 20:06:52,175 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.20 vs. limit=15.0 2023-11-23 20:07:04,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2521526.6666666665, ans=0.125 2023-11-23 20:07:09,667 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5500, loss[loss=0.08652, simple_loss=0.1131, pruned_loss=0.02027, audio_tagging_loss=0.009676, over 14578.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09183, pruned_loss=0.01352, audio_tagging_loss=0.00904, over 3045740.32 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:07:14,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2521593.3333333335, ans=0.125 2023-11-23 20:07:20,543 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378250 2023-11-23 20:07:26,389 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.14 vs. limit=10.0 2023-11-23 20:07:30,356 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.04 vs. limit=15.0 2023-11-23 20:07:35,392 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.43 vs. limit=22.5 2023-11-23 20:07:37,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2521726.6666666665, ans=0.125 2023-11-23 20:07:43,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2521726.6666666665, ans=0.125 2023-11-23 20:08:06,546 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.320e+01 8.896e+01 9.871e+01 1.353e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 20:08:08,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.35 vs. limit=8.0 2023-11-23 20:08:12,059 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5550, loss[loss=0.07301, simple_loss=0.09896, pruned_loss=0.01552, audio_tagging_loss=0.008017, over 15729.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09165, pruned_loss=0.01345, audio_tagging_loss=0.009117, over 3053928.14 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:08:17,734 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.47 vs. limit=15.0 2023-11-23 20:08:23,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378300 2023-11-23 20:09:01,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2522193.3333333335, ans=0.125 2023-11-23 20:09:12,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2522260.0, ans=0.125 2023-11-23 20:09:13,831 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5600, loss[loss=0.07244, simple_loss=0.09353, pruned_loss=0.016, audio_tagging_loss=0.009672, over 15015.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09212, pruned_loss=0.01357, audio_tagging_loss=0.009189, over 3059181.40 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:09:25,223 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378350 2023-11-23 20:09:37,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2522393.3333333335, ans=0.0 2023-11-23 20:09:45,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.33 vs. limit=15.0 2023-11-23 20:09:55,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2522460.0, ans=0.125 2023-11-23 20:09:57,318 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 20:10:11,709 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.330e+01 9.147e+01 9.848e+01 1.519e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-23 20:10:15,359 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5650, loss[loss=0.09479, simple_loss=0.1325, pruned_loss=0.01997, audio_tagging_loss=0.008554, over 15649.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09182, pruned_loss=0.0137, audio_tagging_loss=0.009239, over 3062517.13 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:10:22,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2522593.3333333335, ans=0.125 2023-11-23 20:10:26,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378400 2023-11-23 20:10:54,899 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.16 vs. limit=15.0 2023-11-23 20:10:59,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.56 vs. limit=15.0 2023-11-23 20:11:16,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2522926.6666666665, ans=0.125 2023-11-23 20:11:17,842 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5700, loss[loss=0.06174, simple_loss=0.07827, pruned_loss=0.01303, audio_tagging_loss=0.009575, over 15205.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09132, pruned_loss=0.01344, audio_tagging_loss=0.009243, over 3057122.38 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:11:18,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.98 vs. limit=6.0 2023-11-23 20:11:21,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2522926.6666666665, ans=0.2 2023-11-23 20:11:22,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2522926.6666666665, ans=0.125 2023-11-23 20:11:27,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2522926.6666666665, ans=0.1 2023-11-23 20:11:28,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378450 2023-11-23 20:12:14,662 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.181e+01 8.343e+01 8.908e+01 9.732e+01 1.345e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 20:12:16,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2523193.3333333335, ans=0.0 2023-11-23 20:12:18,289 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5750, loss[loss=0.06983, simple_loss=0.09256, pruned_loss=0.01468, audio_tagging_loss=0.008866, over 14438.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09098, pruned_loss=0.01338, audio_tagging_loss=0.009222, over 3054009.43 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:12:29,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378500 2023-11-23 20:13:00,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2523460.0, ans=0.125 2023-11-23 20:13:02,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2523460.0, ans=0.0 2023-11-23 20:13:13,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2523526.6666666665, ans=0.125 2023-11-23 20:13:20,190 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5800, loss[loss=0.05304, simple_loss=0.06162, pruned_loss=0.01002, audio_tagging_loss=0.01221, over 14352.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09073, pruned_loss=0.01354, audio_tagging_loss=0.009136, over 3048184.17 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:13:21,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2523593.3333333335, ans=0.2 2023-11-23 20:13:24,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2523593.3333333335, ans=0.125 2023-11-23 20:13:24,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2523593.3333333335, ans=0.5 2023-11-23 20:13:31,219 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378550 2023-11-23 20:13:31,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2523660.0, ans=0.125 2023-11-23 20:13:35,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2523660.0, ans=15.0 2023-11-23 20:13:40,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2523660.0, ans=0.2 2023-11-23 20:13:40,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.62 vs. limit=15.0 2023-11-23 20:13:49,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2523726.6666666665, ans=0.1 2023-11-23 20:14:17,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2523860.0, ans=0.0 2023-11-23 20:14:19,416 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.431e+01 8.343e+01 9.004e+01 9.818e+01 1.259e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 20:14:21,786 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5850, loss[loss=0.06289, simple_loss=0.09033, pruned_loss=0.007253, audio_tagging_loss=0.01047, over 14534.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09049, pruned_loss=0.01343, audio_tagging_loss=0.009089, over 3046638.19 frames. ], batch size: 52, lr: 2.12e-03, grad_scale: 8.0 2023-11-23 20:14:28,916 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.33 vs. limit=12.0 2023-11-23 20:14:33,109 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378600 2023-11-23 20:15:05,693 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.46 vs. limit=6.0 2023-11-23 20:15:20,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2524193.3333333335, ans=0.125 2023-11-23 20:15:24,463 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5900, loss[loss=0.05909, simple_loss=0.07887, pruned_loss=0.008458, audio_tagging_loss=0.0112, over 13910.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09134, pruned_loss=0.01349, audio_tagging_loss=0.009102, over 3043027.80 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 8.0 2023-11-23 20:15:25,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2524260.0, ans=0.05 2023-11-23 20:15:35,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378650 2023-11-23 20:16:05,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2524460.0, ans=0.125 2023-11-23 20:16:08,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2524460.0, ans=6.0 2023-11-23 20:16:20,909 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.71 vs. limit=10.0 2023-11-23 20:16:23,875 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.483e+01 8.986e+01 9.661e+01 2.610e+02, threshold=1.797e+02, percent-clipped=1.0 2023-11-23 20:16:26,891 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 5950, loss[loss=0.06158, simple_loss=0.08738, pruned_loss=0.008717, audio_tagging_loss=0.009168, over 14871.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09121, pruned_loss=0.01333, audio_tagging_loss=0.009089, over 3045696.92 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 8.0 2023-11-23 20:16:34,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2524593.3333333335, ans=0.2 2023-11-23 20:16:34,999 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.84 vs. limit=15.0 2023-11-23 20:16:37,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2524593.3333333335, ans=0.1 2023-11-23 20:16:38,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378700 2023-11-23 20:17:15,741 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0 2023-11-23 20:17:28,288 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6000, loss[loss=0.08674, simple_loss=0.1179, pruned_loss=0.01775, audio_tagging_loss=0.01002, over 17005.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09125, pruned_loss=0.0134, audio_tagging_loss=0.00905, over 3041971.39 frames. ], batch size: 63, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:17:28,288 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 20:18:07,444 INFO [train_asr.py:1253] (3/4) Epoch 32, validation: loss=0.05807, simple_loss=0.05104, pruned_loss=0.005144, audio_tagging_loss=0.02741, over 4681554.00 frames. 2023-11-23 20:18:07,445 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 20:18:07,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2524926.6666666665, ans=0.125 2023-11-23 20:18:18,757 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378750 2023-11-23 20:18:25,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2524993.3333333335, ans=0.1 2023-11-23 20:18:50,432 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.64 vs. limit=15.0 2023-11-23 20:18:52,191 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 20:19:06,948 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.325e+01 9.027e+01 9.614e+01 1.325e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-23 20:19:10,004 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6050, loss[loss=0.09589, simple_loss=0.1315, pruned_loss=0.02122, audio_tagging_loss=0.008925, over 15007.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.0911, pruned_loss=0.01347, audio_tagging_loss=0.008971, over 3040199.20 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:19:19,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2525260.0, ans=0.0 2023-11-23 20:19:21,281 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378800 2023-11-23 20:19:31,066 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=12.0 2023-11-23 20:20:10,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.68 vs. limit=15.0 2023-11-23 20:20:12,599 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6100, loss[loss=0.07236, simple_loss=0.1051, pruned_loss=0.01277, audio_tagging_loss=0.007053, over 16249.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09112, pruned_loss=0.01335, audio_tagging_loss=0.008957, over 3050292.88 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:20:24,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378850 2023-11-23 20:20:25,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2525660.0, ans=0.2 2023-11-23 20:20:34,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2525660.0, ans=0.1 2023-11-23 20:20:43,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2525726.6666666665, ans=0.2 2023-11-23 20:20:43,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2525726.6666666665, ans=0.2 2023-11-23 20:20:45,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2525726.6666666665, ans=15.0 2023-11-23 20:20:49,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2525793.3333333335, ans=0.1 2023-11-23 20:21:04,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.00 vs. limit=6.0 2023-11-23 20:21:12,675 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.468e+01 9.222e+01 1.005e+02 1.157e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-23 20:21:15,196 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6150, loss[loss=0.08497, simple_loss=0.1098, pruned_loss=0.01973, audio_tagging_loss=0.01031, over 14564.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09128, pruned_loss=0.01344, audio_tagging_loss=0.008967, over 3051407.34 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:21:20,004 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.65 vs. limit=22.5 2023-11-23 20:21:26,630 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378900 2023-11-23 20:21:30,846 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.61 vs. limit=15.0 2023-11-23 20:21:48,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.38 vs. limit=12.0 2023-11-23 20:21:57,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.47 vs. limit=15.0 2023-11-23 20:22:07,888 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.11 vs. limit=15.0 2023-11-23 20:22:10,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2526193.3333333335, ans=0.125 2023-11-23 20:22:17,100 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6200, loss[loss=0.07775, simple_loss=0.1045, pruned_loss=0.01654, audio_tagging_loss=0.00894, over 15517.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09062, pruned_loss=0.01327, audio_tagging_loss=0.009117, over 3048846.80 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:22:20,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=15.0 2023-11-23 20:22:29,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 378950 2023-11-23 20:22:34,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2526326.6666666665, ans=0.125 2023-11-23 20:22:36,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2526326.6666666665, ans=0.1 2023-11-23 20:22:40,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2526326.6666666665, ans=0.2 2023-11-23 20:22:59,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2526460.0, ans=0.125 2023-11-23 20:23:08,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2526526.6666666665, ans=0.125 2023-11-23 20:23:13,301 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0 2023-11-23 20:23:14,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2526526.6666666665, ans=0.125 2023-11-23 20:23:17,960 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.457e+01 9.040e+01 9.939e+01 1.355e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-23 20:23:18,824 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.52 vs. limit=15.0 2023-11-23 20:23:20,357 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6250, loss[loss=0.07254, simple_loss=0.09705, pruned_loss=0.01503, audio_tagging_loss=0.008985, over 16245.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09103, pruned_loss=0.01334, audio_tagging_loss=0.009096, over 3054341.32 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:23:31,677 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379000 2023-11-23 20:23:40,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2526660.0, ans=0.125 2023-11-23 20:24:22,790 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6300, loss[loss=0.08012, simple_loss=0.1041, pruned_loss=0.01868, audio_tagging_loss=0.009394, over 13657.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09209, pruned_loss=0.01375, audio_tagging_loss=0.009148, over 3052090.21 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:24:32,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.82 vs. limit=15.0 2023-11-23 20:24:34,267 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379050 2023-11-23 20:25:12,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2527193.3333333335, ans=0.05 2023-11-23 20:25:14,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.00 vs. limit=15.0 2023-11-23 20:25:17,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2527193.3333333335, ans=0.125 2023-11-23 20:25:18,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2527193.3333333335, ans=0.2 2023-11-23 20:25:21,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.520e+01 9.067e+01 9.897e+01 1.384e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-23 20:25:24,465 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6350, loss[loss=0.0475, simple_loss=0.05049, pruned_loss=0.00794, audio_tagging_loss=0.01432, over 15451.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09224, pruned_loss=0.01356, audio_tagging_loss=0.00912, over 3056941.36 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:25:24,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2527260.0, ans=0.2 2023-11-23 20:25:29,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2527260.0, ans=0.125 2023-11-23 20:25:35,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379100 2023-11-23 20:25:50,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2527393.3333333335, ans=0.125 2023-11-23 20:26:02,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2527460.0, ans=0.0 2023-11-23 20:26:14,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2527526.6666666665, ans=0.125 2023-11-23 20:26:23,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2527526.6666666665, ans=0.0 2023-11-23 20:26:27,038 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6400, loss[loss=0.06366, simple_loss=0.08175, pruned_loss=0.0148, audio_tagging_loss=0.007988, over 14456.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09172, pruned_loss=0.01352, audio_tagging_loss=0.009182, over 3048177.43 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:26:38,951 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379150 2023-11-23 20:26:48,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2527660.0, ans=0.125 2023-11-23 20:26:54,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.14 vs. limit=15.0 2023-11-23 20:27:22,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.21 vs. limit=22.5 2023-11-23 20:27:27,505 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.271e+01 8.849e+01 9.635e+01 1.697e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-23 20:27:29,882 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6450, loss[loss=0.05883, simple_loss=0.07456, pruned_loss=0.01116, audio_tagging_loss=0.01039, over 15874.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09185, pruned_loss=0.0135, audio_tagging_loss=0.009312, over 3043631.23 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:27:41,499 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379200 2023-11-23 20:27:43,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2527993.3333333335, ans=15.0 2023-11-23 20:27:44,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2527993.3333333335, ans=0.125 2023-11-23 20:27:53,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2528060.0, ans=0.125 2023-11-23 20:28:00,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.07 vs. limit=12.0 2023-11-23 20:28:00,577 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.87 vs. limit=22.5 2023-11-23 20:28:24,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2528193.3333333335, ans=0.2 2023-11-23 20:28:30,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2528193.3333333335, ans=0.05 2023-11-23 20:28:32,311 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6500, loss[loss=0.07624, simple_loss=0.1099, pruned_loss=0.01512, audio_tagging_loss=0.006165, over 17309.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09171, pruned_loss=0.0136, audio_tagging_loss=0.00923, over 3048972.16 frames. ], batch size: 64, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:28:33,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2528260.0, ans=0.125 2023-11-23 20:28:43,265 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379250 2023-11-23 20:29:05,701 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 20:29:16,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=12.0 2023-11-23 20:29:20,052 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.28 vs. limit=15.0 2023-11-23 20:29:32,128 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.606e+01 9.202e+01 9.720e+01 1.328e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-23 20:29:35,069 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6550, loss[loss=0.0802, simple_loss=0.1043, pruned_loss=0.01813, audio_tagging_loss=0.009932, over 15010.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09172, pruned_loss=0.01365, audio_tagging_loss=0.009116, over 3054673.04 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:29:35,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2528593.3333333335, ans=0.0 2023-11-23 20:29:46,331 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379300 2023-11-23 20:29:51,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2528660.0, ans=0.2 2023-11-23 20:29:53,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2528660.0, ans=0.125 2023-11-23 20:30:00,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2528726.6666666665, ans=0.0 2023-11-23 20:30:34,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.96 vs. limit=15.0 2023-11-23 20:30:37,595 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6600, loss[loss=0.06645, simple_loss=0.09306, pruned_loss=0.01465, audio_tagging_loss=0.005268, over 14094.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.0918, pruned_loss=0.01363, audio_tagging_loss=0.009041, over 3058581.89 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:30:48,450 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379350 2023-11-23 20:31:28,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.99 vs. limit=10.0 2023-11-23 20:31:29,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.45 vs. limit=12.0 2023-11-23 20:31:31,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2529193.3333333335, ans=0.125 2023-11-23 20:31:37,889 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.263e+01 8.994e+01 9.633e+01 1.189e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 20:31:40,331 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6650, loss[loss=0.07245, simple_loss=0.08671, pruned_loss=0.01863, audio_tagging_loss=0.01046, over 13967.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09108, pruned_loss=0.01344, audio_tagging_loss=0.00902, over 3053656.32 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:31:44,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2529260.0, ans=0.125 2023-11-23 20:31:51,004 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379400 2023-11-23 20:32:10,902 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.12 vs. limit=15.0 2023-11-23 20:32:29,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2529526.6666666665, ans=0.2 2023-11-23 20:32:36,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2529526.6666666665, ans=0.125 2023-11-23 20:32:42,232 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6700, loss[loss=0.08609, simple_loss=0.1242, pruned_loss=0.01654, audio_tagging_loss=0.007463, over 14953.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09208, pruned_loss=0.01369, audio_tagging_loss=0.008898, over 3047369.88 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:32:44,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2529593.3333333335, ans=0.125 2023-11-23 20:32:53,728 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379450 2023-11-23 20:33:07,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2529726.6666666665, ans=0.05 2023-11-23 20:33:09,899 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0 2023-11-23 20:33:10,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2529726.6666666665, ans=0.2 2023-11-23 20:33:29,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2529793.3333333335, ans=0.0 2023-11-23 20:33:32,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2529860.0, ans=0.0 2023-11-23 20:33:42,138 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.739e+01 8.296e+01 8.825e+01 9.532e+01 1.402e+02, threshold=1.765e+02, percent-clipped=0.0 2023-11-23 20:33:45,228 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6750, loss[loss=0.06563, simple_loss=0.07825, pruned_loss=0.01724, audio_tagging_loss=0.009269, over 14526.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09121, pruned_loss=0.01349, audio_tagging_loss=0.009005, over 3045810.70 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:33:55,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2529926.6666666665, ans=0.0 2023-11-23 20:33:56,672 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379500 2023-11-23 20:34:05,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2529993.3333333335, ans=0.125 2023-11-23 20:34:09,429 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.73 vs. limit=12.0 2023-11-23 20:34:19,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.89 vs. limit=10.0 2023-11-23 20:34:28,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2530126.6666666665, ans=0.125 2023-11-23 20:34:47,433 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6800, loss[loss=0.03598, simple_loss=0.04139, pruned_loss=0.004298, audio_tagging_loss=0.01099, over 16458.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09116, pruned_loss=0.01361, audio_tagging_loss=0.009017, over 3037299.32 frames. ], batch size: 66, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:34:49,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2530260.0, ans=0.0 2023-11-23 20:34:58,808 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379550 2023-11-23 20:35:07,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2530326.6666666665, ans=0.125 2023-11-23 20:35:26,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2530460.0, ans=0.1 2023-11-23 20:35:43,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2530526.6666666665, ans=0.125 2023-11-23 20:35:46,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2530526.6666666665, ans=0.125 2023-11-23 20:35:47,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.96 vs. limit=12.0 2023-11-23 20:35:48,138 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.273e+01 8.968e+01 9.529e+01 1.211e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-23 20:35:49,319 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6850, loss[loss=0.06808, simple_loss=0.08693, pruned_loss=0.01549, audio_tagging_loss=0.009123, over 14823.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09103, pruned_loss=0.01356, audio_tagging_loss=0.009042, over 3041445.17 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:35:53,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2530593.3333333335, ans=0.125 2023-11-23 20:35:57,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2530593.3333333335, ans=0.125 2023-11-23 20:36:00,750 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379600 2023-11-23 20:36:51,611 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6900, loss[loss=0.04824, simple_loss=0.0572, pruned_loss=0.007553, audio_tagging_loss=0.01209, over 16150.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09252, pruned_loss=0.01385, audio_tagging_loss=0.008939, over 3045675.95 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:36:59,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2530926.6666666665, ans=0.125 2023-11-23 20:37:02,951 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379650 2023-11-23 20:37:13,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2530993.3333333335, ans=0.0 2023-11-23 20:37:35,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2531126.6666666665, ans=0.125 2023-11-23 20:37:38,825 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 20:37:41,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.79 vs. limit=10.0 2023-11-23 20:37:47,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2531193.3333333335, ans=0.0 2023-11-23 20:37:52,177 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.634e+01 8.329e+01 8.996e+01 9.675e+01 1.152e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 20:37:52,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2531260.0, ans=0.125 2023-11-23 20:37:53,384 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 6950, loss[loss=0.06249, simple_loss=0.08075, pruned_loss=0.01148, audio_tagging_loss=0.01063, over 15867.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09283, pruned_loss=0.01389, audio_tagging_loss=0.008945, over 3049993.32 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:37:58,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2531260.0, ans=0.1 2023-11-23 20:38:04,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379700 2023-11-23 20:38:19,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2531393.3333333335, ans=0.1 2023-11-23 20:38:39,160 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.47 vs. limit=10.0 2023-11-23 20:38:54,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2531593.3333333335, ans=0.125 2023-11-23 20:38:55,292 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7000, loss[loss=0.08238, simple_loss=0.1195, pruned_loss=0.01435, audio_tagging_loss=0.00828, over 15959.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09271, pruned_loss=0.01383, audio_tagging_loss=0.008992, over 3047442.19 frames. ], batch size: 63, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:39:03,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2531593.3333333335, ans=0.0 2023-11-23 20:39:06,362 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379750 2023-11-23 20:39:10,549 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.85 vs. limit=15.0 2023-11-23 20:39:16,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2531660.0, ans=0.125 2023-11-23 20:39:22,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2531726.6666666665, ans=0.125 2023-11-23 20:39:38,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2531793.3333333335, ans=0.0 2023-11-23 20:39:55,965 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.253e+01 8.867e+01 9.647e+01 1.261e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-23 20:39:57,232 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7050, loss[loss=0.05639, simple_loss=0.06538, pruned_loss=0.01074, audio_tagging_loss=0.01296, over 14202.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09181, pruned_loss=0.01359, audio_tagging_loss=0.009048, over 3042286.14 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:39:59,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2531926.6666666665, ans=0.125 2023-11-23 20:40:05,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.84 vs. limit=15.0 2023-11-23 20:40:08,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379800 2023-11-23 20:40:32,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2532060.0, ans=0.1 2023-11-23 20:40:56,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2532193.3333333335, ans=0.95 2023-11-23 20:40:59,240 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7100, loss[loss=0.07962, simple_loss=0.1153, pruned_loss=0.01501, audio_tagging_loss=0.006965, over 16623.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09123, pruned_loss=0.01352, audio_tagging_loss=0.00911, over 3056192.76 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:41:00,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2532260.0, ans=0.0 2023-11-23 20:41:06,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2532260.0, ans=0.1 2023-11-23 20:41:08,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.89 vs. limit=15.0 2023-11-23 20:41:11,201 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379850 2023-11-23 20:41:25,383 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.66 vs. limit=22.5 2023-11-23 20:42:01,915 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.887e+01 8.490e+01 9.157e+01 9.882e+01 1.439e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-23 20:42:01,959 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7150, loss[loss=0.0741, simple_loss=0.09519, pruned_loss=0.01771, audio_tagging_loss=0.008794, over 15417.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09125, pruned_loss=0.01343, audio_tagging_loss=0.009157, over 3052355.50 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:42:02,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2532593.3333333335, ans=0.2 2023-11-23 20:42:13,324 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379900 2023-11-23 20:42:20,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2532660.0, ans=0.125 2023-11-23 20:43:04,348 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7200, loss[loss=0.05481, simple_loss=0.06142, pruned_loss=0.01195, audio_tagging_loss=0.01215, over 13843.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09095, pruned_loss=0.01348, audio_tagging_loss=0.009283, over 3045716.11 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:43:11,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2532926.6666666665, ans=0.1 2023-11-23 20:43:11,081 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 20:43:15,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 379950 2023-11-23 20:43:17,218 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.44 vs. limit=22.5 2023-11-23 20:43:25,115 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 20:43:39,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2533060.0, ans=0.025 2023-11-23 20:44:05,865 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7250, loss[loss=0.06756, simple_loss=0.09347, pruned_loss=0.01211, audio_tagging_loss=0.008714, over 14397.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09075, pruned_loss=0.01342, audio_tagging_loss=0.009293, over 3049156.68 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:44:06,966 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.235e+01 8.202e+01 8.739e+01 9.383e+01 1.704e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-23 20:44:17,318 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380000 2023-11-23 20:44:36,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.50 vs. limit=8.0 2023-11-23 20:44:54,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.42 vs. limit=12.0 2023-11-23 20:45:02,443 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.51 vs. limit=10.0 2023-11-23 20:45:03,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2533526.6666666665, ans=0.125 2023-11-23 20:45:10,596 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7300, loss[loss=0.08822, simple_loss=0.1055, pruned_loss=0.02435, audio_tagging_loss=0.01112, over 14788.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.08989, pruned_loss=0.01336, audio_tagging_loss=0.00927, over 3043318.09 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:45:17,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2533593.3333333335, ans=0.1 2023-11-23 20:45:22,462 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380050 2023-11-23 20:45:42,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2533726.6666666665, ans=0.125 2023-11-23 20:45:47,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2533793.3333333335, ans=0.125 2023-11-23 20:45:48,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.62 vs. limit=15.0 2023-11-23 20:46:06,600 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.81 vs. limit=15.0 2023-11-23 20:46:14,070 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7350, loss[loss=0.07791, simple_loss=0.1078, pruned_loss=0.01348, audio_tagging_loss=0.01055, over 16108.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09124, pruned_loss=0.01352, audio_tagging_loss=0.009098, over 3047444.78 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:46:15,159 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.465e+01 9.022e+01 1.001e+02 1.675e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-23 20:46:15,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2533926.6666666665, ans=0.125 2023-11-23 20:46:19,556 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.24 vs. limit=15.0 2023-11-23 20:46:25,452 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380100 2023-11-23 20:46:30,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2533993.3333333335, ans=0.0 2023-11-23 20:46:32,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2533993.3333333335, ans=0.125 2023-11-23 20:46:35,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2533993.3333333335, ans=0.0 2023-11-23 20:46:47,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.93 vs. limit=15.0 2023-11-23 20:47:01,669 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=15.64 vs. limit=15.0 2023-11-23 20:47:12,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2534193.3333333335, ans=0.1 2023-11-23 20:47:13,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.20 vs. limit=15.0 2023-11-23 20:47:16,157 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7400, loss[loss=0.06502, simple_loss=0.09127, pruned_loss=0.01268, audio_tagging_loss=0.00671, over 14204.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09101, pruned_loss=0.01345, audio_tagging_loss=0.009045, over 3041911.50 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:47:27,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380150 2023-11-23 20:47:53,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2534460.0, ans=10.0 2023-11-23 20:47:58,015 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.93 vs. limit=15.0 2023-11-23 20:47:59,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2534460.0, ans=0.0 2023-11-23 20:48:18,444 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7450, loss[loss=0.07418, simple_loss=0.08611, pruned_loss=0.01844, audio_tagging_loss=0.01268, over 16340.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09133, pruned_loss=0.01369, audio_tagging_loss=0.008925, over 3039082.06 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:48:19,594 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.435e+01 9.167e+01 9.796e+01 1.283e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-23 20:48:23,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2534593.3333333335, ans=0.125 2023-11-23 20:48:29,289 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380200 2023-11-23 20:48:31,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.56 vs. limit=22.5 2023-11-23 20:48:55,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2534793.3333333335, ans=0.07 2023-11-23 20:49:04,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2534793.3333333335, ans=0.0 2023-11-23 20:49:20,999 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7500, loss[loss=0.06714, simple_loss=0.09331, pruned_loss=0.01327, audio_tagging_loss=0.007207, over 14842.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09161, pruned_loss=0.01394, audio_tagging_loss=0.008935, over 3037323.29 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:49:31,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2534926.6666666665, ans=0.125 2023-11-23 20:49:32,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380250 2023-11-23 20:49:36,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2534993.3333333335, ans=0.125 2023-11-23 20:50:12,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2535193.3333333335, ans=0.1 2023-11-23 20:50:18,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2535193.3333333335, ans=0.0 2023-11-23 20:50:21,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2535193.3333333335, ans=0.0 2023-11-23 20:50:23,265 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7550, loss[loss=0.05879, simple_loss=0.06368, pruned_loss=0.01466, audio_tagging_loss=0.01228, over 14369.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09096, pruned_loss=0.01371, audio_tagging_loss=0.008928, over 3048948.25 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0 2023-11-23 20:50:23,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2535260.0, ans=0.125 2023-11-23 20:50:24,461 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.259e+01 8.879e+01 9.874e+01 1.226e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-23 20:50:34,018 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380300 2023-11-23 20:50:34,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2535326.6666666665, ans=0.5 2023-11-23 20:50:48,394 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.81 vs. limit=10.0 2023-11-23 20:51:20,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2535526.6666666665, ans=0.1 2023-11-23 20:51:25,748 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7600, loss[loss=0.05221, simple_loss=0.06531, pruned_loss=0.01007, audio_tagging_loss=0.009486, over 15180.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.0908, pruned_loss=0.01354, audio_tagging_loss=0.008971, over 3046976.43 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:51:36,513 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380350 2023-11-23 20:51:38,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.45 vs. limit=15.0 2023-11-23 20:52:24,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.97 vs. limit=12.0 2023-11-23 20:52:27,497 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7650, loss[loss=0.0622, simple_loss=0.082, pruned_loss=0.01111, audio_tagging_loss=0.0101, over 14503.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09046, pruned_loss=0.01356, audio_tagging_loss=0.008993, over 3034461.73 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:52:29,181 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.957e+01 8.327e+01 8.912e+01 9.610e+01 1.311e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-23 20:52:39,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380400 2023-11-23 20:52:42,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2535993.3333333335, ans=0.0 2023-11-23 20:52:52,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2536060.0, ans=0.125 2023-11-23 20:52:58,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.68 vs. limit=15.0 2023-11-23 20:53:12,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2536126.6666666665, ans=0.125 2023-11-23 20:53:31,359 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7700, loss[loss=0.08099, simple_loss=0.1094, pruned_loss=0.01835, audio_tagging_loss=0.007923, over 16167.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09112, pruned_loss=0.01361, audio_tagging_loss=0.00906, over 3040653.38 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:53:37,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2536260.0, ans=0.125 2023-11-23 20:53:42,056 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380450 2023-11-23 20:53:57,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.60 vs. limit=22.5 2023-11-23 20:54:30,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2536526.6666666665, ans=0.125 2023-11-23 20:54:32,984 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7750, loss[loss=0.05474, simple_loss=0.07388, pruned_loss=0.008494, audio_tagging_loss=0.00931, over 13927.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09226, pruned_loss=0.01381, audio_tagging_loss=0.009045, over 3044352.78 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:54:34,649 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.446e+01 9.126e+01 9.765e+01 1.172e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-23 20:54:36,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2536593.3333333335, ans=0.0 2023-11-23 20:54:36,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.29 vs. limit=6.0 2023-11-23 20:54:39,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2536593.3333333335, ans=0.0 2023-11-23 20:54:40,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2536593.3333333335, ans=0.125 2023-11-23 20:54:44,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380500 2023-11-23 20:54:44,862 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.87 vs. limit=12.0 2023-11-23 20:55:04,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2536726.6666666665, ans=0.125 2023-11-23 20:55:26,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2536860.0, ans=0.2 2023-11-23 20:55:34,689 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7800, loss[loss=0.07319, simple_loss=0.092, pruned_loss=0.0166, audio_tagging_loss=0.01059, over 16223.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09221, pruned_loss=0.01365, audio_tagging_loss=0.009127, over 3044215.00 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:55:43,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2536926.6666666665, ans=0.125 2023-11-23 20:55:44,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2536926.6666666665, ans=0.125 2023-11-23 20:55:46,029 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380550 2023-11-23 20:56:26,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2537193.3333333335, ans=0.1 2023-11-23 20:56:30,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2537193.3333333335, ans=0.1 2023-11-23 20:56:36,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.14 vs. limit=12.0 2023-11-23 20:56:37,565 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7850, loss[loss=0.0922, simple_loss=0.1315, pruned_loss=0.02085, audio_tagging_loss=0.005622, over 15577.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09282, pruned_loss=0.01375, audio_tagging_loss=0.00909, over 3046413.72 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:56:38,728 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.376e+01 9.070e+01 9.715e+01 1.480e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-23 20:56:42,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.43 vs. limit=22.5 2023-11-23 20:56:48,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2537260.0, ans=0.07 2023-11-23 20:56:48,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.81 vs. limit=22.5 2023-11-23 20:56:49,113 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380600 2023-11-23 20:57:14,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2537460.0, ans=0.125 2023-11-23 20:57:16,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.85 vs. limit=15.0 2023-11-23 20:57:29,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2537526.6666666665, ans=0.125 2023-11-23 20:57:31,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2537526.6666666665, ans=0.125 2023-11-23 20:57:38,614 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.26 vs. limit=15.0 2023-11-23 20:57:40,337 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7900, loss[loss=0.03462, simple_loss=0.03386, pruned_loss=0.006261, audio_tagging_loss=0.01143, over 14686.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09199, pruned_loss=0.01367, audio_tagging_loss=0.009208, over 3046221.50 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:57:51,873 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380650 2023-11-23 20:57:53,813 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.37 vs. limit=15.0 2023-11-23 20:58:21,446 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.00 vs. limit=15.0 2023-11-23 20:58:22,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2537793.3333333335, ans=0.125 2023-11-23 20:58:42,643 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 7950, loss[loss=0.0623, simple_loss=0.0808, pruned_loss=0.01145, audio_tagging_loss=0.01045, over 14635.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09293, pruned_loss=0.01401, audio_tagging_loss=0.009199, over 3043430.78 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:58:43,748 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.414e+01 9.153e+01 9.684e+01 1.303e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-23 20:58:54,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380700 2023-11-23 20:58:55,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2537993.3333333335, ans=0.125 2023-11-23 20:58:57,534 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 20:59:00,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2537993.3333333335, ans=0.0 2023-11-23 20:59:28,933 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 20:59:44,783 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8000, loss[loss=0.08286, simple_loss=0.1045, pruned_loss=0.01971, audio_tagging_loss=0.01088, over 15827.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09254, pruned_loss=0.01394, audio_tagging_loss=0.009335, over 3044075.38 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 20:59:56,914 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380750 2023-11-23 21:00:08,672 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.19 vs. limit=22.5 2023-11-23 21:00:20,677 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.56 vs. limit=15.0 2023-11-23 21:00:47,871 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8050, loss[loss=0.07017, simple_loss=0.09673, pruned_loss=0.01279, audio_tagging_loss=0.009009, over 15188.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09196, pruned_loss=0.01383, audio_tagging_loss=0.009375, over 3049086.36 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0 2023-11-23 21:00:48,949 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.054e+01 8.478e+01 9.043e+01 9.667e+01 1.192e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 21:00:59,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380800 2023-11-23 21:01:00,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2538660.0, ans=0.125 2023-11-23 21:01:12,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=2538726.6666666665, ans=0.02 2023-11-23 21:01:20,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2538726.6666666665, ans=0.1 2023-11-23 21:01:35,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2538793.3333333335, ans=0.125 2023-11-23 21:01:50,367 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8100, loss[loss=0.06353, simple_loss=0.09341, pruned_loss=0.0089, audio_tagging_loss=0.007926, over 15740.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.0917, pruned_loss=0.01375, audio_tagging_loss=0.009278, over 3051397.90 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:01:51,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2538926.6666666665, ans=0.125 2023-11-23 21:02:01,491 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380850 2023-11-23 21:02:10,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2538993.3333333335, ans=0.07 2023-11-23 21:02:21,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2539060.0, ans=0.125 2023-11-23 21:02:22,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2539060.0, ans=0.0 2023-11-23 21:02:39,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.51 vs. limit=22.5 2023-11-23 21:02:40,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2539193.3333333335, ans=0.0 2023-11-23 21:02:43,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2539193.3333333335, ans=0.125 2023-11-23 21:02:45,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.13 vs. limit=15.0 2023-11-23 21:02:52,172 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8150, loss[loss=0.07772, simple_loss=0.108, pruned_loss=0.01714, audio_tagging_loss=0.006583, over 15175.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09153, pruned_loss=0.01365, audio_tagging_loss=0.009147, over 3047047.13 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:02:54,491 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.341e+01 9.005e+01 9.405e+01 1.221e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 21:03:03,691 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380900 2023-11-23 21:03:08,553 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:03:09,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2539326.6666666665, ans=0.125 2023-11-23 21:03:20,396 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:03:21,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2539393.3333333335, ans=0.1 2023-11-23 21:03:31,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2539460.0, ans=0.1 2023-11-23 21:03:42,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2539526.6666666665, ans=0.0 2023-11-23 21:03:50,072 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.36 vs. limit=22.5 2023-11-23 21:03:50,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2539526.6666666665, ans=0.0 2023-11-23 21:03:54,167 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8200, loss[loss=0.0694, simple_loss=0.09407, pruned_loss=0.01473, audio_tagging_loss=0.007641, over 14827.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09311, pruned_loss=0.01391, audio_tagging_loss=0.008925, over 3046305.31 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:03:55,305 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 21:04:05,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 380950 2023-11-23 21:04:13,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2539660.0, ans=0.125 2023-11-23 21:04:56,986 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8250, loss[loss=0.07525, simple_loss=0.1013, pruned_loss=0.01653, audio_tagging_loss=0.008046, over 15175.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09188, pruned_loss=0.01364, audio_tagging_loss=0.008899, over 3045795.22 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:04:59,276 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.237e+01 8.988e+01 9.644e+01 1.224e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-23 21:05:05,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0 2023-11-23 21:05:07,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381000 2023-11-23 21:05:15,107 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0 2023-11-23 21:05:15,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2539993.3333333335, ans=0.1 2023-11-23 21:05:23,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2540060.0, ans=0.125 2023-11-23 21:05:44,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2540126.6666666665, ans=0.2 2023-11-23 21:05:53,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2540193.3333333335, ans=0.0 2023-11-23 21:05:55,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2540193.3333333335, ans=0.125 2023-11-23 21:05:59,295 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8300, loss[loss=0.06757, simple_loss=0.08679, pruned_loss=0.01274, audio_tagging_loss=0.01143, over 15487.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09183, pruned_loss=0.01356, audio_tagging_loss=0.00893, over 3047617.55 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:05:59,587 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:06:04,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2540260.0, ans=0.0 2023-11-23 21:06:10,711 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381050 2023-11-23 21:06:23,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2540393.3333333335, ans=0.125 2023-11-23 21:06:27,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2540393.3333333335, ans=0.0 2023-11-23 21:06:35,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2540460.0, ans=0.0 2023-11-23 21:06:40,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2540460.0, ans=0.2 2023-11-23 21:06:47,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2540526.6666666665, ans=0.125 2023-11-23 21:06:51,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2540526.6666666665, ans=0.1 2023-11-23 21:06:57,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2540526.6666666665, ans=0.125 2023-11-23 21:07:01,059 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8350, loss[loss=0.067, simple_loss=0.09397, pruned_loss=0.01255, audio_tagging_loss=0.007467, over 16134.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09229, pruned_loss=0.01368, audio_tagging_loss=0.008921, over 3059439.75 frames. ], batch size: 63, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:07:03,381 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.724e+01 8.451e+01 9.185e+01 9.824e+01 1.570e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-23 21:07:10,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2540593.3333333335, ans=0.125 2023-11-23 21:07:11,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381100 2023-11-23 21:07:28,659 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.59 vs. limit=22.5 2023-11-23 21:07:55,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2540860.0, ans=0.125 2023-11-23 21:07:58,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2540860.0, ans=0.125 2023-11-23 21:08:02,555 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8400, loss[loss=0.07351, simple_loss=0.08671, pruned_loss=0.01823, audio_tagging_loss=0.01193, over 15080.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09224, pruned_loss=0.01365, audio_tagging_loss=0.008917, over 3058291.83 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:08:03,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2540926.6666666665, ans=0.1 2023-11-23 21:08:05,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2540926.6666666665, ans=0.0 2023-11-23 21:08:11,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2540926.6666666665, ans=0.125 2023-11-23 21:08:13,830 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381150 2023-11-23 21:08:42,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2541126.6666666665, ans=0.125 2023-11-23 21:08:51,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2541193.3333333335, ans=0.0 2023-11-23 21:08:52,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2541193.3333333335, ans=0.125 2023-11-23 21:08:52,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.45 vs. limit=22.5 2023-11-23 21:09:02,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2541193.3333333335, ans=0.125 2023-11-23 21:09:04,790 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8450, loss[loss=0.04269, simple_loss=0.05281, pruned_loss=0.006748, audio_tagging_loss=0.009534, over 15361.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09181, pruned_loss=0.01361, audio_tagging_loss=0.008952, over 3051438.10 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:09:08,224 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.427e+01 8.936e+01 9.652e+01 1.220e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 21:09:15,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381200 2023-11-23 21:09:21,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2541326.6666666665, ans=0.125 2023-11-23 21:09:27,445 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.05 vs. limit=15.0 2023-11-23 21:09:58,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2541526.6666666665, ans=0.05 2023-11-23 21:10:05,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2541526.6666666665, ans=0.2 2023-11-23 21:10:07,130 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8500, loss[loss=0.05799, simple_loss=0.07889, pruned_loss=0.008103, audio_tagging_loss=0.01045, over 14557.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09196, pruned_loss=0.0137, audio_tagging_loss=0.00897, over 3054375.93 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:10:15,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2541593.3333333335, ans=0.0 2023-11-23 21:10:18,022 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381250 2023-11-23 21:10:24,054 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.75 vs. limit=12.0 2023-11-23 21:10:33,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2541726.6666666665, ans=0.1 2023-11-23 21:10:57,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2541860.0, ans=0.0 2023-11-23 21:11:09,324 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8550, loss[loss=0.07732, simple_loss=0.1052, pruned_loss=0.01608, audio_tagging_loss=0.008653, over 16304.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.0927, pruned_loss=0.01376, audio_tagging_loss=0.009, over 3060549.65 frames. ], batch size: 63, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:11:12,875 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.400e+01 9.293e+01 9.776e+01 1.237e+02, threshold=1.859e+02, percent-clipped=0.0 2023-11-23 21:11:18,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.67 vs. limit=15.0 2023-11-23 21:11:20,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381300 2023-11-23 21:11:24,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2541993.3333333335, ans=0.125 2023-11-23 21:11:32,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2541993.3333333335, ans=0.125 2023-11-23 21:11:33,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2542060.0, ans=0.1 2023-11-23 21:11:37,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2542060.0, ans=0.125 2023-11-23 21:11:48,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2542126.6666666665, ans=0.2 2023-11-23 21:11:57,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.02 vs. limit=15.0 2023-11-23 21:11:59,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2542193.3333333335, ans=0.1 2023-11-23 21:12:08,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2542193.3333333335, ans=0.125 2023-11-23 21:12:10,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2542260.0, ans=0.125 2023-11-23 21:12:11,443 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8600, loss[loss=0.07724, simple_loss=0.1031, pruned_loss=0.01548, audio_tagging_loss=0.01022, over 14842.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09359, pruned_loss=0.01397, audio_tagging_loss=0.009039, over 3052543.95 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:12:22,654 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381350 2023-11-23 21:12:24,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2542326.6666666665, ans=0.0 2023-11-23 21:13:01,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2542526.6666666665, ans=0.125 2023-11-23 21:13:07,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.62 vs. limit=15.0 2023-11-23 21:13:13,096 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8650, loss[loss=0.07675, simple_loss=0.0959, pruned_loss=0.01677, audio_tagging_loss=0.01203, over 15162.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09345, pruned_loss=0.0139, audio_tagging_loss=0.009071, over 3048396.30 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:13:16,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.050e+01 8.560e+01 9.209e+01 9.798e+01 1.197e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-23 21:13:18,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.74 vs. limit=6.0 2023-11-23 21:13:23,671 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381400 2023-11-23 21:13:54,990 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.14 vs. limit=15.0 2023-11-23 21:14:15,425 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8700, loss[loss=0.06849, simple_loss=0.08911, pruned_loss=0.01211, audio_tagging_loss=0.01183, over 14350.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09374, pruned_loss=0.01389, audio_tagging_loss=0.009071, over 3046216.55 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:14:17,296 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0 2023-11-23 21:14:26,566 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381450 2023-11-23 21:14:28,390 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.80 vs. limit=15.0 2023-11-23 21:14:29,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2542993.3333333335, ans=0.125 2023-11-23 21:14:36,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2542993.3333333335, ans=0.125 2023-11-23 21:14:38,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2542993.3333333335, ans=0.07 2023-11-23 21:14:39,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2543060.0, ans=0.0 2023-11-23 21:14:47,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2543060.0, ans=0.125 2023-11-23 21:15:17,586 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8750, loss[loss=0.08016, simple_loss=0.1097, pruned_loss=0.01703, audio_tagging_loss=0.008291, over 14842.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09397, pruned_loss=0.01394, audio_tagging_loss=0.009133, over 3051125.42 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:15:21,703 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.453e+01 8.955e+01 9.752e+01 1.523e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-23 21:15:29,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381500 2023-11-23 21:15:36,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.59 vs. limit=22.5 2023-11-23 21:15:38,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2543326.6666666665, ans=0.125 2023-11-23 21:15:54,645 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0 2023-11-23 21:15:56,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2543460.0, ans=0.1 2023-11-23 21:15:58,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2543460.0, ans=0.125 2023-11-23 21:16:02,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2543460.0, ans=0.0 2023-11-23 21:16:12,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2543526.6666666665, ans=0.0 2023-11-23 21:16:20,099 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8800, loss[loss=0.09012, simple_loss=0.1211, pruned_loss=0.02045, audio_tagging_loss=0.009116, over 15942.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09335, pruned_loss=0.01379, audio_tagging_loss=0.009279, over 3055651.53 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:16:31,272 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381550 2023-11-23 21:16:48,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2543726.6666666665, ans=0.125 2023-11-23 21:17:04,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2543793.3333333335, ans=0.1 2023-11-23 21:17:17,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2543860.0, ans=0.0 2023-11-23 21:17:22,301 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8850, loss[loss=0.06966, simple_loss=0.09056, pruned_loss=0.01621, audio_tagging_loss=0.008159, over 15904.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09384, pruned_loss=0.01382, audio_tagging_loss=0.009303, over 3055230.23 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:17:25,953 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.726e+01 8.519e+01 9.142e+01 9.754e+01 1.117e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-23 21:17:33,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381600 2023-11-23 21:17:35,045 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 21:17:45,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2543993.3333333335, ans=0.0 2023-11-23 21:17:54,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2544060.0, ans=0.2 2023-11-23 21:18:15,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2544193.3333333335, ans=0.2 2023-11-23 21:18:21,502 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0 2023-11-23 21:18:25,684 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8900, loss[loss=0.03985, simple_loss=0.04445, pruned_loss=0.007037, audio_tagging_loss=0.01059, over 15093.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09327, pruned_loss=0.01366, audio_tagging_loss=0.009201, over 3060382.34 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:18:28,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2544260.0, ans=0.125 2023-11-23 21:18:33,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2544260.0, ans=0.125 2023-11-23 21:18:34,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2544260.0, ans=0.125 2023-11-23 21:18:36,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381650 2023-11-23 21:18:38,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2544326.6666666665, ans=0.2 2023-11-23 21:18:40,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2544326.6666666665, ans=0.0 2023-11-23 21:18:52,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2544393.3333333335, ans=0.2 2023-11-23 21:19:20,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2544526.6666666665, ans=0.125 2023-11-23 21:19:24,443 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=7.58 vs. limit=12.0 2023-11-23 21:19:27,556 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 8950, loss[loss=0.07778, simple_loss=0.1047, pruned_loss=0.01723, audio_tagging_loss=0.00822, over 14384.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09368, pruned_loss=0.01375, audio_tagging_loss=0.009121, over 3059643.27 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:19:29,322 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0 2023-11-23 21:19:32,786 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.301e+01 8.310e+01 8.895e+01 9.557e+01 1.131e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 21:19:33,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2544593.3333333335, ans=0.125 2023-11-23 21:19:39,376 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381700 2023-11-23 21:19:56,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2544726.6666666665, ans=0.2 2023-11-23 21:19:59,119 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:20:09,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2544793.3333333335, ans=0.125 2023-11-23 21:20:16,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2544860.0, ans=0.125 2023-11-23 21:20:29,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.73 vs. limit=22.5 2023-11-23 21:20:30,425 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9000, loss[loss=0.0871, simple_loss=0.1178, pruned_loss=0.01982, audio_tagging_loss=0.008386, over 16210.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09484, pruned_loss=0.014, audio_tagging_loss=0.009005, over 3055296.71 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:20:30,426 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 21:20:55,444 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6017, 3.6061, 3.8594, 3.3956], device='cuda:3') 2023-11-23 21:21:06,659 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8296, 4.9240, 5.0374, 4.8736], device='cuda:3') 2023-11-23 21:21:08,325 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4018, 4.1362, 4.5178, 4.2479], device='cuda:3') 2023-11-23 21:21:10,436 INFO [train_asr.py:1253] (3/4) Epoch 32, validation: loss=0.05909, simple_loss=0.05099, pruned_loss=0.005136, audio_tagging_loss=0.02845, over 4681554.00 frames. 2023-11-23 21:21:10,437 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 21:21:20,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2544926.6666666665, ans=0.0 2023-11-23 21:21:21,706 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381750 2023-11-23 21:21:27,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2544993.3333333335, ans=0.0 2023-11-23 21:21:29,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2544993.3333333335, ans=0.04949747468305833 2023-11-23 21:21:37,704 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.71 vs. limit=22.5 2023-11-23 21:21:49,598 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.11 vs. limit=10.0 2023-11-23 21:22:04,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2545193.3333333335, ans=0.1 2023-11-23 21:22:04,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2545193.3333333335, ans=0.2 2023-11-23 21:22:12,918 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9050, loss[loss=0.04959, simple_loss=0.05758, pruned_loss=0.01045, audio_tagging_loss=0.01035, over 14007.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09436, pruned_loss=0.01396, audio_tagging_loss=0.008914, over 3048466.22 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:22:17,520 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.661e+01 9.152e+01 9.951e+01 1.252e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-23 21:22:24,244 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381800 2023-11-23 21:22:35,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.91 vs. limit=6.0 2023-11-23 21:23:13,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2545526.6666666665, ans=0.125 2023-11-23 21:23:15,523 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9100, loss[loss=0.07419, simple_loss=0.1087, pruned_loss=0.0155, audio_tagging_loss=0.004361, over 15048.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.0936, pruned_loss=0.01364, audio_tagging_loss=0.008921, over 3049100.59 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:23:26,909 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381850 2023-11-23 21:23:37,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2545660.0, ans=0.2 2023-11-23 21:23:43,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.84 vs. limit=15.0 2023-11-23 21:23:47,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2545726.6666666665, ans=0.0 2023-11-23 21:23:55,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2545793.3333333335, ans=0.125 2023-11-23 21:24:17,789 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9150, loss[loss=0.0802, simple_loss=0.113, pruned_loss=0.01448, audio_tagging_loss=0.009196, over 15319.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.0933, pruned_loss=0.01365, audio_tagging_loss=0.008945, over 3043661.24 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:24:19,706 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.45 vs. limit=22.5 2023-11-23 21:24:23,247 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.696e+01 8.321e+01 8.921e+01 9.557e+01 1.166e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 21:24:29,256 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381900 2023-11-23 21:24:44,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2546060.0, ans=0.125 2023-11-23 21:24:50,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2546060.0, ans=0.125 2023-11-23 21:24:51,367 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.66 vs. limit=15.0 2023-11-23 21:24:54,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2546126.6666666665, ans=0.0 2023-11-23 21:24:56,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2546126.6666666665, ans=10.0 2023-11-23 21:25:05,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2546126.6666666665, ans=0.2 2023-11-23 21:25:20,088 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9200, loss[loss=0.08099, simple_loss=0.1073, pruned_loss=0.01887, audio_tagging_loss=0.008492, over 15137.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09171, pruned_loss=0.01355, audio_tagging_loss=0.008998, over 3042692.52 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:25:31,543 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 381950 2023-11-23 21:25:32,005 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.27 vs. limit=22.5 2023-11-23 21:25:56,341 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:26:04,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2546460.0, ans=10.0 2023-11-23 21:26:06,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2546460.0, ans=0.1 2023-11-23 21:26:21,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2546593.3333333335, ans=0.025 2023-11-23 21:26:22,417 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9250, loss[loss=0.07751, simple_loss=0.1084, pruned_loss=0.01711, audio_tagging_loss=0.006195, over 14274.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09201, pruned_loss=0.01367, audio_tagging_loss=0.008912, over 3041349.83 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:26:22,965 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.12 vs. limit=15.0 2023-11-23 21:26:28,942 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.321e+01 8.948e+01 9.831e+01 1.146e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-23 21:26:33,956 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382000 2023-11-23 21:27:17,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2546860.0, ans=0.5 2023-11-23 21:27:25,766 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9300, loss[loss=0.07212, simple_loss=0.09505, pruned_loss=0.01421, audio_tagging_loss=0.01039, over 15395.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09125, pruned_loss=0.01356, audio_tagging_loss=0.008928, over 3041137.15 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:27:26,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.86 vs. limit=6.0 2023-11-23 21:27:36,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382050 2023-11-23 21:28:08,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2547126.6666666665, ans=0.125 2023-11-23 21:28:15,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2547193.3333333335, ans=0.125 2023-11-23 21:28:19,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2547193.3333333335, ans=0.1 2023-11-23 21:28:27,245 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9350, loss[loss=0.07791, simple_loss=0.1056, pruned_loss=0.01867, audio_tagging_loss=0.006424, over 17001.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09045, pruned_loss=0.01327, audio_tagging_loss=0.008993, over 3038986.45 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:28:33,552 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.368e+01 8.391e+01 8.927e+01 9.865e+01 1.174e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 21:28:38,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382100 2023-11-23 21:28:39,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2547326.6666666665, ans=0.125 2023-11-23 21:28:47,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2547326.6666666665, ans=0.1 2023-11-23 21:28:57,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2547393.3333333335, ans=0.0 2023-11-23 21:28:58,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2547393.3333333335, ans=0.125 2023-11-23 21:29:05,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2547460.0, ans=0.125 2023-11-23 21:29:10,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2547460.0, ans=0.0 2023-11-23 21:29:29,307 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9400, loss[loss=0.06112, simple_loss=0.07843, pruned_loss=0.01187, audio_tagging_loss=0.01003, over 14053.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09054, pruned_loss=0.01338, audio_tagging_loss=0.009089, over 3038657.17 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:29:37,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.88 vs. limit=15.0 2023-11-23 21:29:40,649 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382150 2023-11-23 21:30:00,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.27 vs. limit=15.0 2023-11-23 21:30:05,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2547726.6666666665, ans=0.0 2023-11-23 21:30:29,140 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 21:30:31,444 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9450, loss[loss=0.08457, simple_loss=0.1221, pruned_loss=0.01649, audio_tagging_loss=0.007019, over 15262.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09189, pruned_loss=0.01367, audio_tagging_loss=0.00906, over 3044621.30 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:30:36,979 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:30:37,932 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.414e+01 9.183e+01 9.854e+01 1.204e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-23 21:30:43,354 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382200 2023-11-23 21:30:55,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2547993.3333333335, ans=0.5 2023-11-23 21:31:01,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2548060.0, ans=0.125 2023-11-23 21:31:22,418 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:31:29,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2548193.3333333335, ans=0.125 2023-11-23 21:31:33,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2548260.0, ans=0.1 2023-11-23 21:31:34,401 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9500, loss[loss=0.06752, simple_loss=0.08796, pruned_loss=0.01331, audio_tagging_loss=0.01023, over 16011.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09179, pruned_loss=0.0137, audio_tagging_loss=0.009159, over 3041559.12 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:31:45,741 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382250 2023-11-23 21:31:45,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2548326.6666666665, ans=0.1 2023-11-23 21:32:00,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.87 vs. limit=15.0 2023-11-23 21:32:11,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2548460.0, ans=0.125 2023-11-23 21:32:18,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2548460.0, ans=0.125 2023-11-23 21:32:23,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2548526.6666666665, ans=0.1 2023-11-23 21:32:36,316 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9550, loss[loss=0.07242, simple_loss=0.08643, pruned_loss=0.01874, audio_tagging_loss=0.01046, over 14818.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09167, pruned_loss=0.01356, audio_tagging_loss=0.009237, over 3034399.99 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:32:42,273 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.598e+01 9.160e+01 9.932e+01 1.326e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-23 21:32:47,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382300 2023-11-23 21:33:02,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2548726.6666666665, ans=0.0 2023-11-23 21:33:06,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2548726.6666666665, ans=0.125 2023-11-23 21:33:12,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2548793.3333333335, ans=0.125 2023-11-23 21:33:13,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2548793.3333333335, ans=0.1 2023-11-23 21:33:13,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2548793.3333333335, ans=0.125 2023-11-23 21:33:13,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2548793.3333333335, ans=0.125 2023-11-23 21:33:22,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2548793.3333333335, ans=0.2 2023-11-23 21:33:28,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2548860.0, ans=0.125 2023-11-23 21:33:34,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.86 vs. limit=22.5 2023-11-23 21:33:37,963 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9600, loss[loss=0.07224, simple_loss=0.09352, pruned_loss=0.0138, audio_tagging_loss=0.01169, over 15640.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09074, pruned_loss=0.01342, audio_tagging_loss=0.009289, over 3040491.35 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:33:50,115 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382350 2023-11-23 21:34:01,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2548993.3333333335, ans=0.2 2023-11-23 21:34:24,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2549126.6666666665, ans=0.125 2023-11-23 21:34:41,434 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9650, loss[loss=0.08102, simple_loss=0.1235, pruned_loss=0.0121, audio_tagging_loss=0.007151, over 14228.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09119, pruned_loss=0.01345, audio_tagging_loss=0.009428, over 3038478.35 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:34:47,298 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.285e+01 8.434e+01 9.003e+01 9.684e+01 1.226e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 21:34:49,574 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.95 vs. limit=22.5 2023-11-23 21:34:52,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382400 2023-11-23 21:35:03,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0 2023-11-23 21:35:05,429 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.75 vs. limit=22.5 2023-11-23 21:35:12,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2549393.3333333335, ans=0.125 2023-11-23 21:35:44,310 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9700, loss[loss=0.07512, simple_loss=0.09779, pruned_loss=0.01734, audio_tagging_loss=0.008886, over 15148.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09115, pruned_loss=0.01345, audio_tagging_loss=0.009227, over 3041984.37 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:35:55,022 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382450 2023-11-23 21:36:20,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.29 vs. limit=5.0 2023-11-23 21:36:21,817 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.27 vs. limit=15.0 2023-11-23 21:36:24,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.14 vs. limit=15.0 2023-11-23 21:36:45,381 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9750, loss[loss=0.08978, simple_loss=0.126, pruned_loss=0.01834, audio_tagging_loss=0.00844, over 15134.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09107, pruned_loss=0.01348, audio_tagging_loss=0.009162, over 3040131.41 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:36:51,913 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.362e+01 8.986e+01 9.723e+01 1.186e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 21:36:56,701 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382500 2023-11-23 21:37:06,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2549993.3333333335, ans=0.0 2023-11-23 21:37:10,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2550060.0, ans=0.125 2023-11-23 21:37:13,199 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:37:21,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.81 vs. limit=15.0 2023-11-23 21:37:42,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2550193.3333333335, ans=0.125 2023-11-23 21:37:47,725 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9800, loss[loss=0.0826, simple_loss=0.1039, pruned_loss=0.01973, audio_tagging_loss=0.01092, over 15291.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09076, pruned_loss=0.0135, audio_tagging_loss=0.009097, over 3039372.78 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:37:55,026 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.41 vs. limit=15.0 2023-11-23 21:37:59,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382550 2023-11-23 21:38:03,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2550326.6666666665, ans=0.125 2023-11-23 21:38:16,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2550393.3333333335, ans=0.125 2023-11-23 21:38:24,032 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0 2023-11-23 21:38:30,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2550460.0, ans=0.0 2023-11-23 21:38:43,008 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 21:38:46,497 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=15.0 2023-11-23 21:38:47,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2550526.6666666665, ans=0.0 2023-11-23 21:38:50,712 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9850, loss[loss=0.04929, simple_loss=0.06712, pruned_loss=0.008144, audio_tagging_loss=0.007586, over 14583.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09102, pruned_loss=0.01346, audio_tagging_loss=0.008979, over 3040643.10 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:38:56,526 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.500e+01 9.067e+01 9.984e+01 1.563e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-23 21:39:01,363 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382600 2023-11-23 21:39:02,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2550660.0, ans=0.2 2023-11-23 21:39:44,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2550860.0, ans=0.125 2023-11-23 21:39:52,827 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9900, loss[loss=0.04898, simple_loss=0.06335, pruned_loss=0.006643, audio_tagging_loss=0.01066, over 14734.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09109, pruned_loss=0.01344, audio_tagging_loss=0.008881, over 3040790.75 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:40:04,330 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382650 2023-11-23 21:40:55,636 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 9950, loss[loss=0.05244, simple_loss=0.07434, pruned_loss=0.007761, audio_tagging_loss=0.007504, over 14073.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09154, pruned_loss=0.01361, audio_tagging_loss=0.008829, over 3044600.64 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:40:56,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2551260.0, ans=0.125 2023-11-23 21:40:58,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2551260.0, ans=0.125 2023-11-23 21:41:02,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2551260.0, ans=0.0 2023-11-23 21:41:03,345 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.223e+01 9.080e+01 9.967e+01 1.331e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-23 21:41:04,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2551260.0, ans=0.125 2023-11-23 21:41:08,378 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382700 2023-11-23 21:41:33,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2551460.0, ans=0.0 2023-11-23 21:41:51,682 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.60 vs. limit=15.0 2023-11-23 21:41:59,252 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10000, loss[loss=0.07912, simple_loss=0.1172, pruned_loss=0.01532, audio_tagging_loss=0.005216, over 15577.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09237, pruned_loss=0.01369, audio_tagging_loss=0.008805, over 3045708.19 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:42:09,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382750 2023-11-23 21:42:14,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0 2023-11-23 21:42:26,608 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.86 vs. limit=6.0 2023-11-23 21:42:37,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2551793.3333333335, ans=0.1 2023-11-23 21:42:46,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2551793.3333333335, ans=0.125 2023-11-23 21:42:55,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2551860.0, ans=0.125 2023-11-23 21:43:00,970 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10050, loss[loss=0.0578, simple_loss=0.08191, pruned_loss=0.009422, audio_tagging_loss=0.00742, over 15272.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09203, pruned_loss=0.01366, audio_tagging_loss=0.008889, over 3037453.60 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:43:01,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2551926.6666666665, ans=0.125 2023-11-23 21:43:08,056 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.368e+01 8.922e+01 9.527e+01 1.170e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 21:43:11,946 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382800 2023-11-23 21:43:16,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=13.74 vs. limit=22.5 2023-11-23 21:43:20,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2551993.3333333335, ans=0.04949747468305833 2023-11-23 21:43:31,197 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.06 vs. limit=15.0 2023-11-23 21:43:46,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2552126.6666666665, ans=0.1 2023-11-23 21:43:49,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.03 vs. limit=10.0 2023-11-23 21:43:50,316 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.72 vs. limit=15.0 2023-11-23 21:43:53,523 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:44:02,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2552260.0, ans=0.125 2023-11-23 21:44:03,495 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10100, loss[loss=0.05875, simple_loss=0.08029, pruned_loss=0.01009, audio_tagging_loss=0.008521, over 14848.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09135, pruned_loss=0.01343, audio_tagging_loss=0.008971, over 3042585.45 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:44:14,777 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382850 2023-11-23 21:44:17,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2552326.6666666665, ans=0.0 2023-11-23 21:44:18,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2552326.6666666665, ans=0.2 2023-11-23 21:44:20,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2552326.6666666665, ans=0.2 2023-11-23 21:44:26,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2552326.6666666665, ans=0.025 2023-11-23 21:44:31,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2552393.3333333335, ans=0.125 2023-11-23 21:44:36,790 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.17 vs. limit=10.0 2023-11-23 21:44:42,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2552460.0, ans=0.025 2023-11-23 21:44:53,140 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 21:44:55,070 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=15.0 2023-11-23 21:45:06,044 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10150, loss[loss=0.06015, simple_loss=0.0789, pruned_loss=0.009452, audio_tagging_loss=0.01124, over 14997.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09204, pruned_loss=0.0136, audio_tagging_loss=0.009041, over 3049294.19 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:45:14,976 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.658e+01 9.213e+01 1.003e+02 1.366e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-23 21:45:17,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382900 2023-11-23 21:45:17,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2552660.0, ans=0.2 2023-11-23 21:45:20,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2552660.0, ans=0.2 2023-11-23 21:45:34,780 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 21:45:50,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2552793.3333333335, ans=0.125 2023-11-23 21:45:55,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2552860.0, ans=0.125 2023-11-23 21:46:08,458 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10200, loss[loss=0.0825, simple_loss=0.115, pruned_loss=0.01854, audio_tagging_loss=0.006484, over 15452.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09228, pruned_loss=0.01366, audio_tagging_loss=0.009056, over 3052911.44 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:46:19,162 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 382950 2023-11-23 21:46:19,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.31 vs. limit=10.0 2023-11-23 21:46:22,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.69 vs. limit=10.0 2023-11-23 21:46:30,766 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 21:46:37,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.40 vs. limit=15.0 2023-11-23 21:46:42,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2553060.0, ans=0.0 2023-11-23 21:46:59,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2553193.3333333335, ans=0.125 2023-11-23 21:47:07,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2553193.3333333335, ans=0.125 2023-11-23 21:47:10,204 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10250, loss[loss=0.06805, simple_loss=0.09471, pruned_loss=0.01406, audio_tagging_loss=0.006636, over 16387.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09164, pruned_loss=0.01348, audio_tagging_loss=0.009114, over 3052808.48 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:47:18,954 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.706e+01 9.282e+01 1.010e+02 1.139e+02, threshold=1.856e+02, percent-clipped=0.0 2023-11-23 21:47:21,395 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383000 2023-11-23 21:47:33,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2553326.6666666665, ans=0.0 2023-11-23 21:47:52,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2553460.0, ans=10.0 2023-11-23 21:48:00,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.02 vs. limit=22.5 2023-11-23 21:48:12,189 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10300, loss[loss=0.05162, simple_loss=0.05844, pruned_loss=0.01241, audio_tagging_loss=0.00999, over 15014.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09175, pruned_loss=0.01364, audio_tagging_loss=0.009169, over 3049581.58 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:48:14,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2553593.3333333335, ans=0.125 2023-11-23 21:48:19,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2553593.3333333335, ans=0.0 2023-11-23 21:48:23,507 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383050 2023-11-23 21:48:32,751 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.05 vs. limit=22.5 2023-11-23 21:48:33,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2553660.0, ans=0.5 2023-11-23 21:48:34,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2553660.0, ans=0.125 2023-11-23 21:48:52,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2553793.3333333335, ans=0.125 2023-11-23 21:48:55,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2553793.3333333335, ans=0.1 2023-11-23 21:48:58,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2553793.3333333335, ans=0.07 2023-11-23 21:49:06,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.94 vs. limit=15.0 2023-11-23 21:49:14,278 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10350, loss[loss=0.07175, simple_loss=0.09617, pruned_loss=0.01328, audio_tagging_loss=0.01039, over 15967.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09086, pruned_loss=0.01343, audio_tagging_loss=0.009315, over 3053325.11 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:49:23,131 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.433e+01 8.904e+01 9.438e+01 1.477e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-23 21:49:25,650 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383100 2023-11-23 21:50:03,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2554193.3333333335, ans=0.0 2023-11-23 21:50:09,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2554193.3333333335, ans=0.1 2023-11-23 21:50:11,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2554193.3333333335, ans=0.125 2023-11-23 21:50:16,706 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10400, loss[loss=0.06866, simple_loss=0.08722, pruned_loss=0.01454, audio_tagging_loss=0.01051, over 14521.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.0913, pruned_loss=0.01339, audio_tagging_loss=0.009394, over 3049605.92 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:50:24,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2554260.0, ans=0.0 2023-11-23 21:50:28,776 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383150 2023-11-23 21:50:30,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2554326.6666666665, ans=0.1 2023-11-23 21:50:30,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2554326.6666666665, ans=0.0 2023-11-23 21:50:31,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2554326.6666666665, ans=0.035 2023-11-23 21:50:53,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2554460.0, ans=0.125 2023-11-23 21:50:56,106 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.17 vs. limit=22.5 2023-11-23 21:51:11,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2554526.6666666665, ans=0.0 2023-11-23 21:51:14,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2554526.6666666665, ans=0.125 2023-11-23 21:51:19,720 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10450, loss[loss=0.06024, simple_loss=0.06651, pruned_loss=0.01349, audio_tagging_loss=0.0135, over 15162.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09201, pruned_loss=0.01348, audio_tagging_loss=0.009305, over 3049622.41 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:51:27,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2554593.3333333335, ans=0.0 2023-11-23 21:51:28,556 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.248e+01 8.875e+01 9.733e+01 1.236e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-23 21:51:28,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2554593.3333333335, ans=0.2 2023-11-23 21:51:31,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383200 2023-11-23 21:51:46,712 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:51:52,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2554726.6666666665, ans=0.125 2023-11-23 21:51:54,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2554726.6666666665, ans=0.09899494936611666 2023-11-23 21:52:03,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2554793.3333333335, ans=0.125 2023-11-23 21:52:08,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2554860.0, ans=0.125 2023-11-23 21:52:14,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2554860.0, ans=0.125 2023-11-23 21:52:22,446 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10500, loss[loss=0.06512, simple_loss=0.0874, pruned_loss=0.01485, audio_tagging_loss=0.006564, over 14978.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09128, pruned_loss=0.01339, audio_tagging_loss=0.009157, over 3044535.48 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:52:25,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2554926.6666666665, ans=0.95 2023-11-23 21:52:33,854 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383250 2023-11-23 21:52:34,148 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 21:52:35,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2554993.3333333335, ans=0.125 2023-11-23 21:52:49,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2555060.0, ans=0.07 2023-11-23 21:53:02,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2555126.6666666665, ans=0.125 2023-11-23 21:53:10,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2555126.6666666665, ans=0.125 2023-11-23 21:53:24,430 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10550, loss[loss=0.09207, simple_loss=0.1271, pruned_loss=0.02193, audio_tagging_loss=0.006602, over 15539.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09163, pruned_loss=0.01362, audio_tagging_loss=0.009078, over 3040247.19 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:53:34,351 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.236e+01 8.886e+01 9.649e+01 1.811e+02, threshold=1.777e+02, percent-clipped=1.0 2023-11-23 21:53:36,290 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383300 2023-11-23 21:53:53,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2555393.3333333335, ans=0.0 2023-11-23 21:54:03,615 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.79 vs. limit=15.0 2023-11-23 21:54:26,943 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10600, loss[loss=0.05417, simple_loss=0.07171, pruned_loss=0.01097, audio_tagging_loss=0.007341, over 13934.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09121, pruned_loss=0.01347, audio_tagging_loss=0.008998, over 3036564.34 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:54:36,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2555593.3333333335, ans=0.125 2023-11-23 21:54:38,497 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383350 2023-11-23 21:54:43,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2555660.0, ans=0.0 2023-11-23 21:54:47,598 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.17 vs. limit=15.0 2023-11-23 21:55:07,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2555793.3333333335, ans=0.1 2023-11-23 21:55:08,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2555793.3333333335, ans=0.05 2023-11-23 21:55:24,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2555860.0, ans=0.1 2023-11-23 21:55:25,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2555860.0, ans=0.5 2023-11-23 21:55:29,647 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10650, loss[loss=0.08222, simple_loss=0.1111, pruned_loss=0.01964, audio_tagging_loss=0.00703, over 14922.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09167, pruned_loss=0.01354, audio_tagging_loss=0.008872, over 3036688.42 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:55:38,962 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.595e+01 9.053e+01 9.824e+01 1.350e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-23 21:55:40,275 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383400 2023-11-23 21:56:09,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2556126.6666666665, ans=0.125 2023-11-23 21:56:13,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2556126.6666666665, ans=0.2 2023-11-23 21:56:31,440 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10700, loss[loss=0.06065, simple_loss=0.08537, pruned_loss=0.009805, audio_tagging_loss=0.008159, over 16135.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09126, pruned_loss=0.0136, audio_tagging_loss=0.008986, over 3035532.02 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:56:35,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2556260.0, ans=0.125 2023-11-23 21:56:43,596 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383450 2023-11-23 21:56:49,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2556326.6666666665, ans=0.2 2023-11-23 21:56:57,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2556393.3333333335, ans=0.125 2023-11-23 21:57:02,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2556393.3333333335, ans=0.1 2023-11-23 21:57:34,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.52 vs. limit=10.0 2023-11-23 21:57:35,203 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10750, loss[loss=0.07798, simple_loss=0.09823, pruned_loss=0.02031, audio_tagging_loss=0.008555, over 15005.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09106, pruned_loss=0.01357, audio_tagging_loss=0.008966, over 3037685.54 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:57:35,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2556593.3333333335, ans=0.04949747468305833 2023-11-23 21:57:39,357 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.76 vs. limit=22.5 2023-11-23 21:57:45,137 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.476e+01 9.127e+01 1.007e+02 1.402e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-23 21:57:45,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2556593.3333333335, ans=0.125 2023-11-23 21:57:46,507 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383500 2023-11-23 21:57:59,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2556726.6666666665, ans=0.125 2023-11-23 21:58:00,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.47 vs. limit=15.0 2023-11-23 21:58:02,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2556726.6666666665, ans=0.1 2023-11-23 21:58:19,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2556793.3333333335, ans=0.0 2023-11-23 21:58:36,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2556926.6666666665, ans=0.125 2023-11-23 21:58:37,452 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10800, loss[loss=0.05048, simple_loss=0.05411, pruned_loss=0.01234, audio_tagging_loss=0.01109, over 15580.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09028, pruned_loss=0.01344, audio_tagging_loss=0.008972, over 3038373.04 frames. ], batch size: 63, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 21:58:43,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2556926.6666666665, ans=15.0 2023-11-23 21:58:48,250 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383550 2023-11-23 21:58:51,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2556993.3333333335, ans=0.0 2023-11-23 21:58:58,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=2556993.3333333335, ans=15.0 2023-11-23 21:59:35,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2557193.3333333335, ans=0.0 2023-11-23 21:59:38,841 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10850, loss[loss=0.0577, simple_loss=0.06833, pruned_loss=0.01284, audio_tagging_loss=0.01069, over 14902.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09144, pruned_loss=0.01362, audio_tagging_loss=0.008951, over 3044083.90 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 21:59:49,892 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.543e+01 9.235e+01 1.001e+02 1.289e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-23 21:59:50,063 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383600 2023-11-23 21:59:50,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2557326.6666666665, ans=0.125 2023-11-23 22:00:09,200 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.44 vs. limit=6.0 2023-11-23 22:00:16,506 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.11 vs. limit=15.0 2023-11-23 22:00:36,672 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 22:00:41,469 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10900, loss[loss=0.06656, simple_loss=0.09283, pruned_loss=0.01152, audio_tagging_loss=0.008617, over 15035.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.0922, pruned_loss=0.01365, audio_tagging_loss=0.008948, over 3047099.36 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:00:47,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2557593.3333333335, ans=0.125 2023-11-23 22:00:53,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383650 2023-11-23 22:00:56,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2557660.0, ans=0.125 2023-11-23 22:01:01,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2557660.0, ans=0.07 2023-11-23 22:01:09,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2557726.6666666665, ans=0.0 2023-11-23 22:01:12,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2557726.6666666665, ans=0.125 2023-11-23 22:01:15,558 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.84 vs. limit=15.0 2023-11-23 22:01:44,019 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 10950, loss[loss=0.05821, simple_loss=0.07945, pruned_loss=0.01041, audio_tagging_loss=0.008071, over 15173.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09234, pruned_loss=0.01377, audio_tagging_loss=0.008924, over 3041252.02 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:01:45,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2557926.6666666665, ans=0.2 2023-11-23 22:01:54,643 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.274e+01 8.562e+01 9.166e+01 9.897e+01 1.361e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-23 22:01:54,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383700 2023-11-23 22:02:13,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2558060.0, ans=0.2 2023-11-23 22:02:21,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2558126.6666666665, ans=0.0 2023-11-23 22:02:23,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2558126.6666666665, ans=0.2 2023-11-23 22:02:32,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2558193.3333333335, ans=0.0 2023-11-23 22:02:39,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2558193.3333333335, ans=0.09899494936611666 2023-11-23 22:02:45,435 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11000, loss[loss=0.07494, simple_loss=0.09962, pruned_loss=0.01615, audio_tagging_loss=0.008986, over 14823.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09166, pruned_loss=0.01354, audio_tagging_loss=0.009049, over 3036902.79 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:02:50,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2558260.0, ans=0.125 2023-11-23 22:02:53,624 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 22:02:56,601 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383750 2023-11-23 22:03:26,838 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.76 vs. limit=15.0 2023-11-23 22:03:27,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.45 vs. limit=15.0 2023-11-23 22:03:37,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2558526.6666666665, ans=0.125 2023-11-23 22:03:40,853 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 22:03:46,999 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11050, loss[loss=0.07032, simple_loss=0.09072, pruned_loss=0.01694, audio_tagging_loss=0.00802, over 14919.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09177, pruned_loss=0.01363, audio_tagging_loss=0.009108, over 3041444.55 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:03:49,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.85 vs. limit=12.0 2023-11-23 22:03:58,693 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.213e+01 8.515e+01 9.099e+01 9.905e+01 1.187e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-23 22:03:58,857 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383800 2023-11-23 22:04:21,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2558726.6666666665, ans=0.2 2023-11-23 22:04:28,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2558793.3333333335, ans=0.04949747468305833 2023-11-23 22:04:50,544 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11100, loss[loss=0.07475, simple_loss=0.09832, pruned_loss=0.0137, audio_tagging_loss=0.01189, over 14305.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09173, pruned_loss=0.01359, audio_tagging_loss=0.009288, over 3043370.80 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:04:52,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.45 vs. limit=22.5 2023-11-23 22:04:55,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2558926.6666666665, ans=0.2 2023-11-23 22:05:01,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383850 2023-11-23 22:05:14,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2559060.0, ans=0.2 2023-11-23 22:05:24,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2559060.0, ans=0.125 2023-11-23 22:05:24,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2559060.0, ans=0.2 2023-11-23 22:05:44,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2559193.3333333335, ans=0.0 2023-11-23 22:05:46,694 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.13 vs. limit=6.0 2023-11-23 22:05:51,957 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11150, loss[loss=0.08191, simple_loss=0.1072, pruned_loss=0.01801, audio_tagging_loss=0.0103, over 13602.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09118, pruned_loss=0.01346, audio_tagging_loss=0.009375, over 3036405.83 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:06:00,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2559260.0, ans=0.0 2023-11-23 22:06:02,625 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.354e+01 8.926e+01 9.504e+01 1.198e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 22:06:02,790 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383900 2023-11-23 22:06:05,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2559326.6666666665, ans=0.0 2023-11-23 22:06:33,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2559460.0, ans=0.09899494936611666 2023-11-23 22:06:36,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2559460.0, ans=0.0 2023-11-23 22:06:39,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.12 vs. limit=15.0 2023-11-23 22:06:42,864 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.42 vs. limit=15.0 2023-11-23 22:06:53,539 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11200, loss[loss=0.05531, simple_loss=0.06447, pruned_loss=0.01214, audio_tagging_loss=0.01093, over 15545.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.091, pruned_loss=0.0136, audio_tagging_loss=0.009484, over 3040327.22 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 22:07:03,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0 2023-11-23 22:07:04,940 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 383950 2023-11-23 22:07:06,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2559660.0, ans=0.125 2023-11-23 22:07:08,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2559660.0, ans=0.125 2023-11-23 22:07:11,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2559660.0, ans=0.125 2023-11-23 22:07:21,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2559726.6666666665, ans=0.125 2023-11-23 22:07:25,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2559726.6666666665, ans=0.0 2023-11-23 22:07:52,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2559860.0, ans=0.125 2023-11-23 22:07:55,767 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11250, loss[loss=0.06823, simple_loss=0.08967, pruned_loss=0.01539, audio_tagging_loss=0.008004, over 12986.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09138, pruned_loss=0.0138, audio_tagging_loss=0.009351, over 3040270.17 frames. ], batch size: 50, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 22:08:07,491 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.428e+01 9.033e+01 9.633e+01 1.168e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-23 22:08:07,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384000 2023-11-23 22:08:08,452 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.46 vs. limit=22.5 2023-11-23 22:08:14,815 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0 2023-11-23 22:08:16,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2559993.3333333335, ans=0.0 2023-11-23 22:08:20,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2559993.3333333335, ans=0.0 2023-11-23 22:08:55,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2560193.3333333335, ans=0.1 2023-11-23 22:09:02,412 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11300, loss[loss=0.07134, simple_loss=0.0922, pruned_loss=0.01664, audio_tagging_loss=0.008598, over 14413.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09139, pruned_loss=0.0137, audio_tagging_loss=0.009176, over 3046008.12 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 22:09:02,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=15.0 2023-11-23 22:09:03,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2560260.0, ans=0.125 2023-11-23 22:09:13,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384050 2023-11-23 22:09:58,618 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.95 vs. limit=22.5 2023-11-23 22:10:00,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2560526.6666666665, ans=0.0 2023-11-23 22:10:04,325 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11350, loss[loss=0.06596, simple_loss=0.08905, pruned_loss=0.01422, audio_tagging_loss=0.00721, over 14748.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09088, pruned_loss=0.01358, audio_tagging_loss=0.00907, over 3045246.56 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 22:10:16,254 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.206e+01 9.003e+01 9.537e+01 1.148e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-23 22:10:16,413 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384100 2023-11-23 22:10:22,712 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 22:10:37,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2560726.6666666665, ans=0.0 2023-11-23 22:10:50,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2560793.3333333335, ans=10.0 2023-11-23 22:10:51,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.52 vs. limit=15.0 2023-11-23 22:10:53,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2560860.0, ans=0.125 2023-11-23 22:11:07,781 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11400, loss[loss=0.08541, simple_loss=0.1137, pruned_loss=0.02096, audio_tagging_loss=0.007576, over 15207.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09072, pruned_loss=0.01349, audio_tagging_loss=0.009017, over 3044583.00 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:11:10,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2560926.6666666665, ans=0.2 2023-11-23 22:11:18,992 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384150 2023-11-23 22:11:27,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2560993.3333333335, ans=0.95 2023-11-23 22:11:37,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2561060.0, ans=0.125 2023-11-23 22:11:46,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2561126.6666666665, ans=0.2 2023-11-23 22:11:56,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2561193.3333333335, ans=0.2 2023-11-23 22:11:59,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2561193.3333333335, ans=0.0 2023-11-23 22:12:10,359 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11450, loss[loss=0.02751, simple_loss=0.0312, pruned_loss=0.001106, audio_tagging_loss=0.01081, over 16985.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.0915, pruned_loss=0.01358, audio_tagging_loss=0.009004, over 3052063.53 frames. ], batch size: 68, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:12:13,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2561260.0, ans=0.025 2023-11-23 22:12:21,197 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384200 2023-11-23 22:12:22,241 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.398e+01 8.984e+01 9.548e+01 1.185e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-23 22:12:30,032 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0 2023-11-23 22:12:41,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2561393.3333333335, ans=0.0 2023-11-23 22:12:41,825 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.80 vs. limit=15.0 2023-11-23 22:13:04,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2561526.6666666665, ans=0.0 2023-11-23 22:13:09,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2561526.6666666665, ans=0.0 2023-11-23 22:13:11,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.17 vs. limit=15.0 2023-11-23 22:13:11,994 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11500, loss[loss=0.04455, simple_loss=0.06022, pruned_loss=0.005335, audio_tagging_loss=0.009105, over 14008.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09082, pruned_loss=0.01341, audio_tagging_loss=0.008929, over 3044653.43 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:13:14,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2561593.3333333335, ans=0.1 2023-11-23 22:13:23,215 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384250 2023-11-23 22:13:33,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2561660.0, ans=0.0 2023-11-23 22:14:14,427 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11550, loss[loss=0.07126, simple_loss=0.1004, pruned_loss=0.01257, audio_tagging_loss=0.008497, over 14590.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09121, pruned_loss=0.01352, audio_tagging_loss=0.008876, over 3042363.53 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0 2023-11-23 22:14:24,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2561926.6666666665, ans=0.1 2023-11-23 22:14:25,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384300 2023-11-23 22:14:25,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2561993.3333333335, ans=0.125 2023-11-23 22:14:26,805 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.220e+01 8.896e+01 9.591e+01 1.198e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-23 22:14:27,609 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.18 vs. limit=12.0 2023-11-23 22:14:30,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2561993.3333333335, ans=0.0 2023-11-23 22:14:30,903 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.29 vs. limit=12.0 2023-11-23 22:14:33,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2561993.3333333335, ans=0.2 2023-11-23 22:14:37,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.22 vs. limit=15.0 2023-11-23 22:14:44,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2562060.0, ans=0.07 2023-11-23 22:14:50,988 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 22:14:52,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2562126.6666666665, ans=0.125 2023-11-23 22:14:59,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2562126.6666666665, ans=0.125 2023-11-23 22:15:03,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2562193.3333333335, ans=10.0 2023-11-23 22:15:16,229 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11600, loss[loss=0.07388, simple_loss=0.09949, pruned_loss=0.01498, audio_tagging_loss=0.009153, over 14131.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09149, pruned_loss=0.01355, audio_tagging_loss=0.008831, over 3041479.41 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 22:15:27,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384350 2023-11-23 22:15:27,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2562326.6666666665, ans=0.125 2023-11-23 22:15:30,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2562326.6666666665, ans=0.025 2023-11-23 22:15:44,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.00 vs. limit=15.0 2023-11-23 22:15:54,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2562460.0, ans=0.125 2023-11-23 22:16:12,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2562526.6666666665, ans=0.0 2023-11-23 22:16:17,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2562593.3333333335, ans=0.95 2023-11-23 22:16:18,791 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11650, loss[loss=0.04668, simple_loss=0.0547, pruned_loss=0.008022, audio_tagging_loss=0.01131, over 14741.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09139, pruned_loss=0.01359, audio_tagging_loss=0.008873, over 3040571.69 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 22:16:30,025 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384400 2023-11-23 22:16:31,042 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.399e+01 8.387e+01 8.924e+01 9.715e+01 1.157e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-23 22:16:34,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.56 vs. limit=15.0 2023-11-23 22:17:22,177 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11700, loss[loss=0.06643, simple_loss=0.09451, pruned_loss=0.01195, audio_tagging_loss=0.007228, over 15907.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09125, pruned_loss=0.01339, audio_tagging_loss=0.008951, over 3050771.11 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 32.0 2023-11-23 22:17:23,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2562926.6666666665, ans=0.1 2023-11-23 22:17:33,543 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384450 2023-11-23 22:17:33,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2562993.3333333335, ans=0.125 2023-11-23 22:17:33,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2562993.3333333335, ans=0.0 2023-11-23 22:17:45,903 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.70 vs. limit=15.0 2023-11-23 22:18:05,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2563126.6666666665, ans=0.125 2023-11-23 22:18:12,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.94 vs. limit=15.0 2023-11-23 22:18:19,445 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.73 vs. limit=22.5 2023-11-23 22:18:24,599 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11750, loss[loss=0.05605, simple_loss=0.07334, pruned_loss=0.009506, audio_tagging_loss=0.009872, over 14282.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09238, pruned_loss=0.01363, audio_tagging_loss=0.008866, over 3052916.02 frames. ], batch size: 55, lr: 2.10e-03, grad_scale: 32.0 2023-11-23 22:18:27,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2563260.0, ans=0.0 2023-11-23 22:18:32,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2563260.0, ans=0.125 2023-11-23 22:18:35,395 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384500 2023-11-23 22:18:36,427 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.506e+01 8.915e+01 9.648e+01 1.548e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-23 22:18:39,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2563326.6666666665, ans=0.125 2023-11-23 22:19:05,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2563460.0, ans=0.125 2023-11-23 22:19:26,095 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11800, loss[loss=0.05437, simple_loss=0.07492, pruned_loss=0.00595, audio_tagging_loss=0.01096, over 15419.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09165, pruned_loss=0.01344, audio_tagging_loss=0.008934, over 3048645.68 frames. ], batch size: 61, lr: 2.10e-03, grad_scale: 32.0 2023-11-23 22:19:30,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2563593.3333333335, ans=0.125 2023-11-23 22:19:35,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2563593.3333333335, ans=15.0 2023-11-23 22:19:37,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384550 2023-11-23 22:19:41,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.52 vs. limit=15.0 2023-11-23 22:19:49,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2563660.0, ans=0.125 2023-11-23 22:20:08,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2563793.3333333335, ans=0.05 2023-11-23 22:20:11,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.02 vs. limit=15.0 2023-11-23 22:20:14,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2563860.0, ans=0.0 2023-11-23 22:20:26,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2563860.0, ans=0.125 2023-11-23 22:20:28,399 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11850, loss[loss=0.07063, simple_loss=0.09261, pruned_loss=0.0154, audio_tagging_loss=0.008929, over 15327.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.0924, pruned_loss=0.01367, audio_tagging_loss=0.009041, over 3049343.16 frames. ], batch size: 56, lr: 2.10e-03, grad_scale: 32.0 2023-11-23 22:20:35,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2563926.6666666665, ans=0.0 2023-11-23 22:20:35,758 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.83 vs. limit=15.0 2023-11-23 22:20:39,736 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384600 2023-11-23 22:20:41,328 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.384e+01 8.994e+01 9.780e+01 1.224e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 22:20:48,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2563993.3333333335, ans=0.125 2023-11-23 22:20:51,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.98 vs. limit=15.0 2023-11-23 22:20:57,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2564060.0, ans=0.125 2023-11-23 22:21:12,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.40 vs. limit=15.0 2023-11-23 22:21:23,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2564193.3333333335, ans=0.0 2023-11-23 22:21:31,113 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11900, loss[loss=0.06753, simple_loss=0.09719, pruned_loss=0.01057, audio_tagging_loss=0.008367, over 14893.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09175, pruned_loss=0.01365, audio_tagging_loss=0.009172, over 3046934.02 frames. ], batch size: 55, lr: 2.10e-03, grad_scale: 16.0 2023-11-23 22:21:41,862 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384650 2023-11-23 22:21:41,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2564326.6666666665, ans=0.0 2023-11-23 22:21:45,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2564326.6666666665, ans=0.125 2023-11-23 22:21:46,142 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.02 vs. limit=12.0 2023-11-23 22:21:57,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.52 vs. limit=15.0 2023-11-23 22:22:32,789 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 11950, loss[loss=0.06166, simple_loss=0.08521, pruned_loss=0.01025, audio_tagging_loss=0.008811, over 15203.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09149, pruned_loss=0.01348, audio_tagging_loss=0.009171, over 3047839.78 frames. ], batch size: 58, lr: 2.10e-03, grad_scale: 16.0 2023-11-23 22:22:37,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.84 vs. limit=22.5 2023-11-23 22:22:41,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2564593.3333333335, ans=0.125 2023-11-23 22:22:43,274 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384700 2023-11-23 22:22:46,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.381e+01 9.044e+01 9.786e+01 1.119e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-23 22:22:55,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2564660.0, ans=0.125 2023-11-23 22:23:13,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2564793.3333333335, ans=0.1 2023-11-23 22:23:24,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2564860.0, ans=0.05 2023-11-23 22:23:32,158 INFO [train_asr.py:1221] (3/4) Epoch 32, batch 12000, loss[loss=0.07674, simple_loss=0.1109, pruned_loss=0.0128, audio_tagging_loss=0.008472, over 16710.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09229, pruned_loss=0.01358, audio_tagging_loss=0.009289, over 3047496.71 frames. ], batch size: 60, lr: 2.10e-03, grad_scale: 32.0 2023-11-23 22:23:32,159 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 22:23:54,375 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([5.1627, 4.4160, 4.4318, 4.4593], device='cuda:3') 2023-11-23 22:24:14,279 INFO [train_asr.py:1253] (3/4) Epoch 32, validation: loss=0.05848, simple_loss=0.0511, pruned_loss=0.005239, audio_tagging_loss=0.02769, over 4681554.00 frames. 2023-11-23 22:24:14,279 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 22:24:24,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384750 2023-11-23 22:24:26,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2564993.3333333335, ans=0.125 2023-11-23 22:24:31,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2564993.3333333335, ans=0.1 2023-11-23 22:24:34,262 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.28 vs. limit=15.0 2023-11-23 22:25:13,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565086.6666666665, ans=0.1 2023-11-23 22:25:15,141 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 0, loss[loss=0.07768, simple_loss=0.0947, pruned_loss=0.01111, audio_tagging_loss=0.01922, over 16118.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.0947, pruned_loss=0.01111, audio_tagging_loss=0.01922, over 16118.00 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:25:15,141 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 22:25:50,706 INFO [train_asr.py:1253] (3/4) Epoch 33, validation: loss=0.05781, simple_loss=0.05104, pruned_loss=0.005203, audio_tagging_loss=0.02709, over 4681554.00 frames. 2023-11-23 22:25:50,707 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 22:25:58,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.97 vs. limit=15.0 2023-11-23 22:26:14,789 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.02 vs. limit=15.0 2023-11-23 22:26:26,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2565220.0, ans=0.125 2023-11-23 22:26:34,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384800 2023-11-23 22:26:35,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2565286.6666666665, ans=0.2 2023-11-23 22:26:38,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.158e+01 8.944e+01 9.729e+01 1.034e+02 1.354e+02, threshold=1.946e+02, percent-clipped=0.0 2023-11-23 22:26:38,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2565286.6666666665, ans=0.125 2023-11-23 22:26:39,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2565353.3333333335, ans=0.2 2023-11-23 22:26:44,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2565353.3333333335, ans=0.1 2023-11-23 22:26:52,493 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 50, loss[loss=0.07167, simple_loss=0.08611, pruned_loss=0.01314, audio_tagging_loss=0.01547, over 16725.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09317, pruned_loss=0.01356, audio_tagging_loss=0.01735, over 692151.45 frames. ], batch size: 66, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:27:36,574 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384850 2023-11-23 22:27:44,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2565686.6666666665, ans=0.05 2023-11-23 22:27:49,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2565686.6666666665, ans=0.125 2023-11-23 22:27:52,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2565686.6666666665, ans=0.1 2023-11-23 22:27:56,908 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 100, loss[loss=0.06707, simple_loss=0.08612, pruned_loss=0.0104, audio_tagging_loss=0.01361, over 14964.00 frames. ], tot_loss[loss=0.07716, simple_loss=0.09341, pruned_loss=0.01367, audio_tagging_loss=0.01679, over 1210373.07 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:28:10,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2565820.0, ans=0.09899494936611666 2023-11-23 22:28:20,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2565886.6666666665, ans=0.0 2023-11-23 22:28:38,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384900 2023-11-23 22:28:43,526 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.457e+01 8.971e+01 9.700e+01 1.041e+02 1.375e+02, threshold=1.940e+02, percent-clipped=0.0 2023-11-23 22:28:50,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2566020.0, ans=0.0 2023-11-23 22:28:55,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2566020.0, ans=0.0 2023-11-23 22:28:57,719 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 150, loss[loss=0.08114, simple_loss=0.1144, pruned_loss=0.01348, audio_tagging_loss=0.01047, over 15425.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09292, pruned_loss=0.01366, audio_tagging_loss=0.0149, over 1620462.40 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:29:00,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2566086.6666666665, ans=0.1 2023-11-23 22:29:15,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2566153.3333333335, ans=0.125 2023-11-23 22:29:41,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 384950 2023-11-23 22:29:52,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2566353.3333333335, ans=0.125 2023-11-23 22:29:58,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2566420.0, ans=0.1 2023-11-23 22:29:59,072 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 200, loss[loss=0.09252, simple_loss=0.1322, pruned_loss=0.01895, audio_tagging_loss=0.007498, over 15769.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.092, pruned_loss=0.01356, audio_tagging_loss=0.01328, over 1942026.64 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:30:22,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2566486.6666666665, ans=0.0 2023-11-23 22:30:27,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2566553.3333333335, ans=0.0 2023-11-23 22:30:42,352 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385000 2023-11-23 22:30:46,068 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.668e+01 9.124e+01 1.011e+02 1.346e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-23 22:30:47,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2566686.6666666665, ans=0.0 2023-11-23 22:31:01,365 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 250, loss[loss=0.07347, simple_loss=0.09422, pruned_loss=0.01653, audio_tagging_loss=0.009834, over 15975.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09149, pruned_loss=0.01351, audio_tagging_loss=0.0121, over 2185459.24 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:31:02,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2566753.3333333335, ans=0.035 2023-11-23 22:31:08,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2566753.3333333335, ans=0.125 2023-11-23 22:31:12,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2566753.3333333335, ans=0.125 2023-11-23 22:31:21,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2566820.0, ans=0.0 2023-11-23 22:31:44,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385050 2023-11-23 22:31:45,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2566953.3333333335, ans=0.0 2023-11-23 22:31:51,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.63 vs. limit=22.5 2023-11-23 22:31:54,854 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.71 vs. limit=15.0 2023-11-23 22:31:56,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2567020.0, ans=0.1 2023-11-23 22:31:58,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2567020.0, ans=0.125 2023-11-23 22:31:58,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2567020.0, ans=0.0 2023-11-23 22:32:04,242 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 300, loss[loss=0.07636, simple_loss=0.1092, pruned_loss=0.01386, audio_tagging_loss=0.007909, over 14654.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09237, pruned_loss=0.01386, audio_tagging_loss=0.01121, over 2377228.23 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:32:04,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=2567086.6666666665, ans=15.0 2023-11-23 22:32:11,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2567086.6666666665, ans=0.1 2023-11-23 22:32:18,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2567153.3333333335, ans=0.125 2023-11-23 22:32:47,982 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385100 2023-11-23 22:32:51,319 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.686e+01 8.804e+01 9.384e+01 1.010e+02 1.261e+02, threshold=1.877e+02, percent-clipped=0.0 2023-11-23 22:33:01,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2567353.3333333335, ans=0.1 2023-11-23 22:33:05,398 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 350, loss[loss=0.06485, simple_loss=0.08301, pruned_loss=0.01379, audio_tagging_loss=0.009554, over 14848.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09135, pruned_loss=0.01352, audio_tagging_loss=0.01065, over 2528519.80 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:33:17,741 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.82 vs. limit=15.0 2023-11-23 22:33:19,041 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.38 vs. limit=6.0 2023-11-23 22:33:30,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2567553.3333333335, ans=0.2 2023-11-23 22:33:36,099 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.19 vs. limit=10.0 2023-11-23 22:33:40,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2567553.3333333335, ans=0.125 2023-11-23 22:33:43,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2567620.0, ans=0.125 2023-11-23 22:33:49,637 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385150 2023-11-23 22:33:56,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2567686.6666666665, ans=0.125 2023-11-23 22:34:05,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2567686.6666666665, ans=0.0 2023-11-23 22:34:07,564 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 400, loss[loss=0.06616, simple_loss=0.08839, pruned_loss=0.01276, audio_tagging_loss=0.009201, over 16736.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09113, pruned_loss=0.01343, audio_tagging_loss=0.01019, over 2645639.17 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:34:09,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.34 vs. limit=15.0 2023-11-23 22:34:17,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2567753.3333333335, ans=0.1 2023-11-23 22:34:37,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2567886.6666666665, ans=0.1 2023-11-23 22:34:49,127 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.04 vs. limit=15.0 2023-11-23 22:34:49,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2567953.3333333335, ans=0.125 2023-11-23 22:34:51,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385200 2023-11-23 22:34:54,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2567953.3333333335, ans=0.07 2023-11-23 22:34:54,927 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.173e+01 8.325e+01 8.873e+01 9.611e+01 1.385e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-23 22:34:56,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2568020.0, ans=0.0 2023-11-23 22:35:04,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2568020.0, ans=0.1 2023-11-23 22:35:11,086 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 450, loss[loss=0.06142, simple_loss=0.08218, pruned_loss=0.01215, audio_tagging_loss=0.008178, over 16675.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09157, pruned_loss=0.01371, audio_tagging_loss=0.009927, over 2733586.68 frames. ], batch size: 63, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:35:26,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.81 vs. limit=15.0 2023-11-23 22:35:44,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2568220.0, ans=0.2 2023-11-23 22:35:54,326 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385250 2023-11-23 22:35:56,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2568286.6666666665, ans=0.125 2023-11-23 22:35:59,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2568353.3333333335, ans=0.0 2023-11-23 22:36:00,361 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.21 vs. limit=22.5 2023-11-23 22:36:03,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2568353.3333333335, ans=0.0 2023-11-23 22:36:05,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2568353.3333333335, ans=0.1 2023-11-23 22:36:12,791 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 500, loss[loss=0.07495, simple_loss=0.1019, pruned_loss=0.01295, audio_tagging_loss=0.01106, over 15329.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09138, pruned_loss=0.01362, audio_tagging_loss=0.009698, over 2804175.19 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:36:41,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2568553.3333333335, ans=0.125 2023-11-23 22:36:57,245 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385300 2023-11-23 22:36:59,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.18 vs. limit=15.0 2023-11-23 22:37:00,579 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.320e+01 8.409e+01 9.142e+01 9.799e+01 1.482e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-23 22:37:15,579 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 550, loss[loss=0.06711, simple_loss=0.08703, pruned_loss=0.01404, audio_tagging_loss=0.009557, over 14158.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09205, pruned_loss=0.01373, audio_tagging_loss=0.009535, over 2865173.08 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:37:25,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2568753.3333333335, ans=0.125 2023-11-23 22:37:37,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2568820.0, ans=0.125 2023-11-23 22:37:58,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385350 2023-11-23 22:38:05,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2569020.0, ans=0.125 2023-11-23 22:38:07,379 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.54 vs. limit=22.5 2023-11-23 22:38:10,724 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.18 vs. limit=8.0 2023-11-23 22:38:11,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2569020.0, ans=0.2 2023-11-23 22:38:17,976 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 600, loss[loss=0.05756, simple_loss=0.07654, pruned_loss=0.01076, audio_tagging_loss=0.008522, over 15633.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09218, pruned_loss=0.01371, audio_tagging_loss=0.009463, over 2910544.82 frames. ], batch size: 63, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:38:28,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2569086.6666666665, ans=0.2 2023-11-23 22:38:35,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2569153.3333333335, ans=0.0 2023-11-23 22:38:45,351 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.49 vs. limit=15.0 2023-11-23 22:38:54,700 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.78 vs. limit=10.0 2023-11-23 22:38:58,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2569286.6666666665, ans=0.125 2023-11-23 22:39:01,702 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385400 2023-11-23 22:39:04,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2569286.6666666665, ans=0.2 2023-11-23 22:39:06,580 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.472e+01 9.172e+01 9.611e+01 1.258e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-23 22:39:13,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2569353.3333333335, ans=0.125 2023-11-23 22:39:20,481 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 650, loss[loss=0.05374, simple_loss=0.069, pruned_loss=0.005078, audio_tagging_loss=0.01416, over 16008.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09153, pruned_loss=0.01373, audio_tagging_loss=0.009472, over 2938623.76 frames. ], batch size: 61, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:39:26,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2569420.0, ans=0.0 2023-11-23 22:39:32,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.16 vs. limit=15.0 2023-11-23 22:39:53,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2569553.3333333335, ans=0.125 2023-11-23 22:39:54,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2569553.3333333335, ans=0.125 2023-11-23 22:39:54,904 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.55 vs. limit=15.0 2023-11-23 22:39:55,092 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.93 vs. limit=15.0 2023-11-23 22:40:04,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385450 2023-11-23 22:40:15,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2569686.6666666665, ans=0.125 2023-11-23 22:40:20,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2569686.6666666665, ans=0.125 2023-11-23 22:40:22,210 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 700, loss[loss=0.06668, simple_loss=0.09199, pruned_loss=0.01261, audio_tagging_loss=0.00808, over 15690.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09292, pruned_loss=0.01386, audio_tagging_loss=0.009407, over 2963116.41 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:40:29,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2569753.3333333335, ans=0.125 2023-11-23 22:40:30,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.78 vs. limit=8.0 2023-11-23 22:40:33,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2569753.3333333335, ans=0.125 2023-11-23 22:40:48,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2569886.6666666665, ans=0.0 2023-11-23 22:41:06,028 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385500 2023-11-23 22:41:11,043 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.418e+01 8.479e+01 9.169e+01 9.632e+01 1.119e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-23 22:41:22,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2570020.0, ans=0.125 2023-11-23 22:41:25,256 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 750, loss[loss=0.05139, simple_loss=0.07397, pruned_loss=0.007276, audio_tagging_loss=0.007128, over 15950.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09269, pruned_loss=0.0138, audio_tagging_loss=0.009302, over 2987542.11 frames. ], batch size: 60, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:41:36,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2570153.3333333335, ans=10.0 2023-11-23 22:41:40,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2570153.3333333335, ans=0.125 2023-11-23 22:42:00,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2570220.0, ans=0.125 2023-11-23 22:42:09,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385550 2023-11-23 22:42:12,639 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.75 vs. limit=10.0 2023-11-23 22:42:25,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2570353.3333333335, ans=0.0 2023-11-23 22:42:27,206 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 800, loss[loss=0.09137, simple_loss=0.1322, pruned_loss=0.01659, audio_tagging_loss=0.008685, over 15210.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09287, pruned_loss=0.01388, audio_tagging_loss=0.009368, over 3000358.81 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:42:43,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2570486.6666666665, ans=0.0 2023-11-23 22:42:58,547 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.77 vs. limit=15.0 2023-11-23 22:43:01,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2570553.3333333335, ans=0.125 2023-11-23 22:43:11,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385600 2023-11-23 22:43:16,321 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.424e+01 8.735e+01 9.399e+01 1.001e+02 1.314e+02, threshold=1.880e+02, percent-clipped=0.0 2023-11-23 22:43:25,960 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.46 vs. limit=15.0 2023-11-23 22:43:30,073 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 850, loss[loss=0.05811, simple_loss=0.07849, pruned_loss=0.008251, audio_tagging_loss=0.01061, over 15644.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09228, pruned_loss=0.01376, audio_tagging_loss=0.00935, over 3012702.52 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:43:44,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2570820.0, ans=0.2 2023-11-23 22:43:48,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2570820.0, ans=0.2 2023-11-23 22:43:53,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2570820.0, ans=0.1 2023-11-23 22:43:54,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2570886.6666666665, ans=0.2 2023-11-23 22:43:58,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2570886.6666666665, ans=0.125 2023-11-23 22:44:14,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385650 2023-11-23 22:44:16,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2570953.3333333335, ans=0.125 2023-11-23 22:44:16,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2570953.3333333335, ans=0.125 2023-11-23 22:44:23,548 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-23 22:44:30,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2571020.0, ans=0.125 2023-11-23 22:44:33,793 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 900, loss[loss=0.07143, simple_loss=0.09546, pruned_loss=0.01418, audio_tagging_loss=0.009517, over 14656.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09275, pruned_loss=0.01385, audio_tagging_loss=0.009338, over 3032819.99 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:44:37,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2571086.6666666665, ans=0.125 2023-11-23 22:44:38,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.15 vs. limit=22.5 2023-11-23 22:44:44,267 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.26 vs. limit=10.0 2023-11-23 22:44:45,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2571153.3333333335, ans=0.0 2023-11-23 22:44:57,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2571220.0, ans=0.0 2023-11-23 22:45:01,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.73 vs. limit=15.0 2023-11-23 22:45:17,169 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385700 2023-11-23 22:45:22,414 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.319e+01 9.000e+01 1.000e+02 1.242e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-23 22:45:35,459 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 950, loss[loss=0.04914, simple_loss=0.06321, pruned_loss=0.006063, audio_tagging_loss=0.01147, over 15067.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09204, pruned_loss=0.01353, audio_tagging_loss=0.009259, over 3032757.12 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:45:54,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2571486.6666666665, ans=0.0 2023-11-23 22:45:57,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.87 vs. limit=15.0 2023-11-23 22:46:19,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385750 2023-11-23 22:46:26,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2571686.6666666665, ans=0.07 2023-11-23 22:46:35,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.74 vs. limit=22.5 2023-11-23 22:46:36,892 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.70 vs. limit=15.0 2023-11-23 22:46:37,423 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1000, loss[loss=0.05748, simple_loss=0.0675, pruned_loss=0.01363, audio_tagging_loss=0.01011, over 14848.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09214, pruned_loss=0.01356, audio_tagging_loss=0.00905, over 3032810.14 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:46:38,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2571753.3333333335, ans=0.125 2023-11-23 22:46:41,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=13.30 vs. limit=15.0 2023-11-23 22:46:42,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2571753.3333333335, ans=0.125 2023-11-23 22:46:54,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2571820.0, ans=0.05 2023-11-23 22:46:57,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2571820.0, ans=0.1 2023-11-23 22:47:04,031 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 22:47:20,805 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385800 2023-11-23 22:47:27,681 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.356e+01 8.922e+01 9.562e+01 1.199e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 22:47:29,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2572020.0, ans=0.2 2023-11-23 22:47:40,647 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1050, loss[loss=0.07701, simple_loss=0.1029, pruned_loss=0.01681, audio_tagging_loss=0.008745, over 16330.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09228, pruned_loss=0.01357, audio_tagging_loss=0.008929, over 3042083.20 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:48:24,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385850 2023-11-23 22:48:30,901 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.44 vs. limit=15.0 2023-11-23 22:48:40,202 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0 2023-11-23 22:48:41,257 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.57 vs. limit=12.0 2023-11-23 22:48:41,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.63 vs. limit=15.0 2023-11-23 22:48:43,146 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1100, loss[loss=0.08474, simple_loss=0.1186, pruned_loss=0.01819, audio_tagging_loss=0.00726, over 15314.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09162, pruned_loss=0.01353, audio_tagging_loss=0.008921, over 3048477.52 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:48:44,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2572420.0, ans=0.2 2023-11-23 22:48:45,583 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 22:48:48,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2572420.0, ans=0.125 2023-11-23 22:49:02,299 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.91 vs. limit=6.0 2023-11-23 22:49:04,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2572486.6666666665, ans=0.125 2023-11-23 22:49:07,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2572553.3333333335, ans=0.0 2023-11-23 22:49:07,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2572553.3333333335, ans=0.0 2023-11-23 22:49:27,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385900 2023-11-23 22:49:33,459 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.297e+01 8.837e+01 9.387e+01 1.175e+02, threshold=1.767e+02, percent-clipped=0.0 2023-11-23 22:49:33,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2572686.6666666665, ans=0.125 2023-11-23 22:49:45,370 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1150, loss[loss=0.05759, simple_loss=0.07647, pruned_loss=0.01016, audio_tagging_loss=0.009191, over 14612.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09208, pruned_loss=0.01369, audio_tagging_loss=0.008927, over 3049275.56 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:49:49,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2572753.3333333335, ans=0.125 2023-11-23 22:49:55,577 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.90 vs. limit=15.0 2023-11-23 22:49:59,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2572820.0, ans=0.125 2023-11-23 22:50:03,045 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 22:50:29,324 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 385950 2023-11-23 22:50:29,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2572953.3333333335, ans=0.0 2023-11-23 22:50:47,804 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1200, loss[loss=0.06207, simple_loss=0.0807, pruned_loss=0.01319, audio_tagging_loss=0.008537, over 13946.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09188, pruned_loss=0.01377, audio_tagging_loss=0.009028, over 3039428.98 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:50:47,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2573086.6666666665, ans=0.2 2023-11-23 22:50:49,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2573086.6666666665, ans=0.0 2023-11-23 22:50:57,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2573086.6666666665, ans=0.0 2023-11-23 22:51:14,903 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 22:51:31,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386000 2023-11-23 22:51:31,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2573286.6666666665, ans=0.07 2023-11-23 22:51:38,144 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.994e+01 8.408e+01 9.187e+01 9.835e+01 1.432e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-23 22:51:50,527 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1250, loss[loss=0.06907, simple_loss=0.08711, pruned_loss=0.01461, audio_tagging_loss=0.0109, over 14789.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09232, pruned_loss=0.01381, audio_tagging_loss=0.00891, over 3045236.58 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:52:17,983 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.66 vs. limit=15.0 2023-11-23 22:52:18,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2573553.3333333335, ans=0.125 2023-11-23 22:52:31,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2573620.0, ans=0.125 2023-11-23 22:52:35,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386050 2023-11-23 22:52:35,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0 2023-11-23 22:52:37,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2573620.0, ans=0.2 2023-11-23 22:52:52,993 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1300, loss[loss=0.07453, simple_loss=0.1042, pruned_loss=0.01457, audio_tagging_loss=0.007859, over 14128.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09308, pruned_loss=0.01379, audio_tagging_loss=0.008803, over 3045131.18 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:53:00,606 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 22:53:21,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2573886.6666666665, ans=0.1 2023-11-23 22:53:22,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2573886.6666666665, ans=0.0 2023-11-23 22:53:36,840 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386100 2023-11-23 22:53:42,573 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.267e+01 8.921e+01 9.505e+01 1.791e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-23 22:53:53,275 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.26 vs. limit=15.0 2023-11-23 22:53:55,061 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1350, loss[loss=0.07845, simple_loss=0.1072, pruned_loss=0.01724, audio_tagging_loss=0.007626, over 14613.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09165, pruned_loss=0.01364, audio_tagging_loss=0.008999, over 3046363.75 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:54:38,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386150 2023-11-23 22:54:39,678 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 22:54:39,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2574286.6666666665, ans=0.125 2023-11-23 22:54:45,613 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.40 vs. limit=15.0 2023-11-23 22:54:52,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2574353.3333333335, ans=0.125 2023-11-23 22:54:57,763 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1400, loss[loss=0.07989, simple_loss=0.1098, pruned_loss=0.01626, audio_tagging_loss=0.008727, over 14054.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09244, pruned_loss=0.01372, audio_tagging_loss=0.008988, over 3051451.04 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:54:59,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_na.min_abs, batch_count=2574420.0, ans=0.02 2023-11-23 22:55:09,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2574486.6666666665, ans=0.125 2023-11-23 22:55:19,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2574486.6666666665, ans=0.1 2023-11-23 22:55:20,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2574553.3333333335, ans=0.2 2023-11-23 22:55:41,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386200 2023-11-23 22:55:48,296 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.327e+01 9.200e+01 9.811e+01 1.262e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-23 22:55:49,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2574686.6666666665, ans=0.2 2023-11-23 22:55:56,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2574686.6666666665, ans=0.125 2023-11-23 22:55:58,976 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1450, loss[loss=0.06197, simple_loss=0.07835, pruned_loss=0.01236, audio_tagging_loss=0.01044, over 14357.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09268, pruned_loss=0.01376, audio_tagging_loss=0.009036, over 3046742.00 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:56:05,381 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.18 vs. limit=15.0 2023-11-23 22:56:17,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2574820.0, ans=0.2 2023-11-23 22:56:19,802 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.38 vs. limit=22.5 2023-11-23 22:56:24,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2574886.6666666665, ans=0.1 2023-11-23 22:56:30,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2574886.6666666665, ans=0.2 2023-11-23 22:56:42,111 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386250 2023-11-23 22:57:00,664 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1500, loss[loss=0.05152, simple_loss=0.06555, pruned_loss=0.008353, audio_tagging_loss=0.01039, over 15145.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09278, pruned_loss=0.01389, audio_tagging_loss=0.009113, over 3057448.71 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:57:36,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2575220.0, ans=0.125 2023-11-23 22:57:43,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2575286.6666666665, ans=0.125 2023-11-23 22:57:44,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386300 2023-11-23 22:57:49,071 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.25 vs. limit=15.0 2023-11-23 22:57:50,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2575353.3333333335, ans=0.1 2023-11-23 22:57:51,768 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.322e+01 9.192e+01 9.802e+01 2.224e+02, threshold=1.838e+02, percent-clipped=1.0 2023-11-23 22:57:53,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2575353.3333333335, ans=0.0 2023-11-23 22:57:57,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2575353.3333333335, ans=10.0 2023-11-23 22:58:04,260 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1550, loss[loss=0.05506, simple_loss=0.0608, pruned_loss=0.01069, audio_tagging_loss=0.01397, over 14386.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09279, pruned_loss=0.01389, audio_tagging_loss=0.009207, over 3061296.89 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 22:58:12,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2575420.0, ans=0.0 2023-11-23 22:58:42,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2575620.0, ans=0.125 2023-11-23 22:58:47,313 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386350 2023-11-23 22:58:52,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2575686.6666666665, ans=0.125 2023-11-23 22:59:03,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2575686.6666666665, ans=0.1 2023-11-23 22:59:05,876 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1600, loss[loss=0.08826, simple_loss=0.1117, pruned_loss=0.0237, audio_tagging_loss=0.008703, over 14634.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09221, pruned_loss=0.01383, audio_tagging_loss=0.00937, over 3060654.78 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 22:59:14,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2575753.3333333335, ans=0.0 2023-11-23 22:59:46,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2575953.3333333335, ans=0.1 2023-11-23 22:59:49,999 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386400 2023-11-23 22:59:51,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.35 vs. limit=22.5 2023-11-23 22:59:55,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2576020.0, ans=0.025 2023-11-23 22:59:57,363 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.368e+01 9.062e+01 9.713e+01 1.291e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-23 23:00:04,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2576020.0, ans=0.125 2023-11-23 23:00:08,301 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1650, loss[loss=0.04711, simple_loss=0.06248, pruned_loss=0.007806, audio_tagging_loss=0.008063, over 15779.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09207, pruned_loss=0.01376, audio_tagging_loss=0.009398, over 3055035.14 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:00:08,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2576086.6666666665, ans=0.125 2023-11-23 23:00:34,014 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.67 vs. limit=15.0 2023-11-23 23:00:35,336 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.42 vs. limit=15.0 2023-11-23 23:00:48,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2576286.6666666665, ans=0.1 2023-11-23 23:00:50,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2576286.6666666665, ans=0.1 2023-11-23 23:00:52,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386450 2023-11-23 23:01:01,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2576353.3333333335, ans=0.125 2023-11-23 23:01:05,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.whiten.whitening_limit, batch_count=2576353.3333333335, ans=12.0 2023-11-23 23:01:12,037 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1700, loss[loss=0.05315, simple_loss=0.06541, pruned_loss=0.01144, audio_tagging_loss=0.009006, over 14870.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09222, pruned_loss=0.01383, audio_tagging_loss=0.009417, over 3050854.19 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:01:15,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2576420.0, ans=0.125 2023-11-23 23:01:21,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2576420.0, ans=0.0 2023-11-23 23:01:21,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2576420.0, ans=0.125 2023-11-23 23:01:36,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2576553.3333333335, ans=0.125 2023-11-23 23:01:55,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386500 2023-11-23 23:01:56,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2576620.0, ans=0.125 2023-11-23 23:02:03,012 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.350e+01 8.996e+01 9.717e+01 1.219e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-23 23:02:14,545 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1750, loss[loss=0.06002, simple_loss=0.07989, pruned_loss=0.00996, audio_tagging_loss=0.01011, over 14037.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09216, pruned_loss=0.01386, audio_tagging_loss=0.009308, over 3055531.17 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:02:30,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2576820.0, ans=0.0 2023-11-23 23:02:31,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2576820.0, ans=0.0 2023-11-23 23:02:46,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0 2023-11-23 23:02:58,123 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386550 2023-11-23 23:03:04,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2577020.0, ans=0.125 2023-11-23 23:03:08,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.77 vs. limit=22.5 2023-11-23 23:03:14,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2577086.6666666665, ans=0.0 2023-11-23 23:03:15,783 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1800, loss[loss=0.07706, simple_loss=0.1017, pruned_loss=0.01791, audio_tagging_loss=0.008301, over 14554.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09232, pruned_loss=0.01384, audio_tagging_loss=0.009271, over 3049608.70 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:03:18,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2577086.6666666665, ans=0.1 2023-11-23 23:03:59,270 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386600 2023-11-23 23:04:08,238 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.512e+01 8.514e+01 9.080e+01 9.768e+01 1.273e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-23 23:04:13,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2577353.3333333335, ans=0.0 2023-11-23 23:04:18,923 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1850, loss[loss=0.08834, simple_loss=0.1132, pruned_loss=0.01954, audio_tagging_loss=0.0122, over 14406.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09203, pruned_loss=0.01387, audio_tagging_loss=0.009216, over 3043118.63 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 23:04:20,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2577420.0, ans=0.07 2023-11-23 23:04:42,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2577553.3333333335, ans=0.2 2023-11-23 23:04:57,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2577620.0, ans=0.07 2023-11-23 23:05:01,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2577620.0, ans=0.2 2023-11-23 23:05:02,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386650 2023-11-23 23:05:20,248 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1900, loss[loss=0.05653, simple_loss=0.07288, pruned_loss=0.012, audio_tagging_loss=0.008084, over 14643.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09222, pruned_loss=0.01376, audio_tagging_loss=0.009125, over 3040738.28 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 23:05:32,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.43 vs. limit=15.0 2023-11-23 23:05:33,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2577820.0, ans=0.1 2023-11-23 23:06:03,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2577953.3333333335, ans=0.05 2023-11-23 23:06:04,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386700 2023-11-23 23:06:13,467 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.280e+01 8.966e+01 9.619e+01 1.252e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-23 23:06:16,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2578020.0, ans=0.1 2023-11-23 23:06:23,281 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 1950, loss[loss=0.05092, simple_loss=0.05943, pruned_loss=0.007979, audio_tagging_loss=0.01323, over 14462.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09134, pruned_loss=0.0137, audio_tagging_loss=0.009106, over 3036461.73 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 23:06:23,864 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.99 vs. limit=6.0 2023-11-23 23:06:26,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2578086.6666666665, ans=0.125 2023-11-23 23:06:57,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2578220.0, ans=0.1 2023-11-23 23:07:05,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2578286.6666666665, ans=0.2 2023-11-23 23:07:07,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386750 2023-11-23 23:07:09,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.71 vs. limit=10.0 2023-11-23 23:07:10,316 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.77 vs. limit=15.0 2023-11-23 23:07:13,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2578353.3333333335, ans=0.125 2023-11-23 23:07:26,428 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2000, loss[loss=0.09136, simple_loss=0.1209, pruned_loss=0.02183, audio_tagging_loss=0.009064, over 14635.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09063, pruned_loss=0.01374, audio_tagging_loss=0.009113, over 3037953.78 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:07:35,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2578420.0, ans=0.0 2023-11-23 23:07:56,827 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.15 vs. limit=15.0 2023-11-23 23:07:59,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.94 vs. limit=15.0 2023-11-23 23:08:09,847 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386800 2023-11-23 23:08:19,215 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.226e+01 8.902e+01 9.752e+01 1.387e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-23 23:08:28,805 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2050, loss[loss=0.06521, simple_loss=0.09082, pruned_loss=0.01107, audio_tagging_loss=0.008726, over 15798.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09112, pruned_loss=0.01377, audio_tagging_loss=0.00896, over 3037340.90 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:08:29,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2578753.3333333335, ans=0.125 2023-11-23 23:08:29,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2578753.3333333335, ans=0.09899494936611666 2023-11-23 23:08:31,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2578753.3333333335, ans=0.5 2023-11-23 23:08:31,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2578753.3333333335, ans=0.125 2023-11-23 23:09:09,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2578953.3333333335, ans=0.125 2023-11-23 23:09:12,683 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386850 2023-11-23 23:09:28,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2579020.0, ans=0.1 2023-11-23 23:09:30,967 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2100, loss[loss=0.06009, simple_loss=0.0776, pruned_loss=0.013, audio_tagging_loss=0.008291, over 14234.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09094, pruned_loss=0.01375, audio_tagging_loss=0.008919, over 3035220.88 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:09:45,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2579153.3333333335, ans=0.0 2023-11-23 23:10:08,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2579286.6666666665, ans=0.1 2023-11-23 23:10:14,331 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386900 2023-11-23 23:10:23,221 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.184e+01 8.426e+01 9.085e+01 9.880e+01 1.258e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-23 23:10:33,285 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2150, loss[loss=0.06935, simple_loss=0.09975, pruned_loss=0.01425, audio_tagging_loss=0.005228, over 15143.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09111, pruned_loss=0.01347, audio_tagging_loss=0.008908, over 3033352.41 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:10:41,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2579420.0, ans=0.125 2023-11-23 23:10:42,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2579420.0, ans=0.125 2023-11-23 23:10:52,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2579486.6666666665, ans=0.125 2023-11-23 23:11:10,841 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:11:11,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2579620.0, ans=0.0 2023-11-23 23:11:17,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 386950 2023-11-23 23:11:36,329 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2200, loss[loss=0.06428, simple_loss=0.08788, pruned_loss=0.009697, audio_tagging_loss=0.01064, over 16269.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09086, pruned_loss=0.01352, audio_tagging_loss=0.008902, over 3036407.72 frames. ], batch size: 61, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 23:11:46,465 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.40 vs. limit=10.0 2023-11-23 23:12:05,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.38 vs. limit=22.5 2023-11-23 23:12:20,250 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387000 2023-11-23 23:12:29,890 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.440e+01 8.536e+01 9.005e+01 9.625e+01 2.688e+02, threshold=1.801e+02, percent-clipped=1.0 2023-11-23 23:12:38,246 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2250, loss[loss=0.08531, simple_loss=0.1255, pruned_loss=0.01609, audio_tagging_loss=0.006484, over 15335.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09215, pruned_loss=0.01363, audio_tagging_loss=0.008922, over 3041571.55 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 23:12:39,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.46 vs. limit=15.0 2023-11-23 23:12:40,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2580086.6666666665, ans=0.0 2023-11-23 23:13:13,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2580220.0, ans=0.125 2023-11-23 23:13:20,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2580286.6666666665, ans=0.125 2023-11-23 23:13:22,556 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387050 2023-11-23 23:13:33,728 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.61 vs. limit=12.0 2023-11-23 23:13:41,185 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2300, loss[loss=0.0713, simple_loss=0.1002, pruned_loss=0.01268, audio_tagging_loss=0.008521, over 14654.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.092, pruned_loss=0.01355, audio_tagging_loss=0.008939, over 3042836.28 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 23:13:50,117 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.59 vs. limit=15.0 2023-11-23 23:14:24,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387100 2023-11-23 23:14:30,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2580686.6666666665, ans=0.2 2023-11-23 23:14:31,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2580686.6666666665, ans=0.125 2023-11-23 23:14:34,372 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.489e+01 9.011e+01 9.554e+01 1.239e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-23 23:14:36,322 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:14:40,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2580686.6666666665, ans=0.125 2023-11-23 23:14:43,640 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2350, loss[loss=0.07711, simple_loss=0.1018, pruned_loss=0.01361, audio_tagging_loss=0.01262, over 15218.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09183, pruned_loss=0.01366, audio_tagging_loss=0.009139, over 3042874.05 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 16.0 2023-11-23 23:14:52,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2580753.3333333335, ans=0.0 2023-11-23 23:14:57,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.64 vs. limit=22.5 2023-11-23 23:15:00,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2580820.0, ans=0.0 2023-11-23 23:15:10,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2580886.6666666665, ans=0.125 2023-11-23 23:15:14,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2580886.6666666665, ans=0.125 2023-11-23 23:15:27,060 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387150 2023-11-23 23:15:44,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.09 vs. limit=15.0 2023-11-23 23:15:44,728 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2400, loss[loss=0.05056, simple_loss=0.06865, pruned_loss=0.007768, audio_tagging_loss=0.008466, over 15169.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09207, pruned_loss=0.01358, audio_tagging_loss=0.009143, over 3049495.90 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 32.0 2023-11-23 23:16:17,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0 2023-11-23 23:16:19,394 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.01 vs. limit=12.0 2023-11-23 23:16:19,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.90 vs. limit=6.0 2023-11-23 23:16:28,354 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387200 2023-11-23 23:16:35,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2581353.3333333335, ans=0.0 2023-11-23 23:16:39,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.668e+01 9.147e+01 9.893e+01 1.218e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-23 23:16:46,717 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2450, loss[loss=0.07516, simple_loss=0.1031, pruned_loss=0.0158, audio_tagging_loss=0.007823, over 14709.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09272, pruned_loss=0.01377, audio_tagging_loss=0.009117, over 3046341.90 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:17:00,908 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.47 vs. limit=10.0 2023-11-23 23:17:29,911 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387250 2023-11-23 23:17:45,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2581686.6666666665, ans=0.1 2023-11-23 23:17:49,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2581753.3333333335, ans=0.1 2023-11-23 23:17:50,316 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2500, loss[loss=0.08173, simple_loss=0.1163, pruned_loss=0.01593, audio_tagging_loss=0.00763, over 15518.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.0928, pruned_loss=0.01382, audio_tagging_loss=0.009135, over 3040834.25 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:18:31,219 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.44 vs. limit=15.0 2023-11-23 23:18:32,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2581953.3333333335, ans=0.125 2023-11-23 23:18:33,750 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387300 2023-11-23 23:18:35,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2581953.3333333335, ans=0.125 2023-11-23 23:18:41,387 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=22.5 2023-11-23 23:18:43,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2582020.0, ans=0.5 2023-11-23 23:18:44,144 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.436e+01 9.159e+01 9.960e+01 1.411e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-23 23:18:44,452 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 23:18:51,371 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2550, loss[loss=0.05938, simple_loss=0.08423, pruned_loss=0.01163, audio_tagging_loss=0.005632, over 14645.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09259, pruned_loss=0.01388, audio_tagging_loss=0.009034, over 3040383.61 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:19:08,104 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-23 23:19:17,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2582220.0, ans=0.125 2023-11-23 23:19:22,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2582220.0, ans=0.125 2023-11-23 23:19:27,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2582220.0, ans=0.125 2023-11-23 23:19:35,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387350 2023-11-23 23:19:47,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2582353.3333333335, ans=0.125 2023-11-23 23:19:52,920 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2600, loss[loss=0.05643, simple_loss=0.08002, pruned_loss=0.008874, audio_tagging_loss=0.007547, over 14518.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09203, pruned_loss=0.01368, audio_tagging_loss=0.009017, over 3035267.29 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:19:59,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2582420.0, ans=0.0 2023-11-23 23:20:13,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2582486.6666666665, ans=0.125 2023-11-23 23:20:17,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2582553.3333333335, ans=0.2 2023-11-23 23:20:27,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2582553.3333333335, ans=0.0 2023-11-23 23:20:29,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2582620.0, ans=0.125 2023-11-23 23:20:36,541 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387400 2023-11-23 23:20:48,950 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.392e+01 8.863e+01 9.833e+01 1.286e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-23 23:20:56,694 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2650, loss[loss=0.07504, simple_loss=0.102, pruned_loss=0.01638, audio_tagging_loss=0.007644, over 15659.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09233, pruned_loss=0.01368, audio_tagging_loss=0.008877, over 3038777.79 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:21:00,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2582753.3333333335, ans=0.125 2023-11-23 23:21:01,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2582753.3333333335, ans=0.125 2023-11-23 23:21:05,592 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.03 vs. limit=22.5 2023-11-23 23:21:14,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2582820.0, ans=0.0 2023-11-23 23:21:40,188 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387450 2023-11-23 23:21:55,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2583020.0, ans=0.125 2023-11-23 23:21:58,885 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2700, loss[loss=0.06903, simple_loss=0.1033, pruned_loss=0.008765, audio_tagging_loss=0.008628, over 16343.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09203, pruned_loss=0.01364, audio_tagging_loss=0.008897, over 3038476.34 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:22:02,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2583086.6666666665, ans=0.015 2023-11-23 23:22:02,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2583086.6666666665, ans=0.0 2023-11-23 23:22:42,499 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387500 2023-11-23 23:22:42,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2583286.6666666665, ans=0.125 2023-11-23 23:22:45,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2583286.6666666665, ans=0.125 2023-11-23 23:22:54,185 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.525e+01 8.709e+01 9.550e+01 1.033e+02 1.468e+02, threshold=1.910e+02, percent-clipped=0.0 2023-11-23 23:22:56,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2583353.3333333335, ans=0.0 2023-11-23 23:23:00,141 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2750, loss[loss=0.05116, simple_loss=0.06443, pruned_loss=0.009294, audio_tagging_loss=0.009646, over 15612.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09223, pruned_loss=0.01378, audio_tagging_loss=0.008914, over 3037848.86 frames. ], batch size: 64, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:23:16,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2583486.6666666665, ans=0.09899494936611666 2023-11-23 23:23:44,147 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387550 2023-11-23 23:23:49,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2583686.6666666665, ans=10.0 2023-11-23 23:23:54,120 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:24:02,861 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2800, loss[loss=0.07317, simple_loss=0.09536, pruned_loss=0.0129, audio_tagging_loss=0.01258, over 14939.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09275, pruned_loss=0.0138, audio_tagging_loss=0.008883, over 3046132.09 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:24:21,620 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.51 vs. limit=22.5 2023-11-23 23:24:38,833 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 23:24:40,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2583953.3333333335, ans=0.125 2023-11-23 23:24:46,378 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387600 2023-11-23 23:24:59,635 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.278e+01 8.804e+01 9.483e+01 1.291e+02, threshold=1.761e+02, percent-clipped=0.0 2023-11-23 23:25:06,135 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2850, loss[loss=0.07308, simple_loss=0.09356, pruned_loss=0.01695, audio_tagging_loss=0.009349, over 13884.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09262, pruned_loss=0.01372, audio_tagging_loss=0.008862, over 3047871.19 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:25:26,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2584153.3333333335, ans=0.0 2023-11-23 23:25:49,613 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387650 2023-11-23 23:25:58,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2584353.3333333335, ans=0.125 2023-11-23 23:25:58,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2584353.3333333335, ans=0.09899494936611666 2023-11-23 23:26:07,821 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2900, loss[loss=0.06177, simple_loss=0.08448, pruned_loss=0.01065, audio_tagging_loss=0.008882, over 15463.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09258, pruned_loss=0.01376, audio_tagging_loss=0.008809, over 3044726.47 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:26:15,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2584420.0, ans=0.1 2023-11-23 23:26:51,938 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387700 2023-11-23 23:27:03,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2584686.6666666665, ans=0.0 2023-11-23 23:27:03,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2584686.6666666665, ans=0.125 2023-11-23 23:27:05,507 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.293e+01 9.102e+01 9.710e+01 1.234e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-23 23:27:10,866 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 2950, loss[loss=0.07547, simple_loss=0.1006, pruned_loss=0.01707, audio_tagging_loss=0.008102, over 14416.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09392, pruned_loss=0.014, audio_tagging_loss=0.008777, over 3053632.82 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:27:40,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2584886.6666666665, ans=0.2 2023-11-23 23:27:54,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387750 2023-11-23 23:27:57,344 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2584953.3333333335, ans=0.2 2023-11-23 23:28:10,178 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.05 vs. limit=6.0 2023-11-23 23:28:13,265 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3000, loss[loss=0.04733, simple_loss=0.05936, pruned_loss=0.008676, audio_tagging_loss=0.008976, over 14698.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09348, pruned_loss=0.01399, audio_tagging_loss=0.008874, over 3051138.92 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:28:13,266 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-23 23:28:41,510 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9764, 3.2288, 2.9152, 3.2038, 3.4079, 2.7925, 3.4024, 2.7214], device='cuda:3') 2023-11-23 23:28:53,109 INFO [train_asr.py:1253] (3/4) Epoch 33, validation: loss=0.05846, simple_loss=0.05103, pruned_loss=0.005194, audio_tagging_loss=0.02775, over 4681554.00 frames. 2023-11-23 23:28:53,110 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-23 23:28:55,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2585086.6666666665, ans=0.95 2023-11-23 23:29:04,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2585086.6666666665, ans=0.125 2023-11-23 23:29:11,135 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.93 vs. limit=22.5 2023-11-23 23:29:30,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2585286.6666666665, ans=0.1 2023-11-23 23:29:32,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2585286.6666666665, ans=0.125 2023-11-23 23:29:37,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387800 2023-11-23 23:29:41,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2585286.6666666665, ans=0.1 2023-11-23 23:29:44,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2585353.3333333335, ans=0.0 2023-11-23 23:29:51,590 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.566e+01 9.149e+01 9.920e+01 1.258e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-23 23:29:57,000 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3050, loss[loss=0.07432, simple_loss=0.09598, pruned_loss=0.01921, audio_tagging_loss=0.007121, over 16069.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09405, pruned_loss=0.01413, audio_tagging_loss=0.008917, over 3052890.76 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:30:00,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.33 vs. limit=12.0 2023-11-23 23:30:35,009 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:30:36,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2585620.0, ans=0.5 2023-11-23 23:30:42,377 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387850 2023-11-23 23:30:49,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2585686.6666666665, ans=0.125 2023-11-23 23:30:51,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2585686.6666666665, ans=0.0 2023-11-23 23:31:01,218 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3100, loss[loss=0.08687, simple_loss=0.1198, pruned_loss=0.02003, audio_tagging_loss=0.006919, over 15828.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09422, pruned_loss=0.01416, audio_tagging_loss=0.00891, over 3046109.12 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:31:11,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2585753.3333333335, ans=0.0 2023-11-23 23:31:22,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2585820.0, ans=0.125 2023-11-23 23:31:24,205 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.61 vs. limit=15.0 2023-11-23 23:31:25,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.90 vs. limit=10.0 2023-11-23 23:31:30,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2585886.6666666665, ans=0.0 2023-11-23 23:31:45,707 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387900 2023-11-23 23:31:48,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2585953.3333333335, ans=0.125 2023-11-23 23:31:52,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=15.0 2023-11-23 23:31:59,121 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.717e+01 9.122e+01 9.882e+01 2.384e+02, threshold=1.824e+02, percent-clipped=1.0 2023-11-23 23:32:02,047 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.25 vs. limit=15.0 2023-11-23 23:32:03,854 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3150, loss[loss=0.06341, simple_loss=0.08153, pruned_loss=0.009428, audio_tagging_loss=0.01322, over 14897.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09346, pruned_loss=0.01397, audio_tagging_loss=0.009106, over 3051437.54 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 8.0 2023-11-23 23:32:17,479 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.05 vs. limit=22.5 2023-11-23 23:32:35,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2586220.0, ans=0.07 2023-11-23 23:32:43,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2586286.6666666665, ans=0.125 2023-11-23 23:32:47,539 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 387950 2023-11-23 23:33:06,482 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3200, loss[loss=0.06215, simple_loss=0.0803, pruned_loss=0.008527, audio_tagging_loss=0.01347, over 15249.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09428, pruned_loss=0.01417, audio_tagging_loss=0.009046, over 3051625.66 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:33:09,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2586420.0, ans=0.1 2023-11-23 23:33:20,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2586486.6666666665, ans=0.125 2023-11-23 23:33:20,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2586486.6666666665, ans=0.0 2023-11-23 23:33:40,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2586553.3333333335, ans=0.125 2023-11-23 23:33:40,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2586553.3333333335, ans=0.125 2023-11-23 23:33:50,075 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388000 2023-11-23 23:34:02,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2586686.6666666665, ans=0.125 2023-11-23 23:34:06,877 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.288e+01 8.870e+01 9.620e+01 1.279e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-23 23:34:11,753 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3250, loss[loss=0.06011, simple_loss=0.0681, pruned_loss=0.01268, audio_tagging_loss=0.01338, over 14397.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09284, pruned_loss=0.01394, audio_tagging_loss=0.009183, over 3051668.36 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:34:14,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.62 vs. limit=22.5 2023-11-23 23:34:27,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2586820.0, ans=15.0 2023-11-23 23:34:35,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2586886.6666666665, ans=0.05 2023-11-23 23:34:38,292 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.18 vs. limit=22.5 2023-11-23 23:34:56,657 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388050 2023-11-23 23:35:00,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2586953.3333333335, ans=0.0 2023-11-23 23:35:10,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2587020.0, ans=0.2 2023-11-23 23:35:15,195 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3300, loss[loss=0.05514, simple_loss=0.07013, pruned_loss=0.01049, audio_tagging_loss=0.009594, over 16099.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09137, pruned_loss=0.01381, audio_tagging_loss=0.009313, over 3055021.32 frames. ], batch size: 63, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:35:43,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2587220.0, ans=0.125 2023-11-23 23:35:51,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2587286.6666666665, ans=0.05 2023-11-23 23:35:52,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2587286.6666666665, ans=0.0 2023-11-23 23:35:58,552 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388100 2023-11-23 23:36:05,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2587353.3333333335, ans=0.0 2023-11-23 23:36:12,199 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.544e+01 9.048e+01 9.852e+01 1.338e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-23 23:36:18,179 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3350, loss[loss=0.05027, simple_loss=0.06788, pruned_loss=0.006869, audio_tagging_loss=0.009463, over 15367.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09165, pruned_loss=0.01382, audio_tagging_loss=0.00925, over 3055646.41 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:36:37,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2587486.6666666665, ans=0.0 2023-11-23 23:36:42,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2587553.3333333335, ans=0.1 2023-11-23 23:36:48,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2587553.3333333335, ans=0.1 2023-11-23 23:37:02,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388150 2023-11-23 23:37:02,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2587620.0, ans=0.2 2023-11-23 23:37:03,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2587620.0, ans=0.1 2023-11-23 23:37:20,749 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3400, loss[loss=0.08196, simple_loss=0.1072, pruned_loss=0.02095, audio_tagging_loss=0.007383, over 16624.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09257, pruned_loss=0.01396, audio_tagging_loss=0.009052, over 3054381.44 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:37:23,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2587753.3333333335, ans=0.0 2023-11-23 23:37:29,451 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-23 23:37:29,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2587753.3333333335, ans=0.1 2023-11-23 23:37:33,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2587820.0, ans=0.05 2023-11-23 23:37:50,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2587886.6666666665, ans=0.125 2023-11-23 23:37:50,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2587886.6666666665, ans=0.0 2023-11-23 23:37:51,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2587886.6666666665, ans=0.125 2023-11-23 23:37:53,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2587886.6666666665, ans=0.1 2023-11-23 23:38:04,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388200 2023-11-23 23:38:05,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2587953.3333333335, ans=0.1 2023-11-23 23:38:15,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2588020.0, ans=0.125 2023-11-23 23:38:17,823 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.375e+01 8.962e+01 9.583e+01 1.424e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-23 23:38:22,588 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3450, loss[loss=0.08072, simple_loss=0.1133, pruned_loss=0.01757, audio_tagging_loss=0.006508, over 14605.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09224, pruned_loss=0.01377, audio_tagging_loss=0.008951, over 3052973.61 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:38:46,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.15 vs. limit=22.5 2023-11-23 23:39:03,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2588286.6666666665, ans=0.125 2023-11-23 23:39:07,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388250 2023-11-23 23:39:26,127 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3500, loss[loss=0.04834, simple_loss=0.06273, pruned_loss=0.00846, audio_tagging_loss=0.008515, over 13727.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09133, pruned_loss=0.01367, audio_tagging_loss=0.008915, over 3051147.08 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:39:33,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2588420.0, ans=0.2 2023-11-23 23:39:33,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2588420.0, ans=0.2 2023-11-23 23:39:41,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2588486.6666666665, ans=0.2 2023-11-23 23:39:45,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.95 vs. limit=15.0 2023-11-23 23:39:54,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2588553.3333333335, ans=0.125 2023-11-23 23:39:56,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.17 vs. limit=22.5 2023-11-23 23:39:57,875 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:40:02,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2588620.0, ans=0.0 2023-11-23 23:40:09,089 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388300 2023-11-23 23:40:17,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2588686.6666666665, ans=0.125 2023-11-23 23:40:23,680 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.607e+01 9.289e+01 1.010e+02 1.260e+02, threshold=1.858e+02, percent-clipped=0.0 2023-11-23 23:40:28,516 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3550, loss[loss=0.06013, simple_loss=0.08448, pruned_loss=0.008409, audio_tagging_loss=0.009486, over 13823.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09136, pruned_loss=0.01375, audio_tagging_loss=0.008941, over 3045497.18 frames. ], batch size: 52, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:40:33,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2588753.3333333335, ans=0.1 2023-11-23 23:40:37,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2588753.3333333335, ans=0.0 2023-11-23 23:40:57,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2588886.6666666665, ans=0.0 2023-11-23 23:41:12,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388350 2023-11-23 23:41:12,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2588953.3333333335, ans=0.2 2023-11-23 23:41:18,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2589020.0, ans=0.025 2023-11-23 23:41:23,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2589020.0, ans=0.125 2023-11-23 23:41:25,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2589020.0, ans=0.125 2023-11-23 23:41:26,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2589020.0, ans=0.0 2023-11-23 23:41:29,126 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 23:41:30,139 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3600, loss[loss=0.07392, simple_loss=0.09456, pruned_loss=0.01326, audio_tagging_loss=0.01338, over 15483.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09095, pruned_loss=0.01352, audio_tagging_loss=0.008909, over 3045838.01 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:41:32,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2589086.6666666665, ans=0.125 2023-11-23 23:41:40,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0 2023-11-23 23:41:42,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2589153.3333333335, ans=10.0 2023-11-23 23:42:01,020 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 23:42:01,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2589220.0, ans=0.0 2023-11-23 23:42:14,144 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388400 2023-11-23 23:42:28,666 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.372e+01 8.860e+01 9.569e+01 1.357e+02, threshold=1.772e+02, percent-clipped=0.0 2023-11-23 23:42:32,298 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3650, loss[loss=0.06841, simple_loss=0.09321, pruned_loss=0.01285, audio_tagging_loss=0.008953, over 15715.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.0926, pruned_loss=0.01378, audio_tagging_loss=0.008844, over 3040724.61 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:42:36,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2589420.0, ans=0.125 2023-11-23 23:42:39,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2589420.0, ans=0.2 2023-11-23 23:43:03,135 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.94 vs. limit=15.0 2023-11-23 23:43:16,787 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388450 2023-11-23 23:43:22,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2589686.6666666665, ans=0.2 2023-11-23 23:43:29,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2589686.6666666665, ans=0.125 2023-11-23 23:43:29,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2589686.6666666665, ans=0.2 2023-11-23 23:43:35,933 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3700, loss[loss=0.06321, simple_loss=0.08856, pruned_loss=0.01186, audio_tagging_loss=0.007068, over 15822.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09264, pruned_loss=0.01381, audio_tagging_loss=0.008867, over 3050522.17 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:43:48,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2589820.0, ans=0.125 2023-11-23 23:43:52,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2589820.0, ans=0.0 2023-11-23 23:43:58,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2589886.6666666665, ans=0.125 2023-11-23 23:44:17,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2589953.3333333335, ans=0.0 2023-11-23 23:44:19,881 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388500 2023-11-23 23:44:27,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2590020.0, ans=6.0 2023-11-23 23:44:29,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2590020.0, ans=0.125 2023-11-23 23:44:29,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2590020.0, ans=0.1 2023-11-23 23:44:33,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.89 vs. limit=8.0 2023-11-23 23:44:33,983 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.486e+01 9.089e+01 9.768e+01 1.177e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-23 23:44:35,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2590020.0, ans=0.0 2023-11-23 23:44:37,523 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3750, loss[loss=0.08912, simple_loss=0.1191, pruned_loss=0.01944, audio_tagging_loss=0.01011, over 15428.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09295, pruned_loss=0.0138, audio_tagging_loss=0.00886, over 3047793.33 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:44:48,781 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.72 vs. limit=15.0 2023-11-23 23:45:02,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2590220.0, ans=0.0 2023-11-23 23:45:06,618 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.37 vs. limit=15.0 2023-11-23 23:45:14,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2590286.6666666665, ans=0.125 2023-11-23 23:45:21,629 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:45:21,667 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388550 2023-11-23 23:45:39,700 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3800, loss[loss=0.07488, simple_loss=0.1052, pruned_loss=0.01409, audio_tagging_loss=0.008175, over 15169.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.0928, pruned_loss=0.01378, audio_tagging_loss=0.008879, over 3043117.06 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:45:54,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2590486.6666666665, ans=0.125 2023-11-23 23:46:02,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2590486.6666666665, ans=0.0 2023-11-23 23:46:08,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2590553.3333333335, ans=0.05 2023-11-23 23:46:13,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2590553.3333333335, ans=0.125 2023-11-23 23:46:19,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2590620.0, ans=0.125 2023-11-23 23:46:23,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388600 2023-11-23 23:46:39,771 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 8.380e+01 8.887e+01 9.611e+01 1.323e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-23 23:46:44,008 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3850, loss[loss=0.06043, simple_loss=0.07704, pruned_loss=0.01274, audio_tagging_loss=0.009163, over 15379.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09208, pruned_loss=0.01377, audio_tagging_loss=0.008977, over 3043163.75 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:46:44,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2590753.3333333335, ans=0.2 2023-11-23 23:46:46,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2590753.3333333335, ans=0.0 2023-11-23 23:47:01,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2590820.0, ans=0.125 2023-11-23 23:47:06,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2590820.0, ans=0.125 2023-11-23 23:47:23,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.97 vs. limit=15.0 2023-11-23 23:47:28,768 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388650 2023-11-23 23:47:36,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.35 vs. limit=15.0 2023-11-23 23:47:47,375 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3900, loss[loss=0.07261, simple_loss=0.09582, pruned_loss=0.01564, audio_tagging_loss=0.009063, over 14922.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09111, pruned_loss=0.01342, audio_tagging_loss=0.009112, over 3041963.40 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:47:48,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2591086.6666666665, ans=0.125 2023-11-23 23:47:53,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2591086.6666666665, ans=0.1 2023-11-23 23:47:59,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2591153.3333333335, ans=0.125 2023-11-23 23:48:03,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2591153.3333333335, ans=0.125 2023-11-23 23:48:07,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2591153.3333333335, ans=0.0 2023-11-23 23:48:11,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2591220.0, ans=10.0 2023-11-23 23:48:14,627 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.56 vs. limit=15.0 2023-11-23 23:48:18,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2591220.0, ans=0.125 2023-11-23 23:48:20,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2591220.0, ans=0.2 2023-11-23 23:48:23,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2591220.0, ans=0.125 2023-11-23 23:48:30,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2591286.6666666665, ans=0.125 2023-11-23 23:48:32,166 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388700 2023-11-23 23:48:35,132 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.80 vs. limit=15.0 2023-11-23 23:48:39,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2591353.3333333335, ans=0.05 2023-11-23 23:48:43,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2591353.3333333335, ans=22.5 2023-11-23 23:48:46,588 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.330e+01 8.934e+01 9.660e+01 1.290e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-23 23:48:50,238 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 3950, loss[loss=0.07383, simple_loss=0.09483, pruned_loss=0.01823, audio_tagging_loss=0.008193, over 15990.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09127, pruned_loss=0.01351, audio_tagging_loss=0.00916, over 3047746.07 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:49:08,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2591486.6666666665, ans=0.125 2023-11-23 23:49:18,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2591553.3333333335, ans=0.125 2023-11-23 23:49:34,241 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388750 2023-11-23 23:49:36,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2591620.0, ans=0.07 2023-11-23 23:49:44,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2591686.6666666665, ans=0.125 2023-11-23 23:49:53,530 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4000, loss[loss=0.05685, simple_loss=0.06953, pruned_loss=0.01106, audio_tagging_loss=0.01102, over 14670.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09125, pruned_loss=0.01342, audio_tagging_loss=0.009274, over 3046295.28 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:50:17,204 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-23 23:50:18,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2591886.6666666665, ans=0.0 2023-11-23 23:50:37,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388800 2023-11-23 23:50:54,120 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.337e+01 9.138e+01 9.888e+01 1.432e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-23 23:50:56,669 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4050, loss[loss=0.06452, simple_loss=0.08051, pruned_loss=0.01112, audio_tagging_loss=0.01315, over 16121.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09077, pruned_loss=0.01337, audio_tagging_loss=0.009318, over 3050901.87 frames. ], batch size: 64, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:51:00,288 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:51:40,569 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388850 2023-11-23 23:51:58,739 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4100, loss[loss=0.06376, simple_loss=0.08276, pruned_loss=0.01205, audio_tagging_loss=0.01033, over 15265.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09147, pruned_loss=0.01348, audio_tagging_loss=0.009258, over 3052553.02 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:52:03,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2592420.0, ans=0.0 2023-11-23 23:52:42,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2592620.0, ans=10.0 2023-11-23 23:52:43,299 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388900 2023-11-23 23:52:59,659 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.379e+01 9.260e+01 9.829e+01 1.268e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-23 23:53:01,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2592753.3333333335, ans=0.0 2023-11-23 23:53:02,091 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4150, loss[loss=0.07384, simple_loss=0.1082, pruned_loss=0.01321, audio_tagging_loss=0.006551, over 15203.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09135, pruned_loss=0.01337, audio_tagging_loss=0.009136, over 3056554.53 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:53:16,171 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.61 vs. limit=22.5 2023-11-23 23:53:22,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2592820.0, ans=0.2 2023-11-23 23:53:33,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2592886.6666666665, ans=0.125 2023-11-23 23:53:36,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2592886.6666666665, ans=0.125 2023-11-23 23:53:37,124 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.38 vs. limit=15.0 2023-11-23 23:53:40,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2592953.3333333335, ans=0.125 2023-11-23 23:53:45,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 388950 2023-11-23 23:53:45,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.19 vs. limit=22.5 2023-11-23 23:53:48,208 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-23 23:53:50,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2593020.0, ans=0.125 2023-11-23 23:54:03,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2593086.6666666665, ans=0.0 2023-11-23 23:54:04,221 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4200, loss[loss=0.06248, simple_loss=0.0862, pruned_loss=0.009211, audio_tagging_loss=0.01017, over 15196.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09164, pruned_loss=0.01336, audio_tagging_loss=0.009006, over 3055186.42 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:54:20,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.09 vs. limit=15.0 2023-11-23 23:54:34,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2593220.0, ans=0.125 2023-11-23 23:54:46,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2593286.6666666665, ans=0.125 2023-11-23 23:54:47,989 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389000 2023-11-23 23:55:04,540 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.440e+01 9.074e+01 9.920e+01 1.262e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-23 23:55:06,273 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.23 vs. limit=15.0 2023-11-23 23:55:07,005 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4250, loss[loss=0.08676, simple_loss=0.116, pruned_loss=0.02097, audio_tagging_loss=0.007794, over 16195.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.092, pruned_loss=0.01339, audio_tagging_loss=0.008932, over 3054215.12 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:55:21,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2593486.6666666665, ans=0.125 2023-11-23 23:55:24,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2593486.6666666665, ans=0.0 2023-11-23 23:55:30,020 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.10 vs. limit=15.0 2023-11-23 23:55:32,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2593553.3333333335, ans=0.125 2023-11-23 23:55:32,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.47 vs. limit=10.0 2023-11-23 23:55:38,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2593553.3333333335, ans=0.1 2023-11-23 23:55:51,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389050 2023-11-23 23:55:54,984 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.94 vs. limit=15.0 2023-11-23 23:55:57,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2593686.6666666665, ans=0.0 2023-11-23 23:56:10,042 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4300, loss[loss=0.08377, simple_loss=0.1169, pruned_loss=0.01485, audio_tagging_loss=0.01045, over 15342.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09286, pruned_loss=0.01353, audio_tagging_loss=0.008916, over 3055473.89 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:56:24,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2593820.0, ans=0.0 2023-11-23 23:56:36,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2593886.6666666665, ans=0.125 2023-11-23 23:56:53,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389100 2023-11-23 23:56:57,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2593953.3333333335, ans=0.025 2023-11-23 23:57:10,080 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.164e+01 8.627e+01 9.104e+01 9.923e+01 1.149e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-23 23:57:12,442 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4350, loss[loss=0.06719, simple_loss=0.09268, pruned_loss=0.0122, audio_tagging_loss=0.008648, over 17058.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09307, pruned_loss=0.01358, audio_tagging_loss=0.008877, over 3052754.56 frames. ], batch size: 62, lr: 2.06e-03, grad_scale: 16.0 2023-11-23 23:57:54,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2594286.6666666665, ans=0.125 2023-11-23 23:57:56,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389150 2023-11-23 23:57:56,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2594286.6666666665, ans=0.0 2023-11-23 23:58:15,018 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4400, loss[loss=0.06584, simple_loss=0.09127, pruned_loss=0.01053, audio_tagging_loss=0.009669, over 15857.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09237, pruned_loss=0.01337, audio_tagging_loss=0.008909, over 3045763.53 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 32.0 2023-11-23 23:58:24,083 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.61 vs. limit=15.0 2023-11-23 23:58:27,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2594486.6666666665, ans=10.0 2023-11-23 23:58:36,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.09 vs. limit=15.0 2023-11-23 23:58:43,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2594553.3333333335, ans=0.0 2023-11-23 23:58:54,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2594620.0, ans=0.0 2023-11-23 23:58:58,467 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389200 2023-11-23 23:59:06,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2594686.6666666665, ans=0.0 2023-11-23 23:59:10,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2594686.6666666665, ans=0.125 2023-11-23 23:59:14,494 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.468e+01 8.522e+01 9.303e+01 9.983e+01 1.286e+02, threshold=1.861e+02, percent-clipped=0.0 2023-11-23 23:59:16,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2594753.3333333335, ans=0.125 2023-11-23 23:59:16,885 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4450, loss[loss=0.05987, simple_loss=0.08214, pruned_loss=0.01106, audio_tagging_loss=0.007741, over 16110.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09285, pruned_loss=0.01351, audio_tagging_loss=0.008831, over 3055085.82 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 32.0 2023-11-23 23:59:17,601 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.97 vs. limit=10.0 2023-11-23 23:59:33,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2594820.0, ans=10.0 2023-11-24 00:00:00,992 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389250 2023-11-24 00:00:18,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2595020.0, ans=0.125 2023-11-24 00:00:20,331 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4500, loss[loss=0.06484, simple_loss=0.08953, pruned_loss=0.01394, audio_tagging_loss=0.006144, over 14105.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09255, pruned_loss=0.01337, audio_tagging_loss=0.008846, over 3049662.61 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:00:39,450 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:00:44,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2595220.0, ans=0.125 2023-11-24 00:00:50,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2595220.0, ans=0.0 2023-11-24 00:01:04,322 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389300 2023-11-24 00:01:08,263 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=22.5 2023-11-24 00:01:16,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2595353.3333333335, ans=0.1 2023-11-24 00:01:19,570 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.819e+01 8.069e+01 8.788e+01 9.879e+01 1.178e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-24 00:01:22,011 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4550, loss[loss=0.08207, simple_loss=0.1145, pruned_loss=0.01768, audio_tagging_loss=0.007134, over 15251.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.0923, pruned_loss=0.01347, audio_tagging_loss=0.008807, over 3047049.59 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:01:59,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2595620.0, ans=0.2 2023-11-24 00:02:05,556 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389350 2023-11-24 00:02:10,197 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 00:02:23,781 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4600, loss[loss=0.08901, simple_loss=0.1247, pruned_loss=0.01796, audio_tagging_loss=0.008705, over 15399.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09225, pruned_loss=0.01351, audio_tagging_loss=0.008823, over 3050678.35 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:02:24,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2595753.3333333335, ans=0.1 2023-11-24 00:02:37,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2595820.0, ans=0.04949747468305833 2023-11-24 00:02:37,898 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.20 vs. limit=15.0 2023-11-24 00:02:41,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2595820.0, ans=0.05 2023-11-24 00:02:46,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2595820.0, ans=0.125 2023-11-24 00:02:57,591 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0 2023-11-24 00:02:58,388 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:03:06,753 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389400 2023-11-24 00:03:07,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2595953.3333333335, ans=0.125 2023-11-24 00:03:24,565 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.634e+01 9.201e+01 1.007e+02 1.274e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-24 00:03:26,982 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4650, loss[loss=0.07387, simple_loss=0.102, pruned_loss=0.01263, audio_tagging_loss=0.01023, over 14986.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09199, pruned_loss=0.01341, audio_tagging_loss=0.00892, over 3053154.08 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:03:33,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2596086.6666666665, ans=0.1 2023-11-24 00:03:37,297 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.28 vs. limit=5.0 2023-11-24 00:03:41,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2596153.3333333335, ans=0.0 2023-11-24 00:03:54,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2596220.0, ans=0.2 2023-11-24 00:04:10,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389450 2023-11-24 00:04:15,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2596353.3333333335, ans=0.125 2023-11-24 00:04:28,866 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4700, loss[loss=0.04954, simple_loss=0.06472, pruned_loss=0.008943, audio_tagging_loss=0.008236, over 15535.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09107, pruned_loss=0.01332, audio_tagging_loss=0.009027, over 3053786.58 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:04:37,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.67 vs. limit=22.5 2023-11-24 00:05:03,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.67 vs. limit=22.5 2023-11-24 00:05:12,901 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389500 2023-11-24 00:05:28,439 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.415e+01 9.088e+01 1.001e+02 1.234e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-24 00:05:30,846 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4750, loss[loss=0.07048, simple_loss=0.09776, pruned_loss=0.01075, audio_tagging_loss=0.01085, over 15748.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09078, pruned_loss=0.01325, audio_tagging_loss=0.009055, over 3047911.00 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:05:35,117 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.60 vs. limit=15.0 2023-11-24 00:05:38,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2596753.3333333335, ans=0.0 2023-11-24 00:05:58,171 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.93 vs. limit=15.0 2023-11-24 00:06:14,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389550 2023-11-24 00:06:21,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.29 vs. limit=10.0 2023-11-24 00:06:22,430 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0 2023-11-24 00:06:23,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2597020.0, ans=0.1 2023-11-24 00:06:35,118 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4800, loss[loss=0.08835, simple_loss=0.1115, pruned_loss=0.01968, audio_tagging_loss=0.01293, over 15442.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.091, pruned_loss=0.01331, audio_tagging_loss=0.009164, over 3048116.06 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:07:13,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2597286.6666666665, ans=0.125 2023-11-24 00:07:14,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2597286.6666666665, ans=0.2 2023-11-24 00:07:19,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389600 2023-11-24 00:07:20,723 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.99 vs. limit=22.5 2023-11-24 00:07:30,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2597353.3333333335, ans=0.125 2023-11-24 00:07:37,707 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.181e+01 8.337e+01 8.951e+01 9.611e+01 1.135e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-24 00:07:37,752 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4850, loss[loss=0.04571, simple_loss=0.06097, pruned_loss=0.005691, audio_tagging_loss=0.009538, over 14020.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.0907, pruned_loss=0.01329, audio_tagging_loss=0.009291, over 3043733.29 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:08:05,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2597553.3333333335, ans=0.125 2023-11-24 00:08:21,546 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389650 2023-11-24 00:08:25,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2597620.0, ans=0.1 2023-11-24 00:08:39,295 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4900, loss[loss=0.07098, simple_loss=0.09649, pruned_loss=0.01463, audio_tagging_loss=0.008108, over 15755.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09131, pruned_loss=0.0135, audio_tagging_loss=0.009212, over 3047521.95 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:08:45,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2597753.3333333335, ans=0.04949747468305833 2023-11-24 00:08:57,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2597820.0, ans=0.125 2023-11-24 00:09:08,884 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=17.92 vs. limit=22.5 2023-11-24 00:09:11,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2597886.6666666665, ans=0.0 2023-11-24 00:09:14,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2597886.6666666665, ans=0.0 2023-11-24 00:09:23,657 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389700 2023-11-24 00:09:27,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2597953.3333333335, ans=0.0 2023-11-24 00:09:43,036 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.493e+01 8.944e+01 9.853e+01 1.226e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-24 00:09:43,090 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 4950, loss[loss=0.06796, simple_loss=0.08769, pruned_loss=0.01398, audio_tagging_loss=0.01014, over 15073.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09127, pruned_loss=0.01339, audio_tagging_loss=0.009068, over 3048801.84 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:09:53,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2598086.6666666665, ans=0.95 2023-11-24 00:10:18,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2598286.6666666665, ans=0.0 2023-11-24 00:10:26,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389750 2023-11-24 00:10:45,885 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5000, loss[loss=0.0693, simple_loss=0.09771, pruned_loss=0.01357, audio_tagging_loss=0.006872, over 15892.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09071, pruned_loss=0.01344, audio_tagging_loss=0.008997, over 3043904.61 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:10:54,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2598420.0, ans=0.025 2023-11-24 00:10:59,510 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:11:10,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2598553.3333333335, ans=0.0 2023-11-24 00:11:19,868 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:11:22,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2598620.0, ans=0.0 2023-11-24 00:11:29,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389800 2023-11-24 00:11:33,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2598620.0, ans=0.0 2023-11-24 00:11:47,901 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.099e+01 8.775e+01 9.573e+01 1.290e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-24 00:11:47,946 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5050, loss[loss=0.05478, simple_loss=0.07309, pruned_loss=0.009622, audio_tagging_loss=0.008614, over 14821.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09123, pruned_loss=0.01338, audio_tagging_loss=0.008894, over 3044713.29 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:12:14,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2598886.6666666665, ans=0.125 2023-11-24 00:12:32,499 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389850 2023-11-24 00:12:36,158 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:12:37,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2599020.0, ans=0.1 2023-11-24 00:12:47,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2599020.0, ans=0.0 2023-11-24 00:12:48,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2599020.0, ans=0.125 2023-11-24 00:12:50,813 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5100, loss[loss=0.09055, simple_loss=0.1221, pruned_loss=0.02197, audio_tagging_loss=0.007522, over 15190.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09029, pruned_loss=0.01326, audio_tagging_loss=0.008769, over 3044370.79 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:13:06,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2599153.3333333335, ans=0.5 2023-11-24 00:13:34,799 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389900 2023-11-24 00:13:37,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2599286.6666666665, ans=0.0 2023-11-24 00:13:41,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2599353.3333333335, ans=0.125 2023-11-24 00:13:45,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2599353.3333333335, ans=0.2 2023-11-24 00:13:54,284 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.310e+01 9.019e+01 9.639e+01 1.666e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-24 00:13:54,329 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5150, loss[loss=0.07747, simple_loss=0.1159, pruned_loss=0.01454, audio_tagging_loss=0.004991, over 15422.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09075, pruned_loss=0.0133, audio_tagging_loss=0.008769, over 3043195.41 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:14:09,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2599486.6666666665, ans=0.0 2023-11-24 00:14:18,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2599553.3333333335, ans=0.0 2023-11-24 00:14:19,467 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:14:22,913 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.08 vs. limit=22.5 2023-11-24 00:14:33,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2599620.0, ans=0.0 2023-11-24 00:14:34,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2599620.0, ans=0.0 2023-11-24 00:14:38,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 389950 2023-11-24 00:14:56,522 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5200, loss[loss=0.05482, simple_loss=0.08222, pruned_loss=0.008398, audio_tagging_loss=0.005316, over 15013.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09114, pruned_loss=0.01337, audio_tagging_loss=0.008749, over 3042593.26 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:15:01,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2599753.3333333335, ans=0.0 2023-11-24 00:15:10,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2599820.0, ans=0.2 2023-11-24 00:15:23,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2599886.6666666665, ans=0.0 2023-11-24 00:15:41,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390000 2023-11-24 00:15:48,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2600020.0, ans=0.125 2023-11-24 00:16:00,320 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.701e+01 8.431e+01 8.968e+01 9.666e+01 1.343e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-24 00:16:00,386 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5250, loss[loss=0.04722, simple_loss=0.05756, pruned_loss=0.009124, audio_tagging_loss=0.009318, over 15588.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09113, pruned_loss=0.01338, audio_tagging_loss=0.008762, over 3055630.13 frames. ], batch size: 62, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:16:04,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2600086.6666666665, ans=0.09899494936611666 2023-11-24 00:16:22,452 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:16:36,002 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.67 vs. limit=22.5 2023-11-24 00:16:43,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390050 2023-11-24 00:17:03,026 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5300, loss[loss=0.07977, simple_loss=0.1234, pruned_loss=0.01303, audio_tagging_loss=0.005011, over 17490.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.0921, pruned_loss=0.01343, audio_tagging_loss=0.008663, over 3047496.24 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:17:07,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2600420.0, ans=0.0 2023-11-24 00:17:46,799 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390100 2023-11-24 00:18:00,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2600686.6666666665, ans=0.125 2023-11-24 00:18:04,809 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5350, loss[loss=0.04819, simple_loss=0.05838, pruned_loss=0.009024, audio_tagging_loss=0.009971, over 13912.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09171, pruned_loss=0.01344, audio_tagging_loss=0.008825, over 3048715.16 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:18:05,946 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.377e+01 8.739e+01 9.630e+01 1.201e+02, threshold=1.748e+02, percent-clipped=0.0 2023-11-24 00:18:14,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2600753.3333333335, ans=0.2 2023-11-24 00:18:17,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.19 vs. limit=10.0 2023-11-24 00:18:21,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2600820.0, ans=0.2 2023-11-24 00:18:23,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.66 vs. limit=15.0 2023-11-24 00:18:26,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2600820.0, ans=0.035 2023-11-24 00:18:48,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390150 2023-11-24 00:18:54,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.00 vs. limit=15.0 2023-11-24 00:19:02,060 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0 2023-11-24 00:19:06,221 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5400, loss[loss=0.05737, simple_loss=0.0642, pruned_loss=0.01082, audio_tagging_loss=0.01446, over 14998.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09227, pruned_loss=0.01355, audio_tagging_loss=0.008841, over 3051754.66 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:19:18,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2601153.3333333335, ans=0.0 2023-11-24 00:19:19,246 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.43 vs. limit=15.0 2023-11-24 00:19:50,062 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390200 2023-11-24 00:20:09,852 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5450, loss[loss=0.07718, simple_loss=0.1083, pruned_loss=0.01368, audio_tagging_loss=0.009342, over 15293.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09386, pruned_loss=0.01384, audio_tagging_loss=0.008875, over 3054965.49 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:20:10,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2601420.0, ans=0.0 2023-11-24 00:20:10,928 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.432e+01 8.413e+01 8.965e+01 9.765e+01 1.201e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 00:20:20,106 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.64 vs. limit=15.0 2023-11-24 00:20:34,108 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:20:41,690 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.70 vs. limit=22.5 2023-11-24 00:20:53,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390250 2023-11-24 00:21:11,501 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5500, loss[loss=0.07543, simple_loss=0.1018, pruned_loss=0.01586, audio_tagging_loss=0.008665, over 13828.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09334, pruned_loss=0.01374, audio_tagging_loss=0.008848, over 3058456.67 frames. ], batch size: 52, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:21:11,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2601753.3333333335, ans=0.125 2023-11-24 00:21:26,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2601820.0, ans=0.125 2023-11-24 00:21:27,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2601820.0, ans=0.2 2023-11-24 00:21:31,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2601820.0, ans=0.125 2023-11-24 00:21:43,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2601886.6666666665, ans=0.1 2023-11-24 00:21:45,206 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.26 vs. limit=22.5 2023-11-24 00:21:55,011 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390300 2023-11-24 00:22:01,921 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.70 vs. limit=15.0 2023-11-24 00:22:13,310 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5550, loss[loss=0.06407, simple_loss=0.08187, pruned_loss=0.01365, audio_tagging_loss=0.009482, over 15462.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09275, pruned_loss=0.01365, audio_tagging_loss=0.008933, over 3051117.55 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:22:14,448 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.193e+01 8.280e+01 9.193e+01 9.915e+01 1.422e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 00:22:15,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2602086.6666666665, ans=0.125 2023-11-24 00:22:57,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390350 2023-11-24 00:23:14,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2602353.3333333335, ans=0.125 2023-11-24 00:23:16,751 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5600, loss[loss=0.07914, simple_loss=0.115, pruned_loss=0.0156, audio_tagging_loss=0.006024, over 15559.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09376, pruned_loss=0.0138, audio_tagging_loss=0.009034, over 3058716.92 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:23:20,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2602420.0, ans=0.125 2023-11-24 00:23:49,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2602553.3333333335, ans=0.125 2023-11-24 00:23:59,842 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390400 2023-11-24 00:24:00,935 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 00:24:11,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2602686.6666666665, ans=0.125 2023-11-24 00:24:18,480 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5650, loss[loss=0.06196, simple_loss=0.07136, pruned_loss=0.01397, audio_tagging_loss=0.0123, over 15627.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09305, pruned_loss=0.01361, audio_tagging_loss=0.009097, over 3060008.75 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:24:19,626 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.276e+01 8.461e+01 8.950e+01 9.725e+01 1.309e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-24 00:24:19,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2602753.3333333335, ans=0.125 2023-11-24 00:24:41,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2602820.0, ans=0.125 2023-11-24 00:25:02,283 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390450 2023-11-24 00:25:05,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2602953.3333333335, ans=0.125 2023-11-24 00:25:20,367 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5700, loss[loss=0.06558, simple_loss=0.09687, pruned_loss=0.01106, audio_tagging_loss=0.006082, over 16447.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09267, pruned_loss=0.01344, audio_tagging_loss=0.009086, over 3058064.06 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:25:38,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.68 vs. limit=22.5 2023-11-24 00:25:49,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2603220.0, ans=0.125 2023-11-24 00:26:01,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2603286.6666666665, ans=0.125 2023-11-24 00:26:04,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390500 2023-11-24 00:26:04,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.46 vs. limit=15.0 2023-11-24 00:26:08,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2603353.3333333335, ans=0.2 2023-11-24 00:26:23,067 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5750, loss[loss=0.07797, simple_loss=0.1009, pruned_loss=0.01741, audio_tagging_loss=0.01011, over 15128.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09266, pruned_loss=0.01367, audio_tagging_loss=0.009072, over 3055056.72 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 32.0 2023-11-24 00:26:24,169 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.481e+01 9.036e+01 9.796e+01 1.280e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 00:26:33,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0 2023-11-24 00:26:37,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2603486.6666666665, ans=0.5 2023-11-24 00:26:55,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2603553.3333333335, ans=15.0 2023-11-24 00:27:00,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2603620.0, ans=0.09899494936611666 2023-11-24 00:27:05,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2603620.0, ans=0.0 2023-11-24 00:27:06,342 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390550 2023-11-24 00:27:15,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.32 vs. limit=22.5 2023-11-24 00:27:25,133 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5800, loss[loss=0.06978, simple_loss=0.08812, pruned_loss=0.01288, audio_tagging_loss=0.01284, over 14585.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.0926, pruned_loss=0.01374, audio_tagging_loss=0.009012, over 3049966.93 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 8.0 2023-11-24 00:27:38,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2603820.0, ans=0.5 2023-11-24 00:27:40,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2603820.0, ans=0.2 2023-11-24 00:27:43,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2603820.0, ans=0.125 2023-11-24 00:27:47,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2603820.0, ans=0.2 2023-11-24 00:27:53,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2603886.6666666665, ans=0.0 2023-11-24 00:28:08,518 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390600 2023-11-24 00:28:13,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2604020.0, ans=0.5 2023-11-24 00:28:15,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2604020.0, ans=0.2 2023-11-24 00:28:18,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2604020.0, ans=0.025 2023-11-24 00:28:26,605 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5850, loss[loss=0.06635, simple_loss=0.08681, pruned_loss=0.01537, audio_tagging_loss=0.007568, over 14712.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09185, pruned_loss=0.01376, audio_tagging_loss=0.008913, over 3043737.03 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 8.0 2023-11-24 00:28:30,555 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.379e+01 8.883e+01 9.652e+01 2.888e+02, threshold=1.777e+02, percent-clipped=1.0 2023-11-24 00:29:04,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2604286.6666666665, ans=0.125 2023-11-24 00:29:10,257 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390650 2023-11-24 00:29:27,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2604353.3333333335, ans=0.125 2023-11-24 00:29:29,341 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5900, loss[loss=0.07429, simple_loss=0.09447, pruned_loss=0.01808, audio_tagging_loss=0.008972, over 15127.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09153, pruned_loss=0.01362, audio_tagging_loss=0.008853, over 3037166.82 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 8.0 2023-11-24 00:29:35,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2604420.0, ans=0.0 2023-11-24 00:29:38,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2604420.0, ans=0.0 2023-11-24 00:29:41,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2604486.6666666665, ans=0.125 2023-11-24 00:29:51,385 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.34 vs. limit=15.0 2023-11-24 00:29:52,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2604486.6666666665, ans=0.125 2023-11-24 00:30:04,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2604553.3333333335, ans=0.0 2023-11-24 00:30:05,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2604620.0, ans=0.125 2023-11-24 00:30:05,670 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.65 vs. limit=15.0 2023-11-24 00:30:13,468 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390700 2023-11-24 00:30:19,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2604686.6666666665, ans=0.125 2023-11-24 00:30:24,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2604686.6666666665, ans=0.1 2023-11-24 00:30:31,518 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.18 vs. limit=22.5 2023-11-24 00:30:32,176 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 5950, loss[loss=0.06538, simple_loss=0.09417, pruned_loss=0.01129, audio_tagging_loss=0.007013, over 16168.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09271, pruned_loss=0.01371, audio_tagging_loss=0.008724, over 3044232.28 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 8.0 2023-11-24 00:30:35,908 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.451e+01 9.116e+01 9.981e+01 1.115e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 00:30:41,527 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.74 vs. limit=15.0 2023-11-24 00:30:53,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2604820.0, ans=0.125 2023-11-24 00:31:04,746 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0 2023-11-24 00:31:09,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2604953.3333333335, ans=0.125 2023-11-24 00:31:11,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2604953.3333333335, ans=0.025 2023-11-24 00:31:15,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390750 2023-11-24 00:31:22,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2605020.0, ans=15.0 2023-11-24 00:31:33,666 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6000, loss[loss=0.07049, simple_loss=0.09691, pruned_loss=0.01389, audio_tagging_loss=0.008143, over 15597.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09181, pruned_loss=0.01361, audio_tagging_loss=0.008846, over 3040854.61 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:31:33,666 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 00:31:59,618 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.5779, 3.5575, 3.9014, 3.3171], device='cuda:3') 2023-11-24 00:31:59,684 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9738, 3.2363, 2.8777, 3.0887, 3.4138, 2.7744, 3.4070, 2.6932], device='cuda:3') 2023-11-24 00:32:09,894 INFO [train_asr.py:1253] (3/4) Epoch 33, validation: loss=0.05769, simple_loss=0.05098, pruned_loss=0.005124, audio_tagging_loss=0.02707, over 4681554.00 frames. 2023-11-24 00:32:09,895 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 00:32:21,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2605153.3333333335, ans=0.125 2023-11-24 00:32:41,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2605220.0, ans=0.0 2023-11-24 00:32:45,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2605286.6666666665, ans=0.125 2023-11-24 00:32:52,251 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390800 2023-11-24 00:32:55,887 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 00:33:11,947 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6050, loss[loss=0.04478, simple_loss=0.05578, pruned_loss=0.006524, audio_tagging_loss=0.01037, over 15481.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09163, pruned_loss=0.01353, audio_tagging_loss=0.008796, over 3043042.27 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:33:15,470 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.390e+01 8.432e+01 8.885e+01 9.962e+01 1.302e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-24 00:33:28,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2605486.6666666665, ans=0.2 2023-11-24 00:33:38,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2605553.3333333335, ans=0.125 2023-11-24 00:33:44,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2605553.3333333335, ans=0.95 2023-11-24 00:33:51,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2605620.0, ans=0.0 2023-11-24 00:33:53,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2605620.0, ans=0.0 2023-11-24 00:33:54,996 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390850 2023-11-24 00:33:55,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2605620.0, ans=0.125 2023-11-24 00:34:05,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2605686.6666666665, ans=0.1 2023-11-24 00:34:09,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2605686.6666666665, ans=0.125 2023-11-24 00:34:12,716 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6100, loss[loss=0.06728, simple_loss=0.08758, pruned_loss=0.0141, audio_tagging_loss=0.00939, over 14685.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09308, pruned_loss=0.0137, audio_tagging_loss=0.008755, over 3050507.09 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:34:35,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2605820.0, ans=0.07 2023-11-24 00:34:43,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2605886.6666666665, ans=0.2 2023-11-24 00:34:44,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.74 vs. limit=15.0 2023-11-24 00:34:49,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2605953.3333333335, ans=0.125 2023-11-24 00:34:56,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390900 2023-11-24 00:35:03,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2606020.0, ans=0.0 2023-11-24 00:35:06,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2606020.0, ans=0.0 2023-11-24 00:35:13,050 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.36 vs. limit=6.0 2023-11-24 00:35:14,634 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6150, loss[loss=0.06776, simple_loss=0.08775, pruned_loss=0.01476, audio_tagging_loss=0.009117, over 16234.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09345, pruned_loss=0.01374, audio_tagging_loss=0.008767, over 3044865.18 frames. ], batch size: 63, lr: 2.06e-03, grad_scale: 16.0 2023-11-24 00:35:19,342 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.208e+01 8.886e+01 9.597e+01 1.503e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-24 00:35:26,076 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.07 vs. limit=15.0 2023-11-24 00:35:50,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2606286.6666666665, ans=0.0 2023-11-24 00:35:57,639 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 390950 2023-11-24 00:36:09,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2606353.3333333335, ans=0.0 2023-11-24 00:36:17,588 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6200, loss[loss=0.04835, simple_loss=0.06093, pruned_loss=0.007182, audio_tagging_loss=0.0107, over 15624.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09332, pruned_loss=0.01387, audio_tagging_loss=0.00886, over 3036853.11 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:36:40,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2606553.3333333335, ans=0.2 2023-11-24 00:36:53,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2606553.3333333335, ans=0.125 2023-11-24 00:36:54,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.11 vs. limit=15.0 2023-11-24 00:37:02,089 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391000 2023-11-24 00:37:16,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2606686.6666666665, ans=0.1 2023-11-24 00:37:19,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2606753.3333333335, ans=0.5 2023-11-24 00:37:20,041 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6250, loss[loss=0.06241, simple_loss=0.087, pruned_loss=0.01157, audio_tagging_loss=0.007346, over 15886.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09131, pruned_loss=0.01364, audio_tagging_loss=0.008992, over 3039852.35 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:37:23,527 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.315e+01 8.981e+01 9.556e+01 1.931e+02, threshold=1.796e+02, percent-clipped=1.0 2023-11-24 00:37:25,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2606753.3333333335, ans=0.0 2023-11-24 00:37:26,451 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.62 vs. limit=15.0 2023-11-24 00:38:04,298 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391050 2023-11-24 00:38:18,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2607020.0, ans=0.1 2023-11-24 00:38:22,590 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6300, loss[loss=0.08113, simple_loss=0.109, pruned_loss=0.01708, audio_tagging_loss=0.009556, over 15562.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09286, pruned_loss=0.0138, audio_tagging_loss=0.008984, over 3044337.34 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:38:25,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2607086.6666666665, ans=0.0 2023-11-24 00:38:30,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2607086.6666666665, ans=0.125 2023-11-24 00:38:53,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2607220.0, ans=0.2 2023-11-24 00:39:02,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=2607286.6666666665, ans=0.02 2023-11-24 00:39:05,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391100 2023-11-24 00:39:25,694 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6350, loss[loss=0.08273, simple_loss=0.1218, pruned_loss=0.01449, audio_tagging_loss=0.007329, over 15924.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.0925, pruned_loss=0.01359, audio_tagging_loss=0.009089, over 3044393.17 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:39:29,227 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.017e+01 8.323e+01 9.041e+01 9.620e+01 1.215e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 00:39:40,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2607486.6666666665, ans=0.125 2023-11-24 00:39:49,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2607553.3333333335, ans=0.125 2023-11-24 00:39:50,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2607553.3333333335, ans=0.2 2023-11-24 00:39:52,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2607553.3333333335, ans=0.125 2023-11-24 00:40:09,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391150 2023-11-24 00:40:16,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2607686.6666666665, ans=0.05 2023-11-24 00:40:18,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2607686.6666666665, ans=0.125 2023-11-24 00:40:19,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2607686.6666666665, ans=0.0 2023-11-24 00:40:19,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2607686.6666666665, ans=0.2 2023-11-24 00:40:25,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.24 vs. limit=22.5 2023-11-24 00:40:27,284 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6400, loss[loss=0.06266, simple_loss=0.07619, pruned_loss=0.01237, audio_tagging_loss=0.01219, over 14915.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09212, pruned_loss=0.01347, audio_tagging_loss=0.009223, over 3050790.92 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 32.0 2023-11-24 00:40:31,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2607753.3333333335, ans=0.1 2023-11-24 00:41:05,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2607953.3333333335, ans=0.5 2023-11-24 00:41:11,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391200 2023-11-24 00:41:17,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2608020.0, ans=0.05 2023-11-24 00:41:20,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0 2023-11-24 00:41:21,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2608020.0, ans=0.125 2023-11-24 00:41:26,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2608020.0, ans=0.2 2023-11-24 00:41:29,312 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6450, loss[loss=0.05725, simple_loss=0.07968, pruned_loss=0.01027, audio_tagging_loss=0.007137, over 15856.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09204, pruned_loss=0.01344, audio_tagging_loss=0.009282, over 3049133.32 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:41:30,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2608086.6666666665, ans=0.0 2023-11-24 00:41:34,712 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.349e+01 8.985e+01 9.691e+01 1.713e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-24 00:41:39,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2608086.6666666665, ans=0.125 2023-11-24 00:42:01,491 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.15 vs. limit=12.0 2023-11-24 00:42:02,786 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.37 vs. limit=15.0 2023-11-24 00:42:07,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2608286.6666666665, ans=0.09899494936611666 2023-11-24 00:42:14,127 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391250 2023-11-24 00:42:33,540 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6500, loss[loss=0.03941, simple_loss=0.05109, pruned_loss=0.004777, audio_tagging_loss=0.009088, over 13522.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09176, pruned_loss=0.01343, audio_tagging_loss=0.009113, over 3040129.79 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:42:36,802 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.82 vs. limit=15.0 2023-11-24 00:42:41,991 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.79 vs. limit=6.0 2023-11-24 00:43:17,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391300 2023-11-24 00:43:22,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2608686.6666666665, ans=0.125 2023-11-24 00:43:35,734 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6550, loss[loss=0.06508, simple_loss=0.09162, pruned_loss=0.01193, audio_tagging_loss=0.007338, over 14977.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09167, pruned_loss=0.01337, audio_tagging_loss=0.009021, over 3044290.11 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:43:39,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.07 vs. limit=22.5 2023-11-24 00:43:40,534 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.079e+01 8.397e+01 9.072e+01 9.736e+01 1.207e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 00:43:45,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2608753.3333333335, ans=0.1 2023-11-24 00:44:00,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2608886.6666666665, ans=0.125 2023-11-24 00:44:00,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2608886.6666666665, ans=0.125 2023-11-24 00:44:10,468 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.50 vs. limit=15.0 2023-11-24 00:44:12,254 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:44:18,619 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.94 vs. limit=22.5 2023-11-24 00:44:19,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391350 2023-11-24 00:44:32,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2609020.0, ans=0.0 2023-11-24 00:44:37,632 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6600, loss[loss=0.06713, simple_loss=0.09516, pruned_loss=0.01194, audio_tagging_loss=0.007604, over 15976.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09099, pruned_loss=0.01323, audio_tagging_loss=0.008979, over 3043260.35 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:45:03,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.73 vs. limit=15.0 2023-11-24 00:45:14,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2609286.6666666665, ans=0.0 2023-11-24 00:45:18,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2609286.6666666665, ans=0.2 2023-11-24 00:45:21,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391400 2023-11-24 00:45:37,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2609353.3333333335, ans=0.125 2023-11-24 00:45:41,432 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6650, loss[loss=0.05842, simple_loss=0.07758, pruned_loss=0.01085, audio_tagging_loss=0.00878, over 14668.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09042, pruned_loss=0.01322, audio_tagging_loss=0.009018, over 3037424.63 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:45:46,130 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.357e+01 8.351e+01 8.932e+01 9.624e+01 1.119e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 00:46:12,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2609553.3333333335, ans=0.125 2023-11-24 00:46:25,037 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391450 2023-11-24 00:46:42,719 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6700, loss[loss=0.07237, simple_loss=0.08739, pruned_loss=0.0169, audio_tagging_loss=0.01178, over 14707.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09077, pruned_loss=0.0133, audio_tagging_loss=0.009004, over 3046310.36 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:47:06,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2609886.6666666665, ans=0.1 2023-11-24 00:47:14,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0 2023-11-24 00:47:17,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2609886.6666666665, ans=0.0 2023-11-24 00:47:20,360 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.44 vs. limit=6.0 2023-11-24 00:47:22,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2609953.3333333335, ans=15.0 2023-11-24 00:47:26,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391500 2023-11-24 00:47:45,022 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6750, loss[loss=0.06371, simple_loss=0.08606, pruned_loss=0.01262, audio_tagging_loss=0.008055, over 16356.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09082, pruned_loss=0.01319, audio_tagging_loss=0.008993, over 3046950.02 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:47:48,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2610086.6666666665, ans=0.0 2023-11-24 00:47:49,706 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.433e+01 8.920e+01 9.710e+01 1.340e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-24 00:48:05,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2610153.3333333335, ans=0.07 2023-11-24 00:48:15,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2610220.0, ans=0.125 2023-11-24 00:48:28,858 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391550 2023-11-24 00:48:36,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2610353.3333333335, ans=0.0 2023-11-24 00:48:36,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2610353.3333333335, ans=0.2 2023-11-24 00:48:36,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0 2023-11-24 00:48:48,399 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6800, loss[loss=0.07821, simple_loss=0.1078, pruned_loss=0.01518, audio_tagging_loss=0.009141, over 14781.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09176, pruned_loss=0.01325, audio_tagging_loss=0.008969, over 3042120.89 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 32.0 2023-11-24 00:49:15,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2610553.3333333335, ans=0.125 2023-11-24 00:49:25,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2610620.0, ans=0.125 2023-11-24 00:49:31,709 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391600 2023-11-24 00:49:31,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2610620.0, ans=0.0 2023-11-24 00:49:47,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.97 vs. limit=15.0 2023-11-24 00:49:50,171 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6850, loss[loss=0.05406, simple_loss=0.07384, pruned_loss=0.0101, audio_tagging_loss=0.007035, over 16091.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09101, pruned_loss=0.0131, audio_tagging_loss=0.008935, over 3041945.86 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:49:56,140 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.434e+01 9.033e+01 9.858e+01 1.368e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 00:49:59,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2610753.3333333335, ans=0.125 2023-11-24 00:50:04,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2610820.0, ans=0.125 2023-11-24 00:50:04,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2610820.0, ans=22.5 2023-11-24 00:50:24,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2610886.6666666665, ans=0.125 2023-11-24 00:50:34,021 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391650 2023-11-24 00:50:41,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.63 vs. limit=10.0 2023-11-24 00:50:49,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=2611020.0, ans=0.02 2023-11-24 00:50:52,595 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6900, loss[loss=0.05632, simple_loss=0.07307, pruned_loss=0.01047, audio_tagging_loss=0.009313, over 16127.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09052, pruned_loss=0.01306, audio_tagging_loss=0.008914, over 3045329.81 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:51:07,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2611153.3333333335, ans=0.0 2023-11-24 00:51:10,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2611153.3333333335, ans=0.1 2023-11-24 00:51:15,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2611153.3333333335, ans=0.0 2023-11-24 00:51:23,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2611220.0, ans=0.1 2023-11-24 00:51:36,138 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391700 2023-11-24 00:51:41,370 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 00:51:43,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2611353.3333333335, ans=0.125 2023-11-24 00:51:45,789 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0 2023-11-24 00:51:55,496 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 6950, loss[loss=0.05623, simple_loss=0.07061, pruned_loss=0.008879, audio_tagging_loss=0.01205, over 14929.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09104, pruned_loss=0.01332, audio_tagging_loss=0.008967, over 3045470.62 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 00:51:56,134 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.94 vs. limit=15.0 2023-11-24 00:52:03,206 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.470e+01 9.036e+01 9.958e+01 1.197e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 00:52:09,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2611486.6666666665, ans=0.125 2023-11-24 00:52:10,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2611486.6666666665, ans=0.125 2023-11-24 00:52:18,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2611553.3333333335, ans=0.0 2023-11-24 00:52:25,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2611553.3333333335, ans=0.0 2023-11-24 00:52:27,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0 2023-11-24 00:52:38,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391750 2023-11-24 00:52:51,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2611686.6666666665, ans=0.125 2023-11-24 00:52:57,556 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7000, loss[loss=0.08085, simple_loss=0.1057, pruned_loss=0.02119, audio_tagging_loss=0.006823, over 15204.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09111, pruned_loss=0.01337, audio_tagging_loss=0.008946, over 3046986.98 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 00:53:03,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2611753.3333333335, ans=0.0 2023-11-24 00:53:22,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2611886.6666666665, ans=0.125 2023-11-24 00:53:25,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.37 vs. limit=15.0 2023-11-24 00:53:28,861 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:53:32,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=12.0 2023-11-24 00:53:41,588 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391800 2023-11-24 00:53:41,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2611953.3333333335, ans=0.0 2023-11-24 00:53:41,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2611953.3333333335, ans=0.1 2023-11-24 00:53:44,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2611953.3333333335, ans=0.125 2023-11-24 00:53:59,714 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7050, loss[loss=0.05718, simple_loss=0.07089, pruned_loss=0.01344, audio_tagging_loss=0.008293, over 13913.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09062, pruned_loss=0.01343, audio_tagging_loss=0.008995, over 3034068.31 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 00:54:07,536 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.402e+01 9.074e+01 1.006e+02 1.351e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-24 00:54:09,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2612086.6666666665, ans=0.2 2023-11-24 00:54:20,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2612153.3333333335, ans=0.125 2023-11-24 00:54:43,881 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391850 2023-11-24 00:55:02,811 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7100, loss[loss=0.05357, simple_loss=0.06788, pruned_loss=0.009022, audio_tagging_loss=0.01061, over 15064.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09152, pruned_loss=0.01345, audio_tagging_loss=0.00903, over 3037063.62 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 00:55:05,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2612420.0, ans=0.2 2023-11-24 00:55:07,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2612420.0, ans=0.0 2023-11-24 00:55:11,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2612420.0, ans=0.2 2023-11-24 00:55:11,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.85 vs. limit=15.0 2023-11-24 00:55:45,727 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391900 2023-11-24 00:55:50,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=15.0 2023-11-24 00:56:05,204 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7150, loss[loss=0.05505, simple_loss=0.06397, pruned_loss=0.009359, audio_tagging_loss=0.0137, over 14922.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09103, pruned_loss=0.01339, audio_tagging_loss=0.00913, over 3039489.33 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 00:56:12,275 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.300e+01 8.358e+01 9.047e+01 9.923e+01 1.303e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-24 00:56:38,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2612886.6666666665, ans=0.2 2023-11-24 00:56:47,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2612953.3333333335, ans=0.125 2023-11-24 00:56:49,177 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 391950 2023-11-24 00:57:06,669 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7200, loss[loss=0.06354, simple_loss=0.08243, pruned_loss=0.01022, audio_tagging_loss=0.0121, over 15710.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09048, pruned_loss=0.01332, audio_tagging_loss=0.009292, over 3041355.83 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:57:20,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2613153.3333333335, ans=0.125 2023-11-24 00:57:25,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2613153.3333333335, ans=0.125 2023-11-24 00:57:29,964 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.22 vs. limit=6.0 2023-11-24 00:57:46,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2613286.6666666665, ans=0.2 2023-11-24 00:57:49,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2613286.6666666665, ans=0.035 2023-11-24 00:57:49,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2613286.6666666665, ans=0.0 2023-11-24 00:57:50,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392000 2023-11-24 00:58:12,148 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7250, loss[loss=0.06922, simple_loss=0.09403, pruned_loss=0.01239, audio_tagging_loss=0.009816, over 15027.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09015, pruned_loss=0.01328, audio_tagging_loss=0.009371, over 3036200.02 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:58:20,981 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.420e+01 8.844e+01 9.320e+01 1.006e+02 1.273e+02, threshold=1.864e+02, percent-clipped=0.0 2023-11-24 00:58:41,250 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 00:58:55,269 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392050 2023-11-24 00:59:15,271 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7300, loss[loss=0.08538, simple_loss=0.1213, pruned_loss=0.01592, audio_tagging_loss=0.008798, over 16102.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09008, pruned_loss=0.01343, audio_tagging_loss=0.009415, over 3034788.33 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 00:59:19,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2613753.3333333335, ans=0.5 2023-11-24 00:59:31,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.07 vs. limit=15.0 2023-11-24 00:59:36,174 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.40 vs. limit=15.0 2023-11-24 00:59:44,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.88 vs. limit=22.5 2023-11-24 00:59:58,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392100 2023-11-24 01:00:07,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2614020.0, ans=0.125 2023-11-24 01:00:15,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2614086.6666666665, ans=0.125 2023-11-24 01:00:16,221 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7350, loss[loss=0.06911, simple_loss=0.09049, pruned_loss=0.01486, audio_tagging_loss=0.008998, over 14708.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09071, pruned_loss=0.01354, audio_tagging_loss=0.00916, over 3033760.00 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:00:17,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2614086.6666666665, ans=0.0 2023-11-24 01:00:18,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2614086.6666666665, ans=0.125 2023-11-24 01:00:20,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2614086.6666666665, ans=0.125 2023-11-24 01:00:23,328 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 8.518e+01 9.030e+01 9.706e+01 1.263e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 01:00:25,497 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.89 vs. limit=15.0 2023-11-24 01:00:26,139 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:00:47,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2614220.0, ans=0.0 2023-11-24 01:00:51,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2614220.0, ans=0.0 2023-11-24 01:00:52,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2614220.0, ans=0.1 2023-11-24 01:00:55,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2614286.6666666665, ans=0.2 2023-11-24 01:01:00,289 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392150 2023-11-24 01:01:08,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2614353.3333333335, ans=0.2 2023-11-24 01:01:18,078 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7400, loss[loss=0.08749, simple_loss=0.1162, pruned_loss=0.0231, audio_tagging_loss=0.006283, over 14674.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09106, pruned_loss=0.01351, audio_tagging_loss=0.009002, over 3034711.25 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:01:31,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2614486.6666666665, ans=0.125 2023-11-24 01:01:46,687 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:01:46,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2614553.3333333335, ans=0.2 2023-11-24 01:01:49,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2614553.3333333335, ans=0.04949747468305833 2023-11-24 01:02:02,058 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392200 2023-11-24 01:02:08,495 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:02:21,667 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7450, loss[loss=0.04696, simple_loss=0.0571, pruned_loss=0.007971, audio_tagging_loss=0.01044, over 16307.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09106, pruned_loss=0.01356, audio_tagging_loss=0.009005, over 3042486.85 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:02:24,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.16 vs. limit=15.0 2023-11-24 01:02:25,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2614753.3333333335, ans=0.1 2023-11-24 01:02:28,722 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.459e+01 9.049e+01 9.753e+01 1.412e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 01:02:32,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2614820.0, ans=0.0 2023-11-24 01:02:43,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2614820.0, ans=0.0 2023-11-24 01:02:44,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2614886.6666666665, ans=0.125 2023-11-24 01:02:49,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2614886.6666666665, ans=0.125 2023-11-24 01:03:05,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392250 2023-11-24 01:03:09,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2614953.3333333335, ans=0.2 2023-11-24 01:03:15,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2615020.0, ans=0.125 2023-11-24 01:03:23,817 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7500, loss[loss=0.07365, simple_loss=0.09747, pruned_loss=0.01679, audio_tagging_loss=0.008126, over 15713.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09144, pruned_loss=0.01359, audio_tagging_loss=0.008959, over 3044457.24 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:03:38,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2615153.3333333335, ans=0.125 2023-11-24 01:03:44,810 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.44 vs. limit=15.0 2023-11-24 01:04:07,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392300 2023-11-24 01:04:16,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2615353.3333333335, ans=0.0 2023-11-24 01:04:17,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2615353.3333333335, ans=0.0 2023-11-24 01:04:23,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2615353.3333333335, ans=0.5 2023-11-24 01:04:25,650 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7550, loss[loss=0.08002, simple_loss=0.1054, pruned_loss=0.01789, audio_tagging_loss=0.009422, over 16059.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09195, pruned_loss=0.01387, audio_tagging_loss=0.008907, over 3048767.31 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:04:33,431 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.602e+01 9.110e+01 9.644e+01 1.333e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-24 01:04:40,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.43 vs. limit=15.0 2023-11-24 01:04:56,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2615553.3333333335, ans=0.1 2023-11-24 01:05:04,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2615620.0, ans=0.125 2023-11-24 01:05:09,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392350 2023-11-24 01:05:29,045 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7600, loss[loss=0.05087, simple_loss=0.06936, pruned_loss=0.007545, audio_tagging_loss=0.008646, over 15042.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09155, pruned_loss=0.01367, audio_tagging_loss=0.008934, over 3049333.02 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 32.0 2023-11-24 01:05:44,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2615820.0, ans=0.0 2023-11-24 01:05:54,120 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=15.0 2023-11-24 01:06:08,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2615953.3333333335, ans=0.0 2023-11-24 01:06:12,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392400 2023-11-24 01:06:26,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2616020.0, ans=0.125 2023-11-24 01:06:31,608 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7650, loss[loss=0.08598, simple_loss=0.1171, pruned_loss=0.01911, audio_tagging_loss=0.008294, over 15084.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09149, pruned_loss=0.01377, audio_tagging_loss=0.008986, over 3048061.54 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:06:39,748 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.599e+01 8.392e+01 8.875e+01 9.417e+01 1.557e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-24 01:06:47,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2616153.3333333335, ans=0.04949747468305833 2023-11-24 01:07:13,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2616286.6666666665, ans=0.125 2023-11-24 01:07:15,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392450 2023-11-24 01:07:20,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2616353.3333333335, ans=0.125 2023-11-24 01:07:33,372 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7700, loss[loss=0.0648, simple_loss=0.07924, pruned_loss=0.01216, audio_tagging_loss=0.01302, over 14651.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09229, pruned_loss=0.01397, audio_tagging_loss=0.008906, over 3047411.77 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:08:13,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2616620.0, ans=0.125 2023-11-24 01:08:17,289 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392500 2023-11-24 01:08:24,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2616686.6666666665, ans=0.125 2023-11-24 01:08:36,713 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7750, loss[loss=0.08096, simple_loss=0.1059, pruned_loss=0.02205, audio_tagging_loss=0.005953, over 14488.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09233, pruned_loss=0.01384, audio_tagging_loss=0.008915, over 3042351.32 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:08:37,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2616753.3333333335, ans=0.09899494936611666 2023-11-24 01:08:45,025 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.197e+01 8.416e+01 8.954e+01 9.884e+01 1.252e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-24 01:08:55,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2616820.0, ans=0.2 2023-11-24 01:09:01,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2616886.6666666665, ans=0.1 2023-11-24 01:09:07,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2616886.6666666665, ans=10.0 2023-11-24 01:09:10,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2616886.6666666665, ans=0.05 2023-11-24 01:09:12,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2616953.3333333335, ans=0.1 2023-11-24 01:09:12,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.02 vs. limit=12.0 2023-11-24 01:09:19,628 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392550 2023-11-24 01:09:19,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2616953.3333333335, ans=0.0 2023-11-24 01:09:38,499 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7800, loss[loss=0.07949, simple_loss=0.1057, pruned_loss=0.0181, audio_tagging_loss=0.00851, over 15644.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09217, pruned_loss=0.01394, audio_tagging_loss=0.008969, over 3037489.74 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:09:54,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2617153.3333333335, ans=0.125 2023-11-24 01:09:59,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2617153.3333333335, ans=0.125 2023-11-24 01:10:05,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2617220.0, ans=0.1 2023-11-24 01:10:22,325 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392600 2023-11-24 01:10:23,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2617286.6666666665, ans=0.125 2023-11-24 01:10:32,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2617353.3333333335, ans=0.125 2023-11-24 01:10:39,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2617353.3333333335, ans=0.2 2023-11-24 01:10:41,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.57 vs. limit=15.0 2023-11-24 01:10:41,568 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7850, loss[loss=0.05906, simple_loss=0.06796, pruned_loss=0.01342, audio_tagging_loss=0.01166, over 15091.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09199, pruned_loss=0.01393, audio_tagging_loss=0.009051, over 3038197.57 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:10:49,996 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.631e+01 9.206e+01 9.904e+01 1.275e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-24 01:11:25,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392650 2023-11-24 01:11:43,260 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7900, loss[loss=0.06994, simple_loss=0.1003, pruned_loss=0.01254, audio_tagging_loss=0.007227, over 14751.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09148, pruned_loss=0.01363, audio_tagging_loss=0.009136, over 3039784.72 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:11:46,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2617753.3333333335, ans=0.2 2023-11-24 01:11:48,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2617753.3333333335, ans=0.2 2023-11-24 01:12:11,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.66 vs. limit=12.0 2023-11-24 01:12:16,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2617886.6666666665, ans=0.2 2023-11-24 01:12:26,906 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392700 2023-11-24 01:12:34,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2618020.0, ans=0.2 2023-11-24 01:12:46,555 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 7950, loss[loss=0.0602, simple_loss=0.08159, pruned_loss=0.01049, audio_tagging_loss=0.008917, over 14247.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09083, pruned_loss=0.01359, audio_tagging_loss=0.009172, over 3047120.63 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:12:54,801 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.531e+01 8.932e+01 9.571e+01 1.197e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 01:13:03,668 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 01:13:05,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.01 vs. limit=22.5 2023-11-24 01:13:23,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.80 vs. limit=6.0 2023-11-24 01:13:30,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392750 2023-11-24 01:13:32,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2618286.6666666665, ans=0.125 2023-11-24 01:13:37,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2618353.3333333335, ans=0.1 2023-11-24 01:13:47,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2618420.0, ans=0.2 2023-11-24 01:13:48,831 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8000, loss[loss=0.04981, simple_loss=0.06088, pruned_loss=0.009505, audio_tagging_loss=0.009868, over 14710.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.08973, pruned_loss=0.01325, audio_tagging_loss=0.00921, over 3049614.84 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:13:53,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2618420.0, ans=0.0 2023-11-24 01:14:32,139 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392800 2023-11-24 01:14:33,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2618620.0, ans=0.04949747468305833 2023-11-24 01:14:35,323 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.49 vs. limit=15.0 2023-11-24 01:14:47,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2618686.6666666665, ans=0.2 2023-11-24 01:14:50,744 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8050, loss[loss=0.05975, simple_loss=0.07856, pruned_loss=0.01192, audio_tagging_loss=0.008558, over 15601.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.08912, pruned_loss=0.01302, audio_tagging_loss=0.009343, over 3050380.91 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:14:54,712 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.19 vs. limit=12.0 2023-11-24 01:15:01,891 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.420e+01 8.885e+01 9.707e+01 1.213e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-24 01:15:03,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2618820.0, ans=0.125 2023-11-24 01:15:06,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2618820.0, ans=0.05 2023-11-24 01:15:08,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2618820.0, ans=0.2 2023-11-24 01:15:09,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2618820.0, ans=0.025 2023-11-24 01:15:11,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2618820.0, ans=0.125 2023-11-24 01:15:16,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2618886.6666666665, ans=0.0 2023-11-24 01:15:23,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2618886.6666666665, ans=0.09899494936611666 2023-11-24 01:15:34,921 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392850 2023-11-24 01:15:35,454 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.35 vs. limit=22.5 2023-11-24 01:15:37,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2618953.3333333335, ans=0.125 2023-11-24 01:15:54,261 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8100, loss[loss=0.04765, simple_loss=0.06375, pruned_loss=0.00497, audio_tagging_loss=0.0108, over 14628.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.08873, pruned_loss=0.01298, audio_tagging_loss=0.009283, over 3046702.53 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:15:54,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2619086.6666666665, ans=0.125 2023-11-24 01:16:08,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2619153.3333333335, ans=0.125 2023-11-24 01:16:23,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2619220.0, ans=0.125 2023-11-24 01:16:27,802 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:16:38,366 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392900 2023-11-24 01:16:38,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2619286.6666666665, ans=0.2 2023-11-24 01:16:47,211 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=5.59 vs. limit=15.0 2023-11-24 01:16:50,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2619353.3333333335, ans=0.05 2023-11-24 01:16:56,158 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8150, loss[loss=0.05563, simple_loss=0.06746, pruned_loss=0.01301, audio_tagging_loss=0.008888, over 16065.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.08879, pruned_loss=0.01304, audio_tagging_loss=0.00911, over 3048669.89 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:17:03,233 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.79 vs. limit=10.0 2023-11-24 01:17:06,598 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.428e+01 9.309e+01 1.006e+02 1.505e+02, threshold=1.862e+02, percent-clipped=0.0 2023-11-24 01:17:09,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2619486.6666666665, ans=0.125 2023-11-24 01:17:40,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 392950 2023-11-24 01:17:50,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2619686.6666666665, ans=0.2 2023-11-24 01:17:56,237 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.98 vs. limit=10.0 2023-11-24 01:17:58,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2619753.3333333335, ans=0.125 2023-11-24 01:17:59,137 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8200, loss[loss=0.06091, simple_loss=0.07962, pruned_loss=0.01183, audio_tagging_loss=0.009271, over 15361.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.08985, pruned_loss=0.01312, audio_tagging_loss=0.008912, over 3052645.63 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:18:01,538 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 01:18:03,928 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.81 vs. limit=15.0 2023-11-24 01:18:05,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.11 vs. limit=12.0 2023-11-24 01:18:43,391 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393000 2023-11-24 01:18:47,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2619953.3333333335, ans=0.2 2023-11-24 01:18:48,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2620020.0, ans=0.0 2023-11-24 01:19:03,401 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8250, loss[loss=0.07258, simple_loss=0.09558, pruned_loss=0.01681, audio_tagging_loss=0.007975, over 15001.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09047, pruned_loss=0.01322, audio_tagging_loss=0.00888, over 3050065.10 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:19:04,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2620086.6666666665, ans=0.125 2023-11-24 01:19:10,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2620086.6666666665, ans=0.0 2023-11-24 01:19:13,054 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.535e+01 8.227e+01 9.050e+01 1.002e+02 1.257e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 01:19:18,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2620153.3333333335, ans=0.035 2023-11-24 01:19:28,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2620220.0, ans=0.125 2023-11-24 01:19:44,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2620286.6666666665, ans=0.2 2023-11-24 01:19:46,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2620286.6666666665, ans=0.125 2023-11-24 01:19:47,625 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393050 2023-11-24 01:20:05,499 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8300, loss[loss=0.05347, simple_loss=0.07393, pruned_loss=0.008861, audio_tagging_loss=0.007638, over 15296.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09016, pruned_loss=0.01318, audio_tagging_loss=0.008908, over 3053696.05 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:20:18,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2620486.6666666665, ans=0.125 2023-11-24 01:20:20,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2620486.6666666665, ans=0.0 2023-11-24 01:20:28,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.27 vs. limit=10.0 2023-11-24 01:20:29,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.13 vs. limit=22.5 2023-11-24 01:20:43,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2620620.0, ans=0.05 2023-11-24 01:20:49,954 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393100 2023-11-24 01:20:59,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=2620686.6666666665, ans=0.2 2023-11-24 01:21:02,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2620686.6666666665, ans=0.1 2023-11-24 01:21:05,735 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.11 vs. limit=22.5 2023-11-24 01:21:07,463 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8350, loss[loss=0.07297, simple_loss=0.0944, pruned_loss=0.01873, audio_tagging_loss=0.007036, over 14050.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.08984, pruned_loss=0.0132, audio_tagging_loss=0.008865, over 3054325.34 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:21:16,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2620753.3333333335, ans=0.125 2023-11-24 01:21:19,990 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.568e+01 9.324e+01 1.020e+02 1.487e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 01:21:20,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2620820.0, ans=0.0 2023-11-24 01:21:23,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2620820.0, ans=0.0 2023-11-24 01:21:51,033 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393150 2023-11-24 01:22:11,138 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8400, loss[loss=0.07764, simple_loss=0.1115, pruned_loss=0.01486, audio_tagging_loss=0.007018, over 15438.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.08965, pruned_loss=0.01313, audio_tagging_loss=0.008818, over 3054149.49 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:22:21,053 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.04 vs. limit=22.5 2023-11-24 01:22:44,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.46 vs. limit=15.0 2023-11-24 01:22:54,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393200 2023-11-24 01:22:54,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2621286.6666666665, ans=0.125 2023-11-24 01:23:12,775 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8450, loss[loss=0.05906, simple_loss=0.07844, pruned_loss=0.009221, audio_tagging_loss=0.01062, over 15662.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.08987, pruned_loss=0.01319, audio_tagging_loss=0.008815, over 3056138.21 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:23:23,338 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.428e+01 8.989e+01 9.532e+01 1.176e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-24 01:23:25,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2621486.6666666665, ans=0.1 2023-11-24 01:23:38,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2621553.3333333335, ans=0.0 2023-11-24 01:23:44,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2621553.3333333335, ans=0.125 2023-11-24 01:23:49,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2621620.0, ans=0.125 2023-11-24 01:23:56,089 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393250 2023-11-24 01:24:13,902 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8500, loss[loss=0.09065, simple_loss=0.1262, pruned_loss=0.02059, audio_tagging_loss=0.006949, over 15709.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09057, pruned_loss=0.01326, audio_tagging_loss=0.008803, over 3055250.57 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:24:41,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2621886.6666666665, ans=0.04949747468305833 2023-11-24 01:24:42,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.10 vs. limit=15.0 2023-11-24 01:24:55,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2621953.3333333335, ans=0.09899494936611666 2023-11-24 01:24:57,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393300 2023-11-24 01:25:08,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2622020.0, ans=0.0 2023-11-24 01:25:17,351 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8550, loss[loss=0.05909, simple_loss=0.07789, pruned_loss=0.009736, audio_tagging_loss=0.01041, over 14767.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09077, pruned_loss=0.01325, audio_tagging_loss=0.008884, over 3046162.20 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:25:23,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2622086.6666666665, ans=0.07 2023-11-24 01:25:25,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0 2023-11-24 01:25:29,173 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.811e+01 9.309e+01 9.853e+01 1.247e+02, threshold=1.862e+02, percent-clipped=0.0 2023-11-24 01:25:31,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2622153.3333333335, ans=0.0 2023-11-24 01:25:34,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2622153.3333333335, ans=0.0 2023-11-24 01:25:47,665 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=14.46 vs. limit=15.0 2023-11-24 01:25:55,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2622286.6666666665, ans=0.125 2023-11-24 01:25:59,934 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393350 2023-11-24 01:26:03,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.29 vs. limit=10.0 2023-11-24 01:26:06,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2622353.3333333335, ans=10.0 2023-11-24 01:26:09,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2622353.3333333335, ans=0.125 2023-11-24 01:26:09,418 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:26:12,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2622353.3333333335, ans=0.0 2023-11-24 01:26:18,351 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8600, loss[loss=0.07282, simple_loss=0.0909, pruned_loss=0.01704, audio_tagging_loss=0.01033, over 13476.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09128, pruned_loss=0.01332, audio_tagging_loss=0.00889, over 3046002.23 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:26:26,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2622420.0, ans=0.2 2023-11-24 01:26:31,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2622486.6666666665, ans=0.1 2023-11-24 01:26:42,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2622553.3333333335, ans=0.2 2023-11-24 01:27:00,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2622620.0, ans=0.2 2023-11-24 01:27:01,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393400 2023-11-24 01:27:19,489 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8650, loss[loss=0.08198, simple_loss=0.1093, pruned_loss=0.01886, audio_tagging_loss=0.008446, over 15213.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09167, pruned_loss=0.01349, audio_tagging_loss=0.008935, over 3050262.67 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:27:22,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2622753.3333333335, ans=0.125 2023-11-24 01:27:22,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2622753.3333333335, ans=0.0 2023-11-24 01:27:24,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.15 vs. limit=15.0 2023-11-24 01:27:32,817 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.465e+01 9.032e+01 9.917e+01 1.291e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 01:27:37,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.97 vs. limit=10.0 2023-11-24 01:27:39,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2622820.0, ans=0.2 2023-11-24 01:27:54,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2622886.6666666665, ans=0.0 2023-11-24 01:28:03,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393450 2023-11-24 01:28:05,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2622953.3333333335, ans=0.125 2023-11-24 01:28:13,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2623020.0, ans=0.0 2023-11-24 01:28:14,895 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=22.5 2023-11-24 01:28:22,594 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8700, loss[loss=0.09286, simple_loss=0.1199, pruned_loss=0.02506, audio_tagging_loss=0.007859, over 13887.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09229, pruned_loss=0.01357, audio_tagging_loss=0.008968, over 3048101.04 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:28:24,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2623086.6666666665, ans=0.125 2023-11-24 01:28:27,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2623086.6666666665, ans=0.125 2023-11-24 01:28:28,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2623086.6666666665, ans=0.125 2023-11-24 01:28:38,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2623153.3333333335, ans=0.1 2023-11-24 01:28:39,215 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.73 vs. limit=15.0 2023-11-24 01:28:53,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2623220.0, ans=0.125 2023-11-24 01:28:56,666 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:29:05,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393500 2023-11-24 01:29:18,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2623353.3333333335, ans=0.125 2023-11-24 01:29:23,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2623420.0, ans=0.2 2023-11-24 01:29:24,769 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8750, loss[loss=0.06989, simple_loss=0.09689, pruned_loss=0.01424, audio_tagging_loss=0.007209, over 14389.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09216, pruned_loss=0.01344, audio_tagging_loss=0.009056, over 3043536.87 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:29:26,652 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.69 vs. limit=15.0 2023-11-24 01:29:33,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2623420.0, ans=0.1 2023-11-24 01:29:36,660 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.231e+01 8.417e+01 9.143e+01 1.010e+02 1.352e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 01:29:47,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2623553.3333333335, ans=0.07 2023-11-24 01:30:07,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393550 2023-11-24 01:30:19,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2623686.6666666665, ans=0.125 2023-11-24 01:30:25,616 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8800, loss[loss=0.04633, simple_loss=0.05445, pruned_loss=0.007946, audio_tagging_loss=0.01116, over 13998.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09245, pruned_loss=0.01351, audio_tagging_loss=0.009127, over 3045646.69 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:30:44,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2623820.0, ans=0.125 2023-11-24 01:30:52,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2623886.6666666665, ans=15.0 2023-11-24 01:31:00,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2623886.6666666665, ans=0.125 2023-11-24 01:31:08,767 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393600 2023-11-24 01:31:08,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2623953.3333333335, ans=0.0 2023-11-24 01:31:18,852 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.94 vs. limit=15.0 2023-11-24 01:31:27,898 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8850, loss[loss=0.08842, simple_loss=0.1196, pruned_loss=0.0203, audio_tagging_loss=0.00832, over 15570.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09331, pruned_loss=0.01357, audio_tagging_loss=0.009078, over 3051275.96 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:31:29,914 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.15 vs. limit=22.5 2023-11-24 01:31:39,020 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.33 vs. limit=15.0 2023-11-24 01:31:39,630 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.345e+01 8.328e+01 9.022e+01 9.725e+01 1.238e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-24 01:31:40,849 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 01:31:53,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2624220.0, ans=0.0 2023-11-24 01:32:05,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2624286.6666666665, ans=0.125 2023-11-24 01:32:05,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2624286.6666666665, ans=0.125 2023-11-24 01:32:09,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393650 2023-11-24 01:32:14,823 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:32:28,678 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8900, loss[loss=0.0645, simple_loss=0.09285, pruned_loss=0.01019, audio_tagging_loss=0.007878, over 16015.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09287, pruned_loss=0.01359, audio_tagging_loss=0.009015, over 3048131.99 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:32:44,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.46 vs. limit=12.0 2023-11-24 01:32:51,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2624486.6666666665, ans=0.0 2023-11-24 01:32:52,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2624553.3333333335, ans=0.125 2023-11-24 01:32:52,527 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.89 vs. limit=10.0 2023-11-24 01:32:55,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2624553.3333333335, ans=0.1 2023-11-24 01:33:09,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2624620.0, ans=0.2 2023-11-24 01:33:12,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393700 2023-11-24 01:33:14,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2624620.0, ans=0.1 2023-11-24 01:33:30,362 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 8950, loss[loss=0.07123, simple_loss=0.07832, pruned_loss=0.01711, audio_tagging_loss=0.01496, over 15908.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09212, pruned_loss=0.01344, audio_tagging_loss=0.008935, over 3044773.76 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:33:34,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2624753.3333333335, ans=0.125 2023-11-24 01:33:42,259 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.467e+01 9.115e+01 9.992e+01 1.363e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 01:33:57,402 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.27 vs. limit=15.0 2023-11-24 01:33:59,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2624886.6666666665, ans=0.125 2023-11-24 01:34:07,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2624953.3333333335, ans=0.0 2023-11-24 01:34:08,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2624953.3333333335, ans=0.125 2023-11-24 01:34:13,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393750 2023-11-24 01:34:14,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2624953.3333333335, ans=0.0 2023-11-24 01:34:18,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.67 vs. limit=22.5 2023-11-24 01:34:25,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2625020.0, ans=0.07 2023-11-24 01:34:29,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2625020.0, ans=0.1 2023-11-24 01:34:32,135 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9000, loss[loss=0.06408, simple_loss=0.09592, pruned_loss=0.008686, audio_tagging_loss=0.007432, over 14935.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09163, pruned_loss=0.01333, audio_tagging_loss=0.008901, over 3055970.26 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:34:32,136 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 01:35:10,890 INFO [train_asr.py:1253] (3/4) Epoch 33, validation: loss=0.05892, simple_loss=0.05094, pruned_loss=0.005119, audio_tagging_loss=0.02833, over 4681554.00 frames. 2023-11-24 01:35:10,890 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 01:35:33,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.07 vs. limit=15.0 2023-11-24 01:35:36,388 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.00 vs. limit=15.0 2023-11-24 01:35:49,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2625286.6666666665, ans=0.0 2023-11-24 01:35:53,957 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393800 2023-11-24 01:36:06,173 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.27 vs. limit=22.5 2023-11-24 01:36:12,544 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9050, loss[loss=0.07582, simple_loss=0.09789, pruned_loss=0.02053, audio_tagging_loss=0.006345, over 15279.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09281, pruned_loss=0.01366, audio_tagging_loss=0.008869, over 3059485.91 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:36:24,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2625486.6666666665, ans=0.125 2023-11-24 01:36:25,603 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.819e+01 9.377e+01 1.005e+02 1.265e+02, threshold=1.875e+02, percent-clipped=0.0 2023-11-24 01:36:26,252 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.83 vs. limit=15.0 2023-11-24 01:36:55,937 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393850 2023-11-24 01:36:57,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2625620.0, ans=0.0 2023-11-24 01:37:02,321 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.62 vs. limit=10.0 2023-11-24 01:37:14,626 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9100, loss[loss=0.07408, simple_loss=0.09569, pruned_loss=0.01655, audio_tagging_loss=0.009679, over 15305.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09262, pruned_loss=0.01366, audio_tagging_loss=0.008771, over 3056579.43 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:37:24,549 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.52 vs. limit=15.0 2023-11-24 01:37:25,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.55 vs. limit=22.5 2023-11-24 01:37:30,507 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.09 vs. limit=12.0 2023-11-24 01:37:44,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2625886.6666666665, ans=0.125 2023-11-24 01:37:48,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2625886.6666666665, ans=0.1 2023-11-24 01:37:56,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.00 vs. limit=10.0 2023-11-24 01:37:57,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393900 2023-11-24 01:38:11,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2626020.0, ans=0.0 2023-11-24 01:38:15,239 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9150, loss[loss=0.05335, simple_loss=0.06917, pruned_loss=0.0101, audio_tagging_loss=0.008668, over 14902.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09159, pruned_loss=0.01359, audio_tagging_loss=0.008788, over 3047128.71 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:38:27,508 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 7.991e+01 8.782e+01 9.725e+01 1.593e+02, threshold=1.756e+02, percent-clipped=0.0 2023-11-24 01:38:38,911 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.56 vs. limit=22.5 2023-11-24 01:38:51,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2626286.6666666665, ans=0.1 2023-11-24 01:38:58,283 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 393950 2023-11-24 01:39:06,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2626353.3333333335, ans=0.125 2023-11-24 01:39:09,458 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0 2023-11-24 01:39:16,561 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9200, loss[loss=0.06509, simple_loss=0.07542, pruned_loss=0.01511, audio_tagging_loss=0.01228, over 15486.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09197, pruned_loss=0.01354, audio_tagging_loss=0.00883, over 3049342.34 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 32.0 2023-11-24 01:39:26,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0 2023-11-24 01:39:29,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2626486.6666666665, ans=0.125 2023-11-24 01:39:39,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2626486.6666666665, ans=0.1 2023-11-24 01:39:58,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394000 2023-11-24 01:40:04,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2626686.6666666665, ans=0.1 2023-11-24 01:40:18,494 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9250, loss[loss=0.08594, simple_loss=0.1238, pruned_loss=0.01644, audio_tagging_loss=0.007576, over 14758.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09144, pruned_loss=0.01353, audio_tagging_loss=0.008793, over 3045283.11 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:40:29,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2626820.0, ans=0.025 2023-11-24 01:40:31,440 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.331e+01 9.082e+01 9.914e+01 1.295e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-24 01:40:37,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2626820.0, ans=0.125 2023-11-24 01:40:38,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2626820.0, ans=0.2 2023-11-24 01:40:48,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2626886.6666666665, ans=0.125 2023-11-24 01:41:00,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394050 2023-11-24 01:41:06,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2627020.0, ans=0.125 2023-11-24 01:41:09,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.53 vs. limit=15.0 2023-11-24 01:41:19,481 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9300, loss[loss=0.09715, simple_loss=0.1395, pruned_loss=0.01991, audio_tagging_loss=0.007468, over 16258.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.092, pruned_loss=0.01355, audio_tagging_loss=0.008784, over 3049600.86 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:41:22,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2627086.6666666665, ans=0.1 2023-11-24 01:41:29,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2627086.6666666665, ans=0.1 2023-11-24 01:41:49,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2627220.0, ans=0.1 2023-11-24 01:41:59,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2627286.6666666665, ans=0.0 2023-11-24 01:42:03,205 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394100 2023-11-24 01:42:19,099 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2627353.3333333335, ans=0.125 2023-11-24 01:42:21,232 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9350, loss[loss=0.05638, simple_loss=0.07328, pruned_loss=0.009453, audio_tagging_loss=0.01029, over 15074.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.0913, pruned_loss=0.01339, audio_tagging_loss=0.008885, over 3048553.30 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:42:34,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2627486.6666666665, ans=0.125 2023-11-24 01:42:36,052 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.369e+01 8.431e+01 9.003e+01 9.569e+01 1.110e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-24 01:43:04,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394150 2023-11-24 01:43:20,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2627686.6666666665, ans=0.1 2023-11-24 01:43:22,666 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9400, loss[loss=0.06665, simple_loss=0.08464, pruned_loss=0.01357, audio_tagging_loss=0.01077, over 15253.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.0914, pruned_loss=0.01358, audio_tagging_loss=0.008989, over 3044929.94 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:43:22,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2627753.3333333335, ans=0.0 2023-11-24 01:43:40,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2627820.0, ans=0.0 2023-11-24 01:44:06,273 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394200 2023-11-24 01:44:12,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2628020.0, ans=0.125 2023-11-24 01:44:24,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2628086.6666666665, ans=10.0 2023-11-24 01:44:25,390 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9450, loss[loss=0.05876, simple_loss=0.07611, pruned_loss=0.008589, audio_tagging_loss=0.01212, over 15329.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09079, pruned_loss=0.0134, audio_tagging_loss=0.009041, over 3050830.78 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:44:25,408 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 01:44:39,838 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.278e+01 8.363e+01 8.998e+01 9.818e+01 1.304e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-24 01:44:46,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.51 vs. limit=22.5 2023-11-24 01:45:03,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2628286.6666666665, ans=0.125 2023-11-24 01:45:09,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394250 2023-11-24 01:45:26,608 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9500, loss[loss=0.06251, simple_loss=0.08759, pruned_loss=0.01167, audio_tagging_loss=0.00704, over 15067.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.08987, pruned_loss=0.0132, audio_tagging_loss=0.009154, over 3049019.71 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:45:48,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.56 vs. limit=15.0 2023-11-24 01:46:09,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394300 2023-11-24 01:46:12,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2628620.0, ans=0.125 2023-11-24 01:46:13,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2628620.0, ans=0.125 2023-11-24 01:46:27,674 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9550, loss[loss=0.06969, simple_loss=0.09847, pruned_loss=0.01248, audio_tagging_loss=0.007975, over 14471.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09087, pruned_loss=0.01342, audio_tagging_loss=0.009121, over 3047901.98 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 8.0 2023-11-24 01:46:29,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2628753.3333333335, ans=0.125 2023-11-24 01:46:32,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2628753.3333333335, ans=0.125 2023-11-24 01:46:38,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2628753.3333333335, ans=0.125 2023-11-24 01:46:40,186 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0 2023-11-24 01:46:42,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2628820.0, ans=0.125 2023-11-24 01:46:42,914 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.519e+01 9.047e+01 9.704e+01 1.211e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-24 01:46:43,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2628820.0, ans=0.125 2023-11-24 01:46:48,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.86 vs. limit=15.0 2023-11-24 01:46:58,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2628886.6666666665, ans=0.2 2023-11-24 01:47:10,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394350 2023-11-24 01:47:15,431 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:47:23,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2629020.0, ans=0.1 2023-11-24 01:47:29,289 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9600, loss[loss=0.05746, simple_loss=0.06283, pruned_loss=0.01433, audio_tagging_loss=0.01171, over 14258.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09133, pruned_loss=0.01345, audio_tagging_loss=0.009163, over 3039359.07 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:47:40,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2629153.3333333335, ans=0.1 2023-11-24 01:47:41,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.28 vs. limit=22.5 2023-11-24 01:47:42,249 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.81 vs. limit=15.0 2023-11-24 01:47:56,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2629220.0, ans=0.125 2023-11-24 01:48:04,854 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.14 vs. limit=15.0 2023-11-24 01:48:12,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2629286.6666666665, ans=0.125 2023-11-24 01:48:13,559 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394400 2023-11-24 01:48:16,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2629286.6666666665, ans=0.125 2023-11-24 01:48:26,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2629353.3333333335, ans=0.0 2023-11-24 01:48:27,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2629353.3333333335, ans=0.0 2023-11-24 01:48:31,241 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9650, loss[loss=0.07316, simple_loss=0.1034, pruned_loss=0.01314, audio_tagging_loss=0.008306, over 16650.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09125, pruned_loss=0.01323, audio_tagging_loss=0.009212, over 3045159.19 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:48:38,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2629420.0, ans=0.0 2023-11-24 01:48:45,252 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.217e+01 8.903e+01 9.505e+01 1.138e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-24 01:49:00,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2629553.3333333335, ans=0.1 2023-11-24 01:49:01,976 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.09 vs. limit=22.5 2023-11-24 01:49:05,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2629553.3333333335, ans=0.1 2023-11-24 01:49:05,507 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.33 vs. limit=15.0 2023-11-24 01:49:14,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394450 2023-11-24 01:49:32,172 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9700, loss[loss=0.08116, simple_loss=0.1135, pruned_loss=0.01688, audio_tagging_loss=0.007523, over 14820.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.0902, pruned_loss=0.01311, audio_tagging_loss=0.009127, over 3051620.99 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:49:32,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2629753.3333333335, ans=0.125 2023-11-24 01:49:33,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2629753.3333333335, ans=0.0 2023-11-24 01:49:56,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2629886.6666666665, ans=0.125 2023-11-24 01:50:00,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2629886.6666666665, ans=0.125 2023-11-24 01:50:15,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394500 2023-11-24 01:50:23,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2630020.0, ans=0.0 2023-11-24 01:50:34,842 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9750, loss[loss=0.04142, simple_loss=0.05144, pruned_loss=0.005172, audio_tagging_loss=0.01053, over 15113.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09088, pruned_loss=0.01327, audio_tagging_loss=0.009032, over 3046848.33 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:50:41,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2630086.6666666665, ans=0.05 2023-11-24 01:50:48,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2630153.3333333335, ans=0.1 2023-11-24 01:50:49,139 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.192e+01 8.620e+01 9.194e+01 9.972e+01 1.344e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 01:50:55,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2630153.3333333335, ans=0.0 2023-11-24 01:50:56,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2630153.3333333335, ans=0.0 2023-11-24 01:51:17,694 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394550 2023-11-24 01:51:36,068 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9800, loss[loss=0.06436, simple_loss=0.08923, pruned_loss=0.0116, audio_tagging_loss=0.008149, over 14043.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09203, pruned_loss=0.01352, audio_tagging_loss=0.008922, over 3046931.07 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:51:47,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2630486.6666666665, ans=0.0 2023-11-24 01:51:48,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0 2023-11-24 01:52:02,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2630553.3333333335, ans=0.1 2023-11-24 01:52:04,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2630553.3333333335, ans=0.125 2023-11-24 01:52:19,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394600 2023-11-24 01:52:23,749 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 01:52:31,358 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.10 vs. limit=22.5 2023-11-24 01:52:33,009 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 01:52:34,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2630686.6666666665, ans=0.125 2023-11-24 01:52:34,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=15.0 2023-11-24 01:52:37,798 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9850, loss[loss=0.0828, simple_loss=0.1117, pruned_loss=0.01803, audio_tagging_loss=0.008927, over 15261.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09251, pruned_loss=0.01362, audio_tagging_loss=0.008774, over 3051052.05 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:52:53,143 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.642e+01 9.395e+01 1.002e+02 1.404e+02, threshold=1.879e+02, percent-clipped=0.0 2023-11-24 01:53:07,594 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.59 vs. limit=15.0 2023-11-24 01:53:11,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2630886.6666666665, ans=0.125 2023-11-24 01:53:21,246 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394650 2023-11-24 01:53:36,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2631020.0, ans=0.2 2023-11-24 01:53:40,117 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9900, loss[loss=0.06681, simple_loss=0.08904, pruned_loss=0.007892, audio_tagging_loss=0.01441, over 15698.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09297, pruned_loss=0.01362, audio_tagging_loss=0.008793, over 3045569.43 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:53:40,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2631086.6666666665, ans=0.125 2023-11-24 01:53:48,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2631086.6666666665, ans=0.125 2023-11-24 01:53:56,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2631153.3333333335, ans=0.1 2023-11-24 01:54:23,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394700 2023-11-24 01:54:23,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2631286.6666666665, ans=0.125 2023-11-24 01:54:27,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2631286.6666666665, ans=0.125 2023-11-24 01:54:42,052 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 9950, loss[loss=0.04664, simple_loss=0.05583, pruned_loss=0.00633, audio_tagging_loss=0.0124, over 14387.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09226, pruned_loss=0.01346, audio_tagging_loss=0.00877, over 3050762.50 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0 2023-11-24 01:54:46,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2631420.0, ans=0.125 2023-11-24 01:54:56,124 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.637e+01 9.131e+01 9.854e+01 1.265e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 01:54:57,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2631486.6666666665, ans=0.09899494936611666 2023-11-24 01:55:12,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2631553.3333333335, ans=0.125 2023-11-24 01:55:24,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394750 2023-11-24 01:55:42,560 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10000, loss[loss=0.08155, simple_loss=0.1081, pruned_loss=0.02062, audio_tagging_loss=0.006901, over 15297.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.092, pruned_loss=0.01357, audio_tagging_loss=0.008897, over 3054866.98 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 32.0 2023-11-24 01:55:48,968 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.45 vs. limit=22.5 2023-11-24 01:55:57,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2631820.0, ans=0.125 2023-11-24 01:56:06,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2631886.6666666665, ans=0.0 2023-11-24 01:56:21,518 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.30 vs. limit=22.5 2023-11-24 01:56:25,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394800 2023-11-24 01:56:30,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2632020.0, ans=0.125 2023-11-24 01:56:39,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2632020.0, ans=0.0 2023-11-24 01:56:44,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2632086.6666666665, ans=0.2 2023-11-24 01:56:44,583 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.77 vs. limit=15.0 2023-11-24 01:56:45,109 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10050, loss[loss=0.05831, simple_loss=0.07675, pruned_loss=0.01206, audio_tagging_loss=0.007882, over 15750.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09212, pruned_loss=0.01366, audio_tagging_loss=0.008969, over 3046626.32 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 01:56:45,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2632086.6666666665, ans=0.0 2023-11-24 01:56:53,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2632086.6666666665, ans=0.0 2023-11-24 01:56:55,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2632086.6666666665, ans=0.0 2023-11-24 01:56:59,416 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.081e+01 8.389e+01 9.133e+01 9.609e+01 1.226e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 01:57:05,568 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.66 vs. limit=15.0 2023-11-24 01:57:08,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2632220.0, ans=0.1 2023-11-24 01:57:27,583 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394850 2023-11-24 01:57:31,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2632286.6666666665, ans=0.125 2023-11-24 01:57:37,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.16 vs. limit=15.0 2023-11-24 01:57:46,295 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10100, loss[loss=0.07423, simple_loss=0.1037, pruned_loss=0.01457, audio_tagging_loss=0.007822, over 15620.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09267, pruned_loss=0.01363, audio_tagging_loss=0.009042, over 3046142.23 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 01:57:48,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2632420.0, ans=0.1 2023-11-24 01:57:57,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2632486.6666666665, ans=0.125 2023-11-24 01:58:16,596 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.42 vs. limit=15.0 2023-11-24 01:58:29,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394900 2023-11-24 01:58:36,500 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 01:58:40,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2632686.6666666665, ans=0.0 2023-11-24 01:58:47,636 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10150, loss[loss=0.07659, simple_loss=0.1108, pruned_loss=0.01372, audio_tagging_loss=0.007476, over 15192.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09321, pruned_loss=0.01358, audio_tagging_loss=0.00903, over 3050138.27 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 01:58:56,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2632753.3333333335, ans=0.125 2023-11-24 01:58:57,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.96 vs. limit=12.0 2023-11-24 01:59:02,467 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.176e+01 8.517e+01 9.184e+01 1.003e+02 1.404e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-24 01:59:06,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2632820.0, ans=0.125 2023-11-24 01:59:17,273 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 01:59:18,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2632886.6666666665, ans=0.09899494936611666 2023-11-24 01:59:30,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 394950 2023-11-24 01:59:49,436 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10200, loss[loss=0.06584, simple_loss=0.07704, pruned_loss=0.01697, audio_tagging_loss=0.01035, over 15941.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09262, pruned_loss=0.01358, audio_tagging_loss=0.00912, over 3051082.23 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 01:59:58,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2633086.6666666665, ans=0.125 2023-11-24 02:00:14,012 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 02:00:32,823 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395000 2023-11-24 02:00:42,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2633353.3333333335, ans=0.125 2023-11-24 02:00:50,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2633353.3333333335, ans=0.0 2023-11-24 02:00:52,216 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10250, loss[loss=0.06559, simple_loss=0.09215, pruned_loss=0.01205, audio_tagging_loss=0.007456, over 14706.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09266, pruned_loss=0.01356, audio_tagging_loss=0.009155, over 3047166.45 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:01:07,145 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.621e+01 8.620e+01 9.148e+01 9.659e+01 1.254e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 02:01:13,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.11 vs. limit=6.0 2023-11-24 02:01:17,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2633553.3333333335, ans=0.0 2023-11-24 02:01:25,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2633553.3333333335, ans=0.1 2023-11-24 02:01:36,536 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395050 2023-11-24 02:01:54,957 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10300, loss[loss=0.05996, simple_loss=0.07107, pruned_loss=0.01458, audio_tagging_loss=0.00984, over 14407.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09243, pruned_loss=0.01356, audio_tagging_loss=0.0091, over 3046352.68 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:02:06,297 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.39 vs. limit=15.0 2023-11-24 02:02:31,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2633953.3333333335, ans=0.125 2023-11-24 02:02:34,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2633953.3333333335, ans=0.0 2023-11-24 02:02:35,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.94 vs. limit=15.0 2023-11-24 02:02:38,292 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395100 2023-11-24 02:02:53,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2634020.0, ans=0.0 2023-11-24 02:02:56,671 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10350, loss[loss=0.07395, simple_loss=0.09923, pruned_loss=0.01371, audio_tagging_loss=0.01063, over 14368.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09212, pruned_loss=0.01349, audio_tagging_loss=0.009253, over 3046445.99 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:02:58,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2634086.6666666665, ans=0.0 2023-11-24 02:03:08,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2634153.3333333335, ans=0.07 2023-11-24 02:03:12,159 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.449e+01 8.937e+01 9.719e+01 1.290e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-24 02:03:12,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2634153.3333333335, ans=0.0 2023-11-24 02:03:12,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2634153.3333333335, ans=0.1 2023-11-24 02:03:19,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2634153.3333333335, ans=0.1 2023-11-24 02:03:27,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2634220.0, ans=0.0 2023-11-24 02:03:35,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2634286.6666666665, ans=0.125 2023-11-24 02:03:39,987 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395150 2023-11-24 02:03:58,840 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10400, loss[loss=0.08125, simple_loss=0.1037, pruned_loss=0.0202, audio_tagging_loss=0.009229, over 16457.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09193, pruned_loss=0.01356, audio_tagging_loss=0.009226, over 3045017.91 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:04:00,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2634420.0, ans=0.125 2023-11-24 02:04:06,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2634420.0, ans=0.05 2023-11-24 02:04:38,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.63 vs. limit=15.0 2023-11-24 02:04:41,802 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395200 2023-11-24 02:04:42,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2634620.0, ans=0.0 2023-11-24 02:04:46,166 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.27 vs. limit=15.0 2023-11-24 02:04:59,976 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10450, loss[loss=0.06493, simple_loss=0.08561, pruned_loss=0.01189, audio_tagging_loss=0.01023, over 14375.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09106, pruned_loss=0.01342, audio_tagging_loss=0.009241, over 3034435.72 frames. ], batch size: 54, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:05:14,724 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.359e+01 8.851e+01 9.617e+01 1.233e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-24 02:05:23,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2634820.0, ans=0.1 2023-11-24 02:05:35,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2634886.6666666665, ans=0.125 2023-11-24 02:05:43,214 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395250 2023-11-24 02:05:49,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=5.15 vs. limit=5.0 2023-11-24 02:06:01,542 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10500, loss[loss=0.0443, simple_loss=0.05933, pruned_loss=0.005573, audio_tagging_loss=0.009063, over 14238.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09012, pruned_loss=0.01324, audio_tagging_loss=0.009196, over 3029521.43 frames. ], batch size: 54, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:06:04,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2635086.6666666665, ans=0.1 2023-11-24 02:06:29,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2635220.0, ans=0.125 2023-11-24 02:06:43,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2635286.6666666665, ans=0.125 2023-11-24 02:06:44,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395300 2023-11-24 02:07:03,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2635420.0, ans=0.125 2023-11-24 02:07:04,400 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10550, loss[loss=0.05582, simple_loss=0.07317, pruned_loss=0.009236, audio_tagging_loss=0.01, over 15860.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09052, pruned_loss=0.01314, audio_tagging_loss=0.009081, over 3037127.90 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:07:06,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2635420.0, ans=0.1 2023-11-24 02:07:19,126 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.732e+01 9.283e+01 1.036e+02 1.632e+02, threshold=1.857e+02, percent-clipped=0.0 2023-11-24 02:07:47,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2635620.0, ans=0.0 2023-11-24 02:07:48,221 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395350 2023-11-24 02:07:55,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2635686.6666666665, ans=0.125 2023-11-24 02:08:00,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2635686.6666666665, ans=0.125 2023-11-24 02:08:02,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2635686.6666666665, ans=0.125 2023-11-24 02:08:03,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2635686.6666666665, ans=0.1 2023-11-24 02:08:05,712 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10600, loss[loss=0.05968, simple_loss=0.08085, pruned_loss=0.01268, audio_tagging_loss=0.006564, over 15243.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09186, pruned_loss=0.0134, audio_tagging_loss=0.008926, over 3047414.45 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:08:09,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2635753.3333333335, ans=0.0 2023-11-24 02:08:10,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2635753.3333333335, ans=0.2 2023-11-24 02:08:11,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2635753.3333333335, ans=0.0 2023-11-24 02:08:12,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2635753.3333333335, ans=0.125 2023-11-24 02:08:32,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.06 vs. limit=6.0 2023-11-24 02:08:48,909 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395400 2023-11-24 02:08:49,359 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.40 vs. limit=15.0 2023-11-24 02:09:01,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2636020.0, ans=0.0 2023-11-24 02:09:07,233 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10650, loss[loss=0.04226, simple_loss=0.04768, pruned_loss=0.007533, audio_tagging_loss=0.01088, over 14338.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09145, pruned_loss=0.01336, audio_tagging_loss=0.00894, over 3048015.55 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:09:23,868 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.426e+01 9.214e+01 9.689e+01 1.168e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 02:09:51,021 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395450 2023-11-24 02:10:10,656 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10700, loss[loss=0.05701, simple_loss=0.06807, pruned_loss=0.01032, audio_tagging_loss=0.01265, over 15475.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09099, pruned_loss=0.01324, audio_tagging_loss=0.008937, over 3048661.94 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:10:10,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2636420.0, ans=0.125 2023-11-24 02:10:25,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.84 vs. limit=15.0 2023-11-24 02:10:51,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2636620.0, ans=0.2 2023-11-24 02:10:53,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395500 2023-11-24 02:11:11,234 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10750, loss[loss=0.07362, simple_loss=0.09181, pruned_loss=0.02099, audio_tagging_loss=0.006726, over 15461.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09133, pruned_loss=0.01331, audio_tagging_loss=0.00892, over 3052158.47 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:11:16,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2636753.3333333335, ans=0.07 2023-11-24 02:11:19,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.89 vs. limit=15.0 2023-11-24 02:11:23,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2636820.0, ans=0.125 2023-11-24 02:11:26,616 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.296e+01 8.958e+01 9.992e+01 1.267e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-24 02:11:28,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2636820.0, ans=0.1 2023-11-24 02:11:33,667 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.85 vs. limit=15.0 2023-11-24 02:11:39,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2636886.6666666665, ans=0.125 2023-11-24 02:11:44,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2636886.6666666665, ans=0.125 2023-11-24 02:11:55,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395550 2023-11-24 02:12:08,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2637020.0, ans=0.2 2023-11-24 02:12:12,803 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10800, loss[loss=0.07044, simple_loss=0.1035, pruned_loss=0.01213, audio_tagging_loss=0.00657, over 14450.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09099, pruned_loss=0.01314, audio_tagging_loss=0.008792, over 3055496.16 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:12:28,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2637153.3333333335, ans=0.125 2023-11-24 02:12:39,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2637220.0, ans=0.125 2023-11-24 02:12:40,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2637220.0, ans=0.025 2023-11-24 02:12:51,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2637286.6666666665, ans=0.125 2023-11-24 02:12:56,731 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395600 2023-11-24 02:13:04,638 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0 2023-11-24 02:13:11,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2637353.3333333335, ans=0.1 2023-11-24 02:13:16,950 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10850, loss[loss=0.07841, simple_loss=0.107, pruned_loss=0.01449, audio_tagging_loss=0.01041, over 15743.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09102, pruned_loss=0.01313, audio_tagging_loss=0.008859, over 3057650.85 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:13:33,528 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.225e+01 8.902e+01 9.397e+01 1.304e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-24 02:14:00,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395650 2023-11-24 02:14:12,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2637686.6666666665, ans=0.05 2023-11-24 02:14:16,173 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 02:14:18,578 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10900, loss[loss=0.06536, simple_loss=0.08424, pruned_loss=0.01284, audio_tagging_loss=0.0104, over 15507.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09023, pruned_loss=0.01298, audio_tagging_loss=0.009006, over 3059362.67 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:14:20,249 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.60 vs. limit=15.0 2023-11-24 02:15:01,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395700 2023-11-24 02:15:12,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2638020.0, ans=0.125 2023-11-24 02:15:19,295 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 10950, loss[loss=0.08777, simple_loss=0.1208, pruned_loss=0.01834, audio_tagging_loss=0.009031, over 15120.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09001, pruned_loss=0.01315, audio_tagging_loss=0.009056, over 3055593.00 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:15:21,163 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.12 vs. limit=22.5 2023-11-24 02:15:36,937 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.249e+01 9.167e+01 9.838e+01 1.256e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 02:15:57,619 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.03 vs. limit=15.0 2023-11-24 02:16:00,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2638286.6666666665, ans=0.1 2023-11-24 02:16:02,763 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395750 2023-11-24 02:16:22,049 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11000, loss[loss=0.06135, simple_loss=0.07609, pruned_loss=0.014, audio_tagging_loss=0.009309, over 14779.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09084, pruned_loss=0.01338, audio_tagging_loss=0.00904, over 3053365.54 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:16:33,494 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 02:16:39,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2638486.6666666665, ans=0.125 2023-11-24 02:16:46,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2638553.3333333335, ans=0.125 2023-11-24 02:16:49,330 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.92 vs. limit=15.0 2023-11-24 02:16:50,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2638553.3333333335, ans=0.125 2023-11-24 02:17:02,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2638620.0, ans=0.125 2023-11-24 02:17:04,759 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395800 2023-11-24 02:17:04,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2638620.0, ans=0.04949747468305833 2023-11-24 02:17:24,176 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11050, loss[loss=0.05949, simple_loss=0.07977, pruned_loss=0.01027, audio_tagging_loss=0.009332, over 15856.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09117, pruned_loss=0.01347, audio_tagging_loss=0.009101, over 3054792.81 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:17:31,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2638753.3333333335, ans=0.125 2023-11-24 02:17:35,393 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.97 vs. limit=15.0 2023-11-24 02:17:36,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2638820.0, ans=0.125 2023-11-24 02:17:40,567 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.311e+01 8.882e+01 9.606e+01 1.257e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-24 02:17:59,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2638886.6666666665, ans=0.0 2023-11-24 02:18:03,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2638953.3333333335, ans=0.0 2023-11-24 02:18:07,192 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395850 2023-11-24 02:18:25,476 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11100, loss[loss=0.05688, simple_loss=0.07083, pruned_loss=0.01066, audio_tagging_loss=0.0108, over 16241.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09249, pruned_loss=0.01363, audio_tagging_loss=0.009164, over 3056678.69 frames. ], batch size: 61, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:18:43,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2639153.3333333335, ans=0.0 2023-11-24 02:18:45,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2639153.3333333335, ans=0.125 2023-11-24 02:18:51,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2639220.0, ans=0.0 2023-11-24 02:18:52,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2639220.0, ans=0.05 2023-11-24 02:19:00,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2639220.0, ans=10.0 2023-11-24 02:19:08,914 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395900 2023-11-24 02:19:09,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2639286.6666666665, ans=0.1 2023-11-24 02:19:20,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.59 vs. limit=15.0 2023-11-24 02:19:27,080 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11150, loss[loss=0.06513, simple_loss=0.08451, pruned_loss=0.01184, audio_tagging_loss=0.01104, over 16558.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09152, pruned_loss=0.01342, audio_tagging_loss=0.009333, over 3061993.11 frames. ], batch size: 63, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:19:45,343 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.458e+01 9.119e+01 9.750e+01 1.333e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-24 02:19:50,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2639486.6666666665, ans=0.125 2023-11-24 02:20:10,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 395950 2023-11-24 02:20:10,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2639620.0, ans=0.035 2023-11-24 02:20:23,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2639686.6666666665, ans=0.0 2023-11-24 02:20:29,217 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11200, loss[loss=0.07263, simple_loss=0.08905, pruned_loss=0.01766, audio_tagging_loss=0.01045, over 15292.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09199, pruned_loss=0.01341, audio_tagging_loss=0.009333, over 3062751.18 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:20:38,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2639753.3333333335, ans=0.0 2023-11-24 02:20:59,334 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.51 vs. limit=15.0 2023-11-24 02:21:12,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396000 2023-11-24 02:21:31,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2640020.0, ans=0.2 2023-11-24 02:21:34,636 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11250, loss[loss=0.08654, simple_loss=0.1163, pruned_loss=0.02086, audio_tagging_loss=0.007539, over 14901.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09114, pruned_loss=0.01326, audio_tagging_loss=0.009369, over 3062433.11 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:21:37,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2640086.6666666665, ans=0.0 2023-11-24 02:21:53,432 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.556e+01 8.444e+01 8.848e+01 9.459e+01 1.220e+02, threshold=1.770e+02, percent-clipped=0.0 2023-11-24 02:21:55,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2640153.3333333335, ans=10.0 2023-11-24 02:22:10,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.31 vs. limit=15.0 2023-11-24 02:22:18,379 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396050 2023-11-24 02:22:22,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2640286.6666666665, ans=0.125 2023-11-24 02:22:34,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2640420.0, ans=0.1 2023-11-24 02:22:36,421 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11300, loss[loss=0.04959, simple_loss=0.06665, pruned_loss=0.006355, audio_tagging_loss=0.009908, over 15079.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09104, pruned_loss=0.01337, audio_tagging_loss=0.009216, over 3061367.55 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:23:01,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2640553.3333333335, ans=0.125 2023-11-24 02:23:07,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2640553.3333333335, ans=0.2 2023-11-24 02:23:11,435 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 02:23:13,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2640620.0, ans=0.0 2023-11-24 02:23:19,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396100 2023-11-24 02:23:38,690 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11350, loss[loss=0.06029, simple_loss=0.0794, pruned_loss=0.01196, audio_tagging_loss=0.008634, over 15391.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09084, pruned_loss=0.01337, audio_tagging_loss=0.009076, over 3061315.84 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:23:42,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2640753.3333333335, ans=0.125 2023-11-24 02:23:57,205 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.374e+01 8.369e+01 8.906e+01 9.576e+01 1.384e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-24 02:24:12,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2640886.6666666665, ans=0.125 2023-11-24 02:24:22,829 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396150 2023-11-24 02:24:33,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2641020.0, ans=0.0 2023-11-24 02:24:38,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2641020.0, ans=0.125 2023-11-24 02:24:41,105 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11400, loss[loss=0.06742, simple_loss=0.08803, pruned_loss=0.01427, audio_tagging_loss=0.00914, over 14309.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.0908, pruned_loss=0.0134, audio_tagging_loss=0.009079, over 3051821.04 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:24:51,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2641153.3333333335, ans=0.125 2023-11-24 02:24:53,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2641153.3333333335, ans=0.125 2023-11-24 02:24:57,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2641153.3333333335, ans=0.125 2023-11-24 02:25:09,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.88 vs. limit=10.0 2023-11-24 02:25:23,942 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396200 2023-11-24 02:25:28,667 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 02:25:33,442 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2641353.3333333335, ans=0.125 2023-11-24 02:25:40,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2641353.3333333335, ans=0.2 2023-11-24 02:25:42,738 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11450, loss[loss=0.07736, simple_loss=0.1067, pruned_loss=0.01408, audio_tagging_loss=0.009955, over 15591.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09134, pruned_loss=0.01345, audio_tagging_loss=0.008953, over 3056081.35 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:25:56,484 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 02:25:59,154 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.94 vs. limit=10.0 2023-11-24 02:26:02,135 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.524e+01 9.029e+01 9.681e+01 1.161e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 02:26:13,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.32 vs. limit=22.5 2023-11-24 02:26:18,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2641553.3333333335, ans=0.0 2023-11-24 02:26:20,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2641620.0, ans=0.1 2023-11-24 02:26:26,325 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396250 2023-11-24 02:26:33,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2641686.6666666665, ans=0.125 2023-11-24 02:26:41,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2641686.6666666665, ans=0.0 2023-11-24 02:26:43,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2641686.6666666665, ans=0.0 2023-11-24 02:26:45,796 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11500, loss[loss=0.07813, simple_loss=0.09827, pruned_loss=0.01748, audio_tagging_loss=0.01152, over 14371.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09143, pruned_loss=0.01345, audio_tagging_loss=0.008908, over 3045905.59 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:26:48,877 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.39 vs. limit=15.0 2023-11-24 02:26:55,002 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.63 vs. limit=15.0 2023-11-24 02:26:57,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0 2023-11-24 02:27:21,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2641953.3333333335, ans=0.1 2023-11-24 02:27:30,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396300 2023-11-24 02:27:47,702 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11550, loss[loss=0.06634, simple_loss=0.09426, pruned_loss=0.01063, audio_tagging_loss=0.008577, over 15988.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09085, pruned_loss=0.01337, audio_tagging_loss=0.008914, over 3046460.81 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:27:58,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2642086.6666666665, ans=0.125 2023-11-24 02:28:06,102 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.401e+01 9.028e+01 9.629e+01 1.407e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 02:28:12,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2642220.0, ans=0.1 2023-11-24 02:28:24,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2642286.6666666665, ans=0.0 2023-11-24 02:28:24,826 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.42 vs. limit=12.0 2023-11-24 02:28:26,668 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 02:28:31,410 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396350 2023-11-24 02:28:31,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2642286.6666666665, ans=0.0 2023-11-24 02:28:49,968 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11600, loss[loss=0.07853, simple_loss=0.1078, pruned_loss=0.01741, audio_tagging_loss=0.007221, over 15922.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09168, pruned_loss=0.01354, audio_tagging_loss=0.008851, over 3046241.64 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:29:09,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2642486.6666666665, ans=0.1 2023-11-24 02:29:20,859 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.04 vs. limit=15.0 2023-11-24 02:29:34,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396400 2023-11-24 02:29:36,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2642620.0, ans=0.04949747468305833 2023-11-24 02:29:53,916 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11650, loss[loss=0.06257, simple_loss=0.083, pruned_loss=0.01072, audio_tagging_loss=0.01035, over 14660.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09118, pruned_loss=0.01344, audio_tagging_loss=0.008984, over 3051505.22 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:30:12,884 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.278e+01 8.940e+01 9.530e+01 1.250e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-24 02:30:16,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2642886.6666666665, ans=0.2 2023-11-24 02:30:20,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2642886.6666666665, ans=0.09899494936611666 2023-11-24 02:30:37,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396450 2023-11-24 02:30:50,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2643020.0, ans=0.125 2023-11-24 02:30:55,486 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11700, loss[loss=0.07546, simple_loss=0.1119, pruned_loss=0.01409, audio_tagging_loss=0.005425, over 16653.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09153, pruned_loss=0.01346, audio_tagging_loss=0.009007, over 3043867.67 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:30:58,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2643086.6666666665, ans=0.0 2023-11-24 02:31:15,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2643153.3333333335, ans=0.1 2023-11-24 02:31:26,352 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 02:31:39,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396500 2023-11-24 02:31:42,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2643286.6666666665, ans=0.125 2023-11-24 02:31:56,463 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11750, loss[loss=0.08395, simple_loss=0.1125, pruned_loss=0.02091, audio_tagging_loss=0.00679, over 15698.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09157, pruned_loss=0.0135, audio_tagging_loss=0.009024, over 3053590.46 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:31:56,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2643420.0, ans=0.0 2023-11-24 02:32:01,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2643420.0, ans=0.1 2023-11-24 02:32:08,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2643486.6666666665, ans=0.2 2023-11-24 02:32:16,601 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 02:32:17,388 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.547e+01 9.165e+01 9.829e+01 1.276e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 02:32:23,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2643553.3333333335, ans=0.0 2023-11-24 02:32:40,297 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396550 2023-11-24 02:32:44,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2643620.0, ans=0.125 2023-11-24 02:32:44,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2643620.0, ans=0.125 2023-11-24 02:32:49,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2643686.6666666665, ans=0.0 2023-11-24 02:32:56,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2643686.6666666665, ans=0.125 2023-11-24 02:33:00,492 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11800, loss[loss=0.07213, simple_loss=0.0939, pruned_loss=0.01607, audio_tagging_loss=0.009111, over 15469.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09097, pruned_loss=0.01346, audio_tagging_loss=0.009086, over 3045110.52 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:33:01,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2643753.3333333335, ans=0.1 2023-11-24 02:33:07,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2643753.3333333335, ans=0.125 2023-11-24 02:33:10,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2643753.3333333335, ans=0.125 2023-11-24 02:33:25,624 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.04 vs. limit=6.0 2023-11-24 02:33:27,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2643886.6666666665, ans=0.125 2023-11-24 02:33:37,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2643953.3333333335, ans=0.1 2023-11-24 02:33:43,359 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396600 2023-11-24 02:33:44,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2643953.3333333335, ans=0.0 2023-11-24 02:34:01,631 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11850, loss[loss=0.07238, simple_loss=0.09748, pruned_loss=0.01445, audio_tagging_loss=0.009192, over 15275.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09119, pruned_loss=0.01353, audio_tagging_loss=0.009187, over 3046857.45 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:34:20,345 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.605e+01 8.557e+01 9.128e+01 9.993e+01 1.182e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 02:34:25,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2644220.0, ans=0.0 2023-11-24 02:34:27,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2644220.0, ans=0.1 2023-11-24 02:34:45,221 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396650 2023-11-24 02:34:45,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2644286.6666666665, ans=0.2 2023-11-24 02:34:57,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2644353.3333333335, ans=0.1 2023-11-24 02:35:02,954 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11900, loss[loss=0.07196, simple_loss=0.09772, pruned_loss=0.01147, audio_tagging_loss=0.01162, over 15166.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09143, pruned_loss=0.01351, audio_tagging_loss=0.009278, over 3043475.07 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:35:28,491 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.77 vs. limit=15.0 2023-11-24 02:35:29,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2644553.3333333335, ans=0.1 2023-11-24 02:35:32,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2644553.3333333335, ans=0.1 2023-11-24 02:35:46,484 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396700 2023-11-24 02:36:05,992 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 11950, loss[loss=0.07455, simple_loss=0.09354, pruned_loss=0.01863, audio_tagging_loss=0.009155, over 14664.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09191, pruned_loss=0.01374, audio_tagging_loss=0.009267, over 3039016.05 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0 2023-11-24 02:36:13,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2644753.3333333335, ans=0.125 2023-11-24 02:36:24,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2644820.0, ans=0.125 2023-11-24 02:36:25,409 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.152e+01 8.882e+01 9.494e+01 1.160e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-24 02:36:47,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396750 2023-11-24 02:36:53,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2645020.0, ans=0.0 2023-11-24 02:37:06,014 INFO [train_asr.py:1221] (3/4) Epoch 33, batch 12000, loss[loss=0.08451, simple_loss=0.1095, pruned_loss=0.02206, audio_tagging_loss=0.007702, over 14620.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09168, pruned_loss=0.01371, audio_tagging_loss=0.009238, over 3040523.85 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0 2023-11-24 02:37:06,015 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 02:37:31,923 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.7950, 4.9621, 5.0765, 4.8900], device='cuda:3') 2023-11-24 02:37:43,641 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6068, 3.6424, 3.9910, 3.4770], device='cuda:3') 2023-11-24 02:37:46,444 INFO [train_asr.py:1253] (3/4) Epoch 33, validation: loss=0.05829, simple_loss=0.05098, pruned_loss=0.005164, audio_tagging_loss=0.02763, over 4681554.00 frames. 2023-11-24 02:37:46,445 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 02:37:51,788 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.46 vs. limit=22.5 2023-11-24 02:37:55,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2645086.6666666665, ans=0.125 2023-11-24 02:37:56,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2645153.3333333335, ans=0.125 2023-11-24 02:37:57,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2645153.3333333335, ans=0.1 2023-11-24 02:38:03,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2645153.3333333335, ans=0.2 2023-11-24 02:38:12,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2645220.0, ans=0.1 2023-11-24 02:38:48,868 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 0, loss[loss=0.06867, simple_loss=0.06593, pruned_loss=0.008707, audio_tagging_loss=0.027, over 16670.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.06593, pruned_loss=0.008707, audio_tagging_loss=0.027, over 16670.00 frames. ], batch size: 66, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:38:48,869 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 02:39:24,516 INFO [train_asr.py:1253] (3/4) Epoch 34, validation: loss=0.058, simple_loss=0.05102, pruned_loss=0.005202, audio_tagging_loss=0.02729, over 4681554.00 frames. 2023-11-24 02:39:24,516 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 02:39:29,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2645253.3333333335, ans=0.0 2023-11-24 02:39:37,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396800 2023-11-24 02:40:16,360 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.450e+01 9.137e+01 9.827e+01 1.054e+02 1.431e+02, threshold=1.965e+02, percent-clipped=0.0 2023-11-24 02:40:21,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2645520.0, ans=0.0 2023-11-24 02:40:27,165 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 50, loss[loss=0.05591, simple_loss=0.05824, pruned_loss=0.008072, audio_tagging_loss=0.01872, over 14930.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.09174, pruned_loss=0.01304, audio_tagging_loss=0.01738, over 683928.48 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:40:28,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2645586.6666666665, ans=0.0 2023-11-24 02:40:29,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2645586.6666666665, ans=0.125 2023-11-24 02:40:39,311 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396850 2023-11-24 02:40:44,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2645653.3333333335, ans=0.125 2023-11-24 02:40:45,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2645653.3333333335, ans=0.1 2023-11-24 02:40:51,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2645720.0, ans=0.0 2023-11-24 02:41:20,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2645853.3333333335, ans=0.125 2023-11-24 02:41:26,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2645853.3333333335, ans=0.0 2023-11-24 02:41:29,491 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 100, loss[loss=0.07174, simple_loss=0.09474, pruned_loss=0.01016, audio_tagging_loss=0.01421, over 15215.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09226, pruned_loss=0.01339, audio_tagging_loss=0.01662, over 1211060.61 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:41:29,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2645920.0, ans=0.125 2023-11-24 02:41:41,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396900 2023-11-24 02:41:47,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.37 vs. limit=12.0 2023-11-24 02:42:07,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2646120.0, ans=0.0 2023-11-24 02:42:20,400 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.325e+01 8.964e+01 9.568e+01 1.027e+02 1.310e+02, threshold=1.914e+02, percent-clipped=0.0 2023-11-24 02:42:31,555 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 150, loss[loss=0.05932, simple_loss=0.07533, pruned_loss=0.01336, audio_tagging_loss=0.008291, over 14225.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.09211, pruned_loss=0.01355, audio_tagging_loss=0.01492, over 1612038.79 frames. ], batch size: 55, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:42:43,926 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=23.52 vs. limit=22.5 2023-11-24 02:42:44,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 396950 2023-11-24 02:42:44,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2646320.0, ans=0.125 2023-11-24 02:43:00,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2646386.6666666665, ans=0.1 2023-11-24 02:43:00,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.52 vs. limit=15.0 2023-11-24 02:43:07,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2646453.3333333335, ans=0.125 2023-11-24 02:43:17,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2646453.3333333335, ans=0.125 2023-11-24 02:43:21,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2646520.0, ans=0.125 2023-11-24 02:43:21,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2646520.0, ans=0.125 2023-11-24 02:43:25,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2646520.0, ans=0.125 2023-11-24 02:43:34,094 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 200, loss[loss=0.08214, simple_loss=0.1153, pruned_loss=0.01843, audio_tagging_loss=0.006055, over 15856.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09141, pruned_loss=0.01356, audio_tagging_loss=0.01325, over 1928412.98 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:43:41,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2646586.6666666665, ans=0.2 2023-11-24 02:43:46,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397000 2023-11-24 02:43:52,342 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 02:43:52,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2646653.3333333335, ans=0.0 2023-11-24 02:43:57,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2646720.0, ans=0.125 2023-11-24 02:43:59,909 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.47 vs. limit=10.0 2023-11-24 02:44:02,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.47 vs. limit=15.0 2023-11-24 02:44:18,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2646786.6666666665, ans=0.125 2023-11-24 02:44:18,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2646786.6666666665, ans=0.125 2023-11-24 02:44:24,871 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.517e+01 9.117e+01 1.003e+02 1.656e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 02:44:28,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2646853.3333333335, ans=0.125 2023-11-24 02:44:34,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2646920.0, ans=0.0 2023-11-24 02:44:35,767 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 250, loss[loss=0.04206, simple_loss=0.0538, pruned_loss=0.007825, audio_tagging_loss=0.007334, over 14565.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09184, pruned_loss=0.0134, audio_tagging_loss=0.01194, over 2183037.74 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:44:36,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2646920.0, ans=0.1 2023-11-24 02:44:39,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2646920.0, ans=0.0 2023-11-24 02:44:47,726 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397050 2023-11-24 02:44:57,637 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2646986.6666666665, ans=0.125 2023-11-24 02:45:06,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2647053.3333333335, ans=0.125 2023-11-24 02:45:11,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2647053.3333333335, ans=0.0 2023-11-24 02:45:13,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2647120.0, ans=0.125 2023-11-24 02:45:36,988 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 300, loss[loss=0.06139, simple_loss=0.08808, pruned_loss=0.00987, audio_tagging_loss=0.007484, over 15020.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09221, pruned_loss=0.01327, audio_tagging_loss=0.01102, over 2375287.58 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 02:45:50,874 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397100 2023-11-24 02:46:02,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2647386.6666666665, ans=0.1 2023-11-24 02:46:09,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2647386.6666666665, ans=0.0 2023-11-24 02:46:23,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2647453.3333333335, ans=0.2 2023-11-24 02:46:30,122 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.278e+01 8.816e+01 9.876e+01 1.220e+02, threshold=1.763e+02, percent-clipped=0.0 2023-11-24 02:46:32,599 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.07 vs. limit=15.0 2023-11-24 02:46:40,780 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 350, loss[loss=0.0543, simple_loss=0.06609, pruned_loss=0.01008, audio_tagging_loss=0.01118, over 15184.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09133, pruned_loss=0.0132, audio_tagging_loss=0.0105, over 2521059.95 frames. ], batch size: 61, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 02:46:49,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2647586.6666666665, ans=0.125 2023-11-24 02:46:52,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397150 2023-11-24 02:46:55,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2647653.3333333335, ans=0.125 2023-11-24 02:47:11,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=2647720.0, ans=0.2 2023-11-24 02:47:26,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2647786.6666666665, ans=0.125 2023-11-24 02:47:37,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2647853.3333333335, ans=0.0 2023-11-24 02:47:42,078 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 400, loss[loss=0.05537, simple_loss=0.0811, pruned_loss=0.00528, audio_tagging_loss=0.009541, over 15319.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09091, pruned_loss=0.01316, audio_tagging_loss=0.01017, over 2637020.43 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:47:47,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2647920.0, ans=0.2 2023-11-24 02:47:49,993 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0 2023-11-24 02:47:50,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2647920.0, ans=0.07 2023-11-24 02:47:52,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2647986.6666666665, ans=0.0 2023-11-24 02:47:54,098 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397200 2023-11-24 02:48:06,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2648053.3333333335, ans=0.125 2023-11-24 02:48:23,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2648120.0, ans=0.125 2023-11-24 02:48:35,269 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.562e+01 9.066e+01 9.786e+01 1.375e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-24 02:48:44,725 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 450, loss[loss=0.08806, simple_loss=0.1253, pruned_loss=0.01897, audio_tagging_loss=0.006453, over 16884.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09114, pruned_loss=0.01324, audio_tagging_loss=0.009857, over 2729560.08 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:48:57,569 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397250 2023-11-24 02:49:11,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2648386.6666666665, ans=0.125 2023-11-24 02:49:21,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2648453.3333333335, ans=0.125 2023-11-24 02:49:46,800 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 500, loss[loss=0.06098, simple_loss=0.07827, pruned_loss=0.01067, audio_tagging_loss=0.01117, over 14935.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09112, pruned_loss=0.01342, audio_tagging_loss=0.009618, over 2798304.44 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:49:50,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2648586.6666666665, ans=0.5 2023-11-24 02:49:52,233 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.89 vs. limit=22.5 2023-11-24 02:49:59,254 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397300 2023-11-24 02:50:09,238 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.01 vs. limit=15.0 2023-11-24 02:50:16,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2648720.0, ans=0.2 2023-11-24 02:50:19,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2648720.0, ans=0.0 2023-11-24 02:50:32,081 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 02:50:35,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2648853.3333333335, ans=0.125 2023-11-24 02:50:38,758 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.347e+01 9.082e+01 9.817e+01 1.239e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-24 02:50:48,952 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 550, loss[loss=0.03787, simple_loss=0.04086, pruned_loss=0.004472, audio_tagging_loss=0.01296, over 13809.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09083, pruned_loss=0.01335, audio_tagging_loss=0.009503, over 2846072.46 frames. ], batch size: 55, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:50:54,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2648920.0, ans=0.1 2023-11-24 02:51:01,011 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397350 2023-11-24 02:51:01,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.50 vs. limit=10.0 2023-11-24 02:51:05,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2648986.6666666665, ans=0.125 2023-11-24 02:51:12,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2649053.3333333335, ans=0.125 2023-11-24 02:51:18,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2649053.3333333335, ans=0.04949747468305833 2023-11-24 02:51:40,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2649186.6666666665, ans=0.0 2023-11-24 02:51:46,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2649186.6666666665, ans=0.0 2023-11-24 02:51:49,959 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 600, loss[loss=0.08473, simple_loss=0.1146, pruned_loss=0.0152, audio_tagging_loss=0.01222, over 15110.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09145, pruned_loss=0.01338, audio_tagging_loss=0.009412, over 2897416.88 frames. ], batch size: 54, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:52:03,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397400 2023-11-24 02:52:09,701 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.63 vs. limit=10.0 2023-11-24 02:52:27,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2649453.3333333335, ans=0.0 2023-11-24 02:52:29,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2649453.3333333335, ans=0.125 2023-11-24 02:52:36,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2649453.3333333335, ans=0.05 2023-11-24 02:52:42,871 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.375e+01 9.177e+01 9.802e+01 2.402e+02, threshold=1.835e+02, percent-clipped=1.0 2023-11-24 02:52:52,986 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 650, loss[loss=0.06578, simple_loss=0.08774, pruned_loss=0.01161, audio_tagging_loss=0.0103, over 16603.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09158, pruned_loss=0.01336, audio_tagging_loss=0.009381, over 2928972.35 frames. ], batch size: 61, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 02:53:02,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2649586.6666666665, ans=0.95 2023-11-24 02:53:04,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397450 2023-11-24 02:53:18,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2649720.0, ans=0.125 2023-11-24 02:53:18,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2649720.0, ans=0.2 2023-11-24 02:53:40,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.59 vs. limit=10.0 2023-11-24 02:53:47,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.96 vs. limit=15.0 2023-11-24 02:53:51,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2649853.3333333335, ans=0.0 2023-11-24 02:53:52,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2649853.3333333335, ans=0.125 2023-11-24 02:53:54,373 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 700, loss[loss=0.09209, simple_loss=0.1233, pruned_loss=0.02165, audio_tagging_loss=0.008782, over 15583.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.0922, pruned_loss=0.01348, audio_tagging_loss=0.009258, over 2961389.45 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 02:53:55,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2649920.0, ans=0.125 2023-11-24 02:54:06,951 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397500 2023-11-24 02:54:10,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2649986.6666666665, ans=0.2 2023-11-24 02:54:11,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.74 vs. limit=22.5 2023-11-24 02:54:16,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2649986.6666666665, ans=0.125 2023-11-24 02:54:27,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2650053.3333333335, ans=0.125 2023-11-24 02:54:47,552 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.698e+01 9.188e+01 9.937e+01 1.605e+02, threshold=1.838e+02, percent-clipped=0.0 2023-11-24 02:54:55,823 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 750, loss[loss=0.07398, simple_loss=0.09393, pruned_loss=0.01701, audio_tagging_loss=0.01001, over 14987.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09182, pruned_loss=0.01341, audio_tagging_loss=0.009255, over 2981446.88 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 02:55:06,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2650253.3333333335, ans=0.95 2023-11-24 02:55:09,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397550 2023-11-24 02:55:35,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2650453.3333333335, ans=0.125 2023-11-24 02:55:47,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2650520.0, ans=0.2 2023-11-24 02:55:47,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2650520.0, ans=0.025 2023-11-24 02:55:58,170 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 800, loss[loss=0.07973, simple_loss=0.1183, pruned_loss=0.01336, audio_tagging_loss=0.007232, over 16147.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.0922, pruned_loss=0.01348, audio_tagging_loss=0.00924, over 2995706.10 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:56:06,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0 2023-11-24 02:56:10,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397600 2023-11-24 02:56:13,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2650653.3333333335, ans=0.125 2023-11-24 02:56:34,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.34 vs. limit=15.0 2023-11-24 02:56:41,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2650786.6666666665, ans=0.125 2023-11-24 02:56:50,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.44 vs. limit=22.5 2023-11-24 02:56:51,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 8.448e+01 9.134e+01 9.872e+01 1.389e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 02:56:59,554 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 850, loss[loss=0.07953, simple_loss=0.1041, pruned_loss=0.01779, audio_tagging_loss=0.009691, over 15683.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09249, pruned_loss=0.01358, audio_tagging_loss=0.009347, over 3003211.19 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0 2023-11-24 02:57:03,958 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.07 vs. limit=12.0 2023-11-24 02:57:05,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.35 vs. limit=22.5 2023-11-24 02:57:12,021 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397650 2023-11-24 02:57:20,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2650986.6666666665, ans=0.1 2023-11-24 02:57:37,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.07 vs. limit=6.0 2023-11-24 02:57:45,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2651120.0, ans=0.05 2023-11-24 02:58:02,068 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 900, loss[loss=0.08374, simple_loss=0.115, pruned_loss=0.01828, audio_tagging_loss=0.007955, over 15089.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09217, pruned_loss=0.01351, audio_tagging_loss=0.009465, over 3011668.00 frames. ], batch size: 52, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 02:58:04,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2651253.3333333335, ans=0.1 2023-11-24 02:58:14,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397700 2023-11-24 02:58:15,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2651320.0, ans=0.1 2023-11-24 02:58:32,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2651386.6666666665, ans=0.0 2023-11-24 02:58:33,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2651386.6666666665, ans=0.1 2023-11-24 02:58:38,686 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.54 vs. limit=12.0 2023-11-24 02:58:39,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2651453.3333333335, ans=0.0 2023-11-24 02:58:42,237 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.40 vs. limit=15.0 2023-11-24 02:58:50,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2651520.0, ans=0.125 2023-11-24 02:58:53,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2651520.0, ans=0.0 2023-11-24 02:58:56,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.194e+01 8.559e+01 9.218e+01 1.010e+02 1.214e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 02:58:58,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.05 vs. limit=15.0 2023-11-24 02:59:01,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0 2023-11-24 02:59:04,565 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 950, loss[loss=0.05662, simple_loss=0.08012, pruned_loss=0.00742, audio_tagging_loss=0.009144, over 13550.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09226, pruned_loss=0.01341, audio_tagging_loss=0.009364, over 3018685.77 frames. ], batch size: 52, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 02:59:17,039 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397750 2023-11-24 02:59:22,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2651653.3333333335, ans=0.125 2023-11-24 02:59:26,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2651653.3333333335, ans=0.125 2023-11-24 02:59:36,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2651720.0, ans=0.125 2023-11-24 02:59:45,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2651786.6666666665, ans=0.2 2023-11-24 02:59:45,347 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.54 vs. limit=22.5 2023-11-24 02:59:47,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2651786.6666666665, ans=0.0 2023-11-24 03:00:03,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2651853.3333333335, ans=0.0 2023-11-24 03:00:06,454 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1000, loss[loss=0.07082, simple_loss=0.09671, pruned_loss=0.01318, audio_tagging_loss=0.00929, over 15968.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09198, pruned_loss=0.01336, audio_tagging_loss=0.009115, over 3024199.72 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:00:09,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2651920.0, ans=0.125 2023-11-24 03:00:18,402 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397800 2023-11-24 03:00:19,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2651986.6666666665, ans=0.0 2023-11-24 03:00:19,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2651986.6666666665, ans=0.125 2023-11-24 03:00:32,261 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:00:32,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.56 vs. limit=10.0 2023-11-24 03:00:34,306 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:00:35,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.70 vs. limit=15.0 2023-11-24 03:01:01,760 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.415e+01 8.930e+01 9.689e+01 1.161e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 03:01:04,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2652186.6666666665, ans=0.1 2023-11-24 03:01:09,026 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1050, loss[loss=0.07002, simple_loss=0.08142, pruned_loss=0.01419, audio_tagging_loss=0.01512, over 13609.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09209, pruned_loss=0.01348, audio_tagging_loss=0.008942, over 3021321.95 frames. ], batch size: 53, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:01:21,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397850 2023-11-24 03:01:38,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.11 vs. limit=15.0 2023-11-24 03:01:52,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.11 vs. limit=15.0 2023-11-24 03:01:55,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.86 vs. limit=10.0 2023-11-24 03:02:11,741 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1100, loss[loss=0.08023, simple_loss=0.1029, pruned_loss=0.01727, audio_tagging_loss=0.01153, over 15599.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09092, pruned_loss=0.01331, audio_tagging_loss=0.00902, over 3023850.64 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:02:13,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2652586.6666666665, ans=0.1 2023-11-24 03:02:15,978 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:02:16,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2652586.6666666665, ans=0.125 2023-11-24 03:02:17,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2652586.6666666665, ans=0.0 2023-11-24 03:02:19,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2652586.6666666665, ans=0.1 2023-11-24 03:02:24,338 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397900 2023-11-24 03:02:30,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2652653.3333333335, ans=0.125 2023-11-24 03:03:06,344 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.355e+01 8.539e+01 9.041e+01 9.654e+01 1.220e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 03:03:07,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2652853.3333333335, ans=0.0 2023-11-24 03:03:13,515 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1150, loss[loss=0.06104, simple_loss=0.07446, pruned_loss=0.01293, audio_tagging_loss=0.01088, over 14617.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09079, pruned_loss=0.01317, audio_tagging_loss=0.008957, over 3022481.11 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:03:17,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.09 vs. limit=22.5 2023-11-24 03:03:17,829 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.30 vs. limit=10.0 2023-11-24 03:03:25,502 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 397950 2023-11-24 03:03:43,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2653053.3333333335, ans=0.0 2023-11-24 03:03:44,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.56 vs. limit=22.5 2023-11-24 03:03:47,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2653053.3333333335, ans=0.1 2023-11-24 03:04:03,089 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.80 vs. limit=15.0 2023-11-24 03:04:03,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2653186.6666666665, ans=0.0 2023-11-24 03:04:06,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2653186.6666666665, ans=0.0 2023-11-24 03:04:10,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2653186.6666666665, ans=0.0 2023-11-24 03:04:14,295 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1200, loss[loss=0.08036, simple_loss=0.1065, pruned_loss=0.01879, audio_tagging_loss=0.008343, over 14846.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09057, pruned_loss=0.01336, audio_tagging_loss=0.008912, over 3032565.33 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:04:26,352 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398000 2023-11-24 03:04:43,044 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.37 vs. limit=15.0 2023-11-24 03:04:59,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2653453.3333333335, ans=0.0 2023-11-24 03:05:09,795 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.593e+01 9.162e+01 9.902e+01 1.327e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 03:05:16,318 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1250, loss[loss=0.06386, simple_loss=0.08684, pruned_loss=0.01325, audio_tagging_loss=0.007196, over 14139.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09116, pruned_loss=0.01346, audio_tagging_loss=0.008766, over 3032276.43 frames. ], batch size: 53, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:05:27,655 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.11 vs. limit=10.0 2023-11-24 03:05:29,469 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398050 2023-11-24 03:05:30,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2653653.3333333335, ans=0.0 2023-11-24 03:05:41,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2653720.0, ans=0.125 2023-11-24 03:05:54,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2653786.6666666665, ans=0.0 2023-11-24 03:05:57,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2653786.6666666665, ans=0.0 2023-11-24 03:05:58,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2653786.6666666665, ans=0.1 2023-11-24 03:06:13,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2653853.3333333335, ans=0.125 2023-11-24 03:06:18,466 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1300, loss[loss=0.07169, simple_loss=0.1022, pruned_loss=0.01272, audio_tagging_loss=0.007852, over 15908.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09053, pruned_loss=0.0132, audio_tagging_loss=0.008806, over 3029148.92 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:06:18,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2653920.0, ans=0.125 2023-11-24 03:06:30,364 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398100 2023-11-24 03:07:09,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2654186.6666666665, ans=0.09899494936611666 2023-11-24 03:07:13,604 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.353e+01 8.205e+01 8.973e+01 9.705e+01 1.641e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-24 03:07:15,651 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.27 vs. limit=22.5 2023-11-24 03:07:17,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=2654186.6666666665, ans=0.5 2023-11-24 03:07:19,490 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1350, loss[loss=0.06216, simple_loss=0.08726, pruned_loss=0.01209, audio_tagging_loss=0.006438, over 16429.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09135, pruned_loss=0.0134, audio_tagging_loss=0.008812, over 3034368.11 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:07:31,311 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398150 2023-11-24 03:07:48,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.49 vs. limit=6.0 2023-11-24 03:07:55,914 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:08:03,978 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:08:05,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2654453.3333333335, ans=0.125 2023-11-24 03:08:09,512 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.15 vs. limit=15.0 2023-11-24 03:08:20,664 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1400, loss[loss=0.04573, simple_loss=0.05674, pruned_loss=0.006782, audio_tagging_loss=0.01058, over 14992.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09099, pruned_loss=0.01339, audio_tagging_loss=0.008972, over 3037636.60 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:08:34,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398200 2023-11-24 03:08:48,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.23 vs. limit=15.0 2023-11-24 03:08:50,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2654720.0, ans=0.1 2023-11-24 03:08:54,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.12 vs. limit=12.0 2023-11-24 03:09:04,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2654786.6666666665, ans=0.125 2023-11-24 03:09:15,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2654853.3333333335, ans=0.1 2023-11-24 03:09:17,338 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.476e+01 9.016e+01 9.845e+01 1.286e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-24 03:09:21,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2654853.3333333335, ans=0.2 2023-11-24 03:09:24,004 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1450, loss[loss=0.07222, simple_loss=0.0843, pruned_loss=0.01735, audio_tagging_loss=0.01272, over 15287.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09165, pruned_loss=0.01353, audio_tagging_loss=0.009058, over 3040285.07 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:09:27,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2654920.0, ans=0.125 2023-11-24 03:09:35,974 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398250 2023-11-24 03:09:39,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2654986.6666666665, ans=0.125 2023-11-24 03:09:39,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2654986.6666666665, ans=0.125 2023-11-24 03:09:53,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2655053.3333333335, ans=0.125 2023-11-24 03:09:54,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2655053.3333333335, ans=0.0 2023-11-24 03:09:57,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2655053.3333333335, ans=0.2 2023-11-24 03:10:14,336 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.85 vs. limit=15.0 2023-11-24 03:10:19,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2655186.6666666665, ans=0.0 2023-11-24 03:10:25,530 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1500, loss[loss=0.08223, simple_loss=0.1131, pruned_loss=0.01938, audio_tagging_loss=0.00631, over 16934.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09031, pruned_loss=0.0133, audio_tagging_loss=0.009082, over 3036340.86 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:10:37,542 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398300 2023-11-24 03:10:42,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2655320.0, ans=0.1 2023-11-24 03:10:42,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2655320.0, ans=0.125 2023-11-24 03:11:06,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.24 vs. limit=15.0 2023-11-24 03:11:21,389 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.576e+01 9.222e+01 1.014e+02 1.240e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 03:11:22,911 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:11:25,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2655520.0, ans=0.0 2023-11-24 03:11:27,359 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1550, loss[loss=0.06499, simple_loss=0.08786, pruned_loss=0.01068, audio_tagging_loss=0.01039, over 15961.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09139, pruned_loss=0.01348, audio_tagging_loss=0.009124, over 3047507.50 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0 2023-11-24 03:11:41,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398350 2023-11-24 03:12:12,623 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.46 vs. limit=10.0 2023-11-24 03:12:16,197 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.79 vs. limit=22.5 2023-11-24 03:12:31,485 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1600, loss[loss=0.0799, simple_loss=0.1112, pruned_loss=0.01661, audio_tagging_loss=0.007662, over 15896.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09104, pruned_loss=0.01353, audio_tagging_loss=0.009238, over 3040762.62 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:12:40,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2655920.0, ans=0.0 2023-11-24 03:12:42,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2655986.6666666665, ans=0.1 2023-11-24 03:12:44,000 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398400 2023-11-24 03:12:59,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2656053.3333333335, ans=0.0 2023-11-24 03:13:06,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2656053.3333333335, ans=0.04949747468305833 2023-11-24 03:13:08,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2656120.0, ans=0.125 2023-11-24 03:13:21,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2656186.6666666665, ans=0.125 2023-11-24 03:13:28,026 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.243e+01 8.406e+01 9.091e+01 9.636e+01 1.326e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-24 03:13:30,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2656186.6666666665, ans=0.125 2023-11-24 03:13:33,937 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1650, loss[loss=0.06314, simple_loss=0.0749, pruned_loss=0.0164, audio_tagging_loss=0.009283, over 13421.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09067, pruned_loss=0.01341, audio_tagging_loss=0.009352, over 3039084.77 frames. ], batch size: 52, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:13:45,985 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398450 2023-11-24 03:13:48,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2656320.0, ans=0.125 2023-11-24 03:13:51,101 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:14:03,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2656386.6666666665, ans=0.125 2023-11-24 03:14:08,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2656386.6666666665, ans=0.09899494936611666 2023-11-24 03:14:15,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2656453.3333333335, ans=0.5 2023-11-24 03:14:24,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2656520.0, ans=0.0 2023-11-24 03:14:31,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2656520.0, ans=0.1 2023-11-24 03:14:32,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.05 vs. limit=12.0 2023-11-24 03:14:36,429 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1700, loss[loss=0.07998, simple_loss=0.1057, pruned_loss=0.01831, audio_tagging_loss=0.008806, over 15233.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09116, pruned_loss=0.01344, audio_tagging_loss=0.009223, over 3049581.09 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:14:37,009 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.93 vs. limit=10.0 2023-11-24 03:14:43,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2656586.6666666665, ans=0.1 2023-11-24 03:14:49,108 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398500 2023-11-24 03:14:51,215 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.10 vs. limit=15.0 2023-11-24 03:14:58,999 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.67 vs. limit=10.0 2023-11-24 03:15:29,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2656853.3333333335, ans=0.0 2023-11-24 03:15:32,718 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.306e+01 8.348e+01 8.790e+01 9.631e+01 1.141e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-24 03:15:39,522 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1750, loss[loss=0.08854, simple_loss=0.1202, pruned_loss=0.01939, audio_tagging_loss=0.009049, over 14976.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09154, pruned_loss=0.01341, audio_tagging_loss=0.00912, over 3059868.30 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:15:51,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398550 2023-11-24 03:15:51,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2656986.6666666665, ans=0.09899494936611666 2023-11-24 03:16:06,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2657053.3333333335, ans=0.1 2023-11-24 03:16:27,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2657120.0, ans=0.125 2023-11-24 03:16:29,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2657186.6666666665, ans=0.0 2023-11-24 03:16:30,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2657186.6666666665, ans=0.125 2023-11-24 03:16:41,191 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1800, loss[loss=0.0399, simple_loss=0.05343, pruned_loss=0.006429, audio_tagging_loss=0.006757, over 14872.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09187, pruned_loss=0.0134, audio_tagging_loss=0.008976, over 3058558.61 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:16:44,445 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:16:48,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2657253.3333333335, ans=0.125 2023-11-24 03:16:53,729 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398600 2023-11-24 03:16:56,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2657320.0, ans=0.125 2023-11-24 03:17:23,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.08 vs. limit=15.0 2023-11-24 03:17:28,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2657453.3333333335, ans=0.125 2023-11-24 03:17:30,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2657520.0, ans=0.0 2023-11-24 03:17:31,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2657520.0, ans=0.0 2023-11-24 03:17:37,904 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.560e+01 9.161e+01 9.912e+01 1.351e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 03:17:39,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.91 vs. limit=22.5 2023-11-24 03:17:43,954 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1850, loss[loss=0.07026, simple_loss=0.09056, pruned_loss=0.0155, audio_tagging_loss=0.009475, over 15475.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09211, pruned_loss=0.01332, audio_tagging_loss=0.008889, over 3056511.08 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:17:54,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2657586.6666666665, ans=0.0 2023-11-24 03:17:56,715 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398650 2023-11-24 03:18:07,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2657653.3333333335, ans=0.1 2023-11-24 03:18:07,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2657653.3333333335, ans=0.125 2023-11-24 03:18:18,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2657720.0, ans=0.1 2023-11-24 03:18:39,520 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:18:47,049 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1900, loss[loss=0.06871, simple_loss=0.09788, pruned_loss=0.01193, audio_tagging_loss=0.007844, over 15189.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09245, pruned_loss=0.01344, audio_tagging_loss=0.008726, over 3052700.71 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:18:49,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2657920.0, ans=0.0 2023-11-24 03:18:55,768 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:18:59,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398700 2023-11-24 03:19:23,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2658120.0, ans=0.1 2023-11-24 03:19:39,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2658186.6666666665, ans=0.125 2023-11-24 03:19:43,000 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.306e+01 8.939e+01 9.602e+01 1.298e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-24 03:19:47,778 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 1950, loss[loss=0.08649, simple_loss=0.1157, pruned_loss=0.01531, audio_tagging_loss=0.01333, over 15346.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09137, pruned_loss=0.01329, audio_tagging_loss=0.008781, over 3042454.03 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:19:59,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2658320.0, ans=0.0 2023-11-24 03:20:00,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398750 2023-11-24 03:20:00,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2658320.0, ans=0.125 2023-11-24 03:20:07,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2658320.0, ans=0.04949747468305833 2023-11-24 03:20:14,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2658386.6666666665, ans=0.0 2023-11-24 03:20:27,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.16 vs. limit=15.0 2023-11-24 03:20:47,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2658520.0, ans=0.125 2023-11-24 03:20:49,783 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2000, loss[loss=0.06559, simple_loss=0.08334, pruned_loss=0.0158, audio_tagging_loss=0.008118, over 14953.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09069, pruned_loss=0.01329, audio_tagging_loss=0.008932, over 3040013.47 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:20:59,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.30 vs. limit=10.0 2023-11-24 03:20:59,100 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.46 vs. limit=22.5 2023-11-24 03:21:01,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2658653.3333333335, ans=0.0 2023-11-24 03:21:02,289 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398800 2023-11-24 03:21:09,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2658653.3333333335, ans=0.1 2023-11-24 03:21:33,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2658786.6666666665, ans=0.07 2023-11-24 03:21:45,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2658853.3333333335, ans=0.0 2023-11-24 03:21:46,679 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.321e+01 9.002e+01 9.770e+01 1.545e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-24 03:21:49,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2658853.3333333335, ans=0.125 2023-11-24 03:21:51,427 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2050, loss[loss=0.06973, simple_loss=0.08783, pruned_loss=0.01483, audio_tagging_loss=0.01098, over 14773.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09017, pruned_loss=0.01341, audio_tagging_loss=0.008941, over 3027681.79 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:22:02,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2658920.0, ans=0.125 2023-11-24 03:22:04,427 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398850 2023-11-24 03:22:06,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2658986.6666666665, ans=0.0 2023-11-24 03:22:25,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2659053.3333333335, ans=0.125 2023-11-24 03:22:38,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2659120.0, ans=0.95 2023-11-24 03:22:52,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2659186.6666666665, ans=0.1 2023-11-24 03:22:54,378 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2100, loss[loss=0.03808, simple_loss=0.04463, pruned_loss=0.005967, audio_tagging_loss=0.009798, over 15856.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09112, pruned_loss=0.01351, audio_tagging_loss=0.008868, over 3041102.03 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:23:06,745 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398900 2023-11-24 03:23:14,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2659320.0, ans=0.125 2023-11-24 03:23:52,488 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.464e+01 8.942e+01 9.567e+01 1.124e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-24 03:23:56,162 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2150, loss[loss=0.08018, simple_loss=0.0983, pruned_loss=0.02061, audio_tagging_loss=0.01042, over 15771.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09116, pruned_loss=0.01339, audio_tagging_loss=0.008949, over 3043259.01 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:24:04,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2659586.6666666665, ans=0.2 2023-11-24 03:24:09,334 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 398950 2023-11-24 03:24:34,091 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:24:40,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.51 vs. limit=15.0 2023-11-24 03:24:50,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2659853.3333333335, ans=0.0 2023-11-24 03:24:51,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2659853.3333333335, ans=0.125 2023-11-24 03:24:59,228 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2200, loss[loss=0.06048, simple_loss=0.0865, pruned_loss=0.00846, audio_tagging_loss=0.008764, over 15123.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09069, pruned_loss=0.01334, audio_tagging_loss=0.008962, over 3044247.83 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:25:12,636 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399000 2023-11-24 03:25:58,368 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.513e+01 9.169e+01 1.008e+02 1.249e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-24 03:26:01,966 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2250, loss[loss=0.05502, simple_loss=0.07111, pruned_loss=0.01075, audio_tagging_loss=0.008709, over 14716.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09212, pruned_loss=0.01365, audio_tagging_loss=0.008969, over 3045235.63 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:26:04,530 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:26:13,844 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399050 2023-11-24 03:26:21,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2660320.0, ans=0.1 2023-11-24 03:26:47,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2660453.3333333335, ans=0.125 2023-11-24 03:26:48,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2660453.3333333335, ans=10.0 2023-11-24 03:27:01,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2660586.6666666665, ans=0.125 2023-11-24 03:27:02,882 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2300, loss[loss=0.04748, simple_loss=0.06377, pruned_loss=0.006878, audio_tagging_loss=0.008716, over 15467.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09212, pruned_loss=0.01369, audio_tagging_loss=0.008994, over 3042126.63 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:27:09,430 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.89 vs. limit=15.0 2023-11-24 03:27:14,865 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399100 2023-11-24 03:27:14,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2660653.3333333335, ans=0.125 2023-11-24 03:27:26,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2660653.3333333335, ans=0.125 2023-11-24 03:27:48,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2660786.6666666665, ans=0.125 2023-11-24 03:27:57,453 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:27:58,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.15 vs. limit=22.5 2023-11-24 03:28:00,933 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.716e+01 8.488e+01 8.965e+01 9.835e+01 1.215e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 03:28:05,083 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2350, loss[loss=0.06653, simple_loss=0.09373, pruned_loss=0.01183, audio_tagging_loss=0.00783, over 14505.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09148, pruned_loss=0.01354, audio_tagging_loss=0.009004, over 3042669.51 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:28:13,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2660920.0, ans=0.125 2023-11-24 03:28:18,714 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399150 2023-11-24 03:28:24,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2660986.6666666665, ans=0.125 2023-11-24 03:28:48,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2661120.0, ans=0.0 2023-11-24 03:28:49,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.29 vs. limit=15.0 2023-11-24 03:28:53,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0 2023-11-24 03:29:09,000 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2400, loss[loss=0.07814, simple_loss=0.1061, pruned_loss=0.01679, audio_tagging_loss=0.008307, over 15587.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.0911, pruned_loss=0.01337, audio_tagging_loss=0.009057, over 3037928.48 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:29:21,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399200 2023-11-24 03:29:21,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2661320.0, ans=0.0 2023-11-24 03:29:41,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2661386.6666666665, ans=0.1 2023-11-24 03:30:01,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2661520.0, ans=0.0 2023-11-24 03:30:08,342 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.386e+01 9.132e+01 1.001e+02 1.473e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 03:30:10,823 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2450, loss[loss=0.07672, simple_loss=0.1015, pruned_loss=0.01528, audio_tagging_loss=0.01068, over 15375.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09091, pruned_loss=0.01322, audio_tagging_loss=0.009087, over 3039061.49 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:30:22,916 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399250 2023-11-24 03:30:25,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2661653.3333333335, ans=0.035 2023-11-24 03:30:44,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2661720.0, ans=0.125 2023-11-24 03:30:55,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2661786.6666666665, ans=0.1 2023-11-24 03:31:00,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2661853.3333333335, ans=0.0 2023-11-24 03:31:07,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2661853.3333333335, ans=0.125 2023-11-24 03:31:11,906 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2500, loss[loss=0.06386, simple_loss=0.08359, pruned_loss=0.01282, audio_tagging_loss=0.009238, over 14523.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09092, pruned_loss=0.01322, audio_tagging_loss=0.009067, over 3038841.63 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:31:17,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2661920.0, ans=0.0 2023-11-24 03:31:18,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2661920.0, ans=0.2 2023-11-24 03:31:26,317 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399300 2023-11-24 03:31:40,108 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.62 vs. limit=15.0 2023-11-24 03:31:48,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=2662053.3333333335, ans=6.0 2023-11-24 03:31:49,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2662120.0, ans=0.2 2023-11-24 03:31:53,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2662120.0, ans=10.0 2023-11-24 03:32:01,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2662186.6666666665, ans=0.2 2023-11-24 03:32:06,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2662186.6666666665, ans=0.0 2023-11-24 03:32:13,411 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.419e+01 8.936e+01 9.796e+01 1.161e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-24 03:32:16,389 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2550, loss[loss=0.08933, simple_loss=0.1272, pruned_loss=0.02123, audio_tagging_loss=0.004531, over 16296.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.0911, pruned_loss=0.01331, audio_tagging_loss=0.009049, over 3039483.67 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:32:16,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2662253.3333333335, ans=0.2 2023-11-24 03:32:26,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2662253.3333333335, ans=0.125 2023-11-24 03:32:28,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399350 2023-11-24 03:33:06,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2662520.0, ans=0.0 2023-11-24 03:33:11,612 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:33:18,428 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2600, loss[loss=0.0592, simple_loss=0.07674, pruned_loss=0.01053, audio_tagging_loss=0.0103, over 14531.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09125, pruned_loss=0.0133, audio_tagging_loss=0.00896, over 3039060.60 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:33:20,213 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.81 vs. limit=15.0 2023-11-24 03:33:30,220 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399400 2023-11-24 03:33:50,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2662720.0, ans=0.0 2023-11-24 03:33:58,951 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:34:01,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2662786.6666666665, ans=0.05 2023-11-24 03:34:10,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2662853.3333333335, ans=0.0 2023-11-24 03:34:15,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2662853.3333333335, ans=0.125 2023-11-24 03:34:15,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2662853.3333333335, ans=0.2 2023-11-24 03:34:17,523 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.667e+01 9.247e+01 9.854e+01 1.236e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-24 03:34:19,852 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2650, loss[loss=0.07928, simple_loss=0.09263, pruned_loss=0.0211, audio_tagging_loss=0.01187, over 14377.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09128, pruned_loss=0.01335, audio_tagging_loss=0.008898, over 3039582.15 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:34:20,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2662920.0, ans=0.125 2023-11-24 03:34:23,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2662920.0, ans=0.125 2023-11-24 03:34:32,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399450 2023-11-24 03:34:47,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2663053.3333333335, ans=0.125 2023-11-24 03:35:11,886 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.37 vs. limit=22.5 2023-11-24 03:35:23,187 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2700, loss[loss=0.07765, simple_loss=0.1023, pruned_loss=0.0163, audio_tagging_loss=0.01022, over 14558.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09147, pruned_loss=0.01334, audio_tagging_loss=0.00884, over 3036293.17 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:35:35,659 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399500 2023-11-24 03:35:42,302 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.25 vs. limit=22.5 2023-11-24 03:35:52,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2663386.6666666665, ans=0.1 2023-11-24 03:35:59,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2663453.3333333335, ans=0.1 2023-11-24 03:36:01,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.38 vs. limit=8.0 2023-11-24 03:36:04,224 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.68 vs. limit=10.0 2023-11-24 03:36:05,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2663453.3333333335, ans=0.125 2023-11-24 03:36:23,395 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.382e+01 8.295e+01 8.758e+01 9.490e+01 1.330e+02, threshold=1.752e+02, percent-clipped=0.0 2023-11-24 03:36:25,807 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2750, loss[loss=0.06455, simple_loss=0.07854, pruned_loss=0.01215, audio_tagging_loss=0.01313, over 15653.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09069, pruned_loss=0.01326, audio_tagging_loss=0.008883, over 3038935.57 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:36:31,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2663586.6666666665, ans=0.125 2023-11-24 03:36:33,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2663586.6666666665, ans=0.125 2023-11-24 03:36:38,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399550 2023-11-24 03:36:39,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2663653.3333333335, ans=0.0 2023-11-24 03:36:39,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2663653.3333333335, ans=0.125 2023-11-24 03:37:18,089 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:37:19,722 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.38 vs. limit=22.5 2023-11-24 03:37:25,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2663853.3333333335, ans=0.125 2023-11-24 03:37:27,422 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2800, loss[loss=0.0523, simple_loss=0.07237, pruned_loss=0.008121, audio_tagging_loss=0.00799, over 15588.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09054, pruned_loss=0.0133, audio_tagging_loss=0.008949, over 3039820.76 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:37:35,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2663920.0, ans=0.125 2023-11-24 03:37:38,212 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.21 vs. limit=6.0 2023-11-24 03:37:39,649 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.76 vs. limit=6.0 2023-11-24 03:37:40,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399600 2023-11-24 03:38:01,504 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:38:27,682 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.351e+01 8.911e+01 9.699e+01 1.155e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-24 03:38:30,610 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2850, loss[loss=0.05188, simple_loss=0.06662, pruned_loss=0.01047, audio_tagging_loss=0.008103, over 15210.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09096, pruned_loss=0.01337, audio_tagging_loss=0.0088, over 3036984.98 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:38:42,936 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399650 2023-11-24 03:38:55,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2664386.6666666665, ans=0.125 2023-11-24 03:39:04,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2664386.6666666665, ans=0.0 2023-11-24 03:39:06,046 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:39:10,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2664453.3333333335, ans=0.125 2023-11-24 03:39:14,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=2664453.3333333335, ans=15.0 2023-11-24 03:39:19,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2664520.0, ans=0.125 2023-11-24 03:39:32,506 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2900, loss[loss=0.07928, simple_loss=0.1082, pruned_loss=0.01708, audio_tagging_loss=0.00808, over 15096.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09119, pruned_loss=0.01325, audio_tagging_loss=0.008857, over 3037747.52 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:39:40,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2664586.6666666665, ans=0.1 2023-11-24 03:39:45,044 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399700 2023-11-24 03:39:47,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2664653.3333333335, ans=0.125 2023-11-24 03:39:49,107 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.61 vs. limit=15.0 2023-11-24 03:39:59,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2664720.0, ans=0.125 2023-11-24 03:40:11,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2664786.6666666665, ans=0.125 2023-11-24 03:40:23,531 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.96 vs. limit=10.0 2023-11-24 03:40:26,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2664853.3333333335, ans=0.0 2023-11-24 03:40:31,990 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.311e+01 9.104e+01 9.838e+01 1.191e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 03:40:34,373 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 2950, loss[loss=0.06493, simple_loss=0.08037, pruned_loss=0.01415, audio_tagging_loss=0.0106, over 14799.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09219, pruned_loss=0.01343, audio_tagging_loss=0.008842, over 3043809.04 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:40:35,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2664920.0, ans=0.0 2023-11-24 03:40:44,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2664920.0, ans=0.125 2023-11-24 03:40:47,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399750 2023-11-24 03:41:17,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2665120.0, ans=0.2 2023-11-24 03:41:23,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2665186.6666666665, ans=0.125 2023-11-24 03:41:25,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2665186.6666666665, ans=0.0 2023-11-24 03:41:36,939 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3000, loss[loss=0.04777, simple_loss=0.06179, pruned_loss=0.007159, audio_tagging_loss=0.00971, over 15293.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09202, pruned_loss=0.01351, audio_tagging_loss=0.008958, over 3049987.86 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:41:36,940 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 03:42:18,354 INFO [train_asr.py:1253] (3/4) Epoch 34, validation: loss=0.05766, simple_loss=0.05087, pruned_loss=0.005081, audio_tagging_loss=0.02714, over 4681554.00 frames. 2023-11-24 03:42:18,355 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 03:42:31,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399800 2023-11-24 03:42:45,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2665386.6666666665, ans=0.125 2023-11-24 03:42:45,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2665386.6666666665, ans=0.07 2023-11-24 03:42:53,773 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:42:58,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2665453.3333333335, ans=0.125 2023-11-24 03:43:18,712 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.546e+01 8.558e+01 9.257e+01 9.837e+01 1.292e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 03:43:18,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2665520.0, ans=0.125 2023-11-24 03:43:21,068 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3050, loss[loss=0.07558, simple_loss=0.09952, pruned_loss=0.01597, audio_tagging_loss=0.00985, over 14664.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09343, pruned_loss=0.01378, audio_tagging_loss=0.009005, over 3047870.70 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:43:34,247 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399850 2023-11-24 03:43:39,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2665653.3333333335, ans=0.05 2023-11-24 03:43:48,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2665720.0, ans=0.0 2023-11-24 03:43:49,248 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=2665720.0, ans=0.1 2023-11-24 03:43:49,777 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.20 vs. limit=15.0 2023-11-24 03:43:53,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2665720.0, ans=0.0 2023-11-24 03:43:57,324 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:43:57,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2665786.6666666665, ans=0.125 2023-11-24 03:44:13,913 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.60 vs. limit=15.0 2023-11-24 03:44:18,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2665853.3333333335, ans=0.0 2023-11-24 03:44:23,998 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3100, loss[loss=0.07706, simple_loss=0.1, pruned_loss=0.01648, audio_tagging_loss=0.01057, over 15786.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09308, pruned_loss=0.01376, audio_tagging_loss=0.00912, over 3050280.20 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:44:30,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.90 vs. limit=12.0 2023-11-24 03:44:36,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399900 2023-11-24 03:44:41,756 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.05 vs. limit=12.0 2023-11-24 03:45:09,311 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.28 vs. limit=15.0 2023-11-24 03:45:23,796 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.765e+01 9.339e+01 1.018e+02 1.384e+02, threshold=1.868e+02, percent-clipped=0.0 2023-11-24 03:45:25,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2666253.3333333335, ans=0.125 2023-11-24 03:45:26,231 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3150, loss[loss=0.06676, simple_loss=0.09953, pruned_loss=0.01089, audio_tagging_loss=0.006102, over 15986.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09336, pruned_loss=0.01378, audio_tagging_loss=0.00908, over 3051006.19 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:45:34,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2666253.3333333335, ans=0.0 2023-11-24 03:45:38,322 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 399950 2023-11-24 03:45:43,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2666320.0, ans=0.2 2023-11-24 03:45:44,829 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:45:46,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2666320.0, ans=0.125 2023-11-24 03:46:03,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2666453.3333333335, ans=0.125 2023-11-24 03:46:24,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2666520.0, ans=0.04949747468305833 2023-11-24 03:46:28,857 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3200, loss[loss=0.07597, simple_loss=0.1053, pruned_loss=0.01467, audio_tagging_loss=0.008638, over 14846.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09318, pruned_loss=0.01382, audio_tagging_loss=0.009181, over 3050523.21 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:46:29,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2666586.6666666665, ans=0.125 2023-11-24 03:46:34,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.02 vs. limit=22.5 2023-11-24 03:46:41,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400000 2023-11-24 03:47:03,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.24 vs. limit=22.5 2023-11-24 03:47:17,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2666786.6666666665, ans=0.125 2023-11-24 03:47:21,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2666853.3333333335, ans=0.1 2023-11-24 03:47:24,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2666853.3333333335, ans=0.0 2023-11-24 03:47:31,742 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.504e+01 9.216e+01 9.938e+01 1.136e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 03:47:34,784 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3250, loss[loss=0.04827, simple_loss=0.069, pruned_loss=0.004671, audio_tagging_loss=0.009098, over 14559.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09234, pruned_loss=0.01378, audio_tagging_loss=0.009229, over 3041782.63 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:47:47,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400050 2023-11-24 03:47:50,227 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.90 vs. limit=15.0 2023-11-24 03:47:50,536 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.45 vs. limit=10.0 2023-11-24 03:47:53,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2666986.6666666665, ans=10.0 2023-11-24 03:47:56,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2666986.6666666665, ans=0.125 2023-11-24 03:48:25,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2667186.6666666665, ans=0.1 2023-11-24 03:48:26,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2667186.6666666665, ans=0.2 2023-11-24 03:48:37,194 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3300, loss[loss=0.08047, simple_loss=0.1148, pruned_loss=0.01502, audio_tagging_loss=0.008059, over 15380.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09238, pruned_loss=0.01375, audio_tagging_loss=0.009241, over 3043357.78 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:48:49,250 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400100 2023-11-24 03:49:00,062 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.24 vs. limit=15.0 2023-11-24 03:49:11,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2667386.6666666665, ans=0.0 2023-11-24 03:49:17,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2667453.3333333335, ans=10.0 2023-11-24 03:49:32,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2667520.0, ans=0.125 2023-11-24 03:49:35,233 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:49:37,335 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.541e+01 9.179e+01 9.840e+01 1.429e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-24 03:49:38,562 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3350, loss[loss=0.04803, simple_loss=0.06448, pruned_loss=0.006559, audio_tagging_loss=0.009232, over 15226.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09278, pruned_loss=0.01384, audio_tagging_loss=0.009188, over 3041781.38 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:49:46,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2667586.6666666665, ans=0.0 2023-11-24 03:49:50,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2667653.3333333335, ans=0.125 2023-11-24 03:49:51,078 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.14 vs. limit=12.0 2023-11-24 03:49:51,826 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400150 2023-11-24 03:49:57,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2667653.3333333335, ans=0.125 2023-11-24 03:50:00,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2667653.3333333335, ans=0.09899494936611666 2023-11-24 03:50:03,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2667720.0, ans=0.125 2023-11-24 03:50:12,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2667720.0, ans=0.125 2023-11-24 03:50:24,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2667786.6666666665, ans=0.2 2023-11-24 03:50:25,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2667786.6666666665, ans=0.0 2023-11-24 03:50:26,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2667786.6666666665, ans=0.1 2023-11-24 03:50:40,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2667920.0, ans=0.0 2023-11-24 03:50:41,767 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3400, loss[loss=0.05375, simple_loss=0.06523, pruned_loss=0.01212, audio_tagging_loss=0.00902, over 16052.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09302, pruned_loss=0.01391, audio_tagging_loss=0.009032, over 3046774.43 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:50:46,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2667920.0, ans=0.125 2023-11-24 03:50:54,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400200 2023-11-24 03:51:02,712 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.39 vs. limit=22.5 2023-11-24 03:51:24,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2668120.0, ans=0.0 2023-11-24 03:51:26,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2668120.0, ans=0.09899494936611666 2023-11-24 03:51:31,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2668186.6666666665, ans=0.2 2023-11-24 03:51:37,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2668186.6666666665, ans=0.125 2023-11-24 03:51:43,866 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.468e+01 9.078e+01 9.580e+01 1.134e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-24 03:51:45,115 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3450, loss[loss=0.06731, simple_loss=0.09808, pruned_loss=0.008334, audio_tagging_loss=0.009936, over 14878.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09277, pruned_loss=0.01376, audio_tagging_loss=0.008955, over 3042342.92 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:51:55,273 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.79 vs. limit=6.0 2023-11-24 03:51:57,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400250 2023-11-24 03:52:00,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2668320.0, ans=0.125 2023-11-24 03:52:15,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2668386.6666666665, ans=0.0 2023-11-24 03:52:24,422 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.08 vs. limit=15.0 2023-11-24 03:52:47,000 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3500, loss[loss=0.05621, simple_loss=0.06242, pruned_loss=0.0149, audio_tagging_loss=0.01011, over 14923.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09155, pruned_loss=0.01353, audio_tagging_loss=0.008993, over 3046074.55 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:52:51,127 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.01 vs. limit=15.0 2023-11-24 03:52:54,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2668586.6666666665, ans=0.125 2023-11-24 03:52:54,711 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=15.07 vs. limit=15.0 2023-11-24 03:52:59,027 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400300 2023-11-24 03:53:05,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2668653.3333333335, ans=0.1 2023-11-24 03:53:18,510 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:53:47,629 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.784e+01 9.365e+01 1.022e+02 1.307e+02, threshold=1.873e+02, percent-clipped=0.0 2023-11-24 03:53:48,877 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3550, loss[loss=0.05609, simple_loss=0.07547, pruned_loss=0.01086, audio_tagging_loss=0.007504, over 15284.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09062, pruned_loss=0.01341, audio_tagging_loss=0.008981, over 3052624.27 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:53:50,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2668920.0, ans=0.125 2023-11-24 03:54:02,715 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400350 2023-11-24 03:54:02,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2668986.6666666665, ans=0.0 2023-11-24 03:54:20,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.86 vs. limit=22.5 2023-11-24 03:54:26,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2669120.0, ans=0.125 2023-11-24 03:54:52,802 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3600, loss[loss=0.07086, simple_loss=0.1002, pruned_loss=0.01297, audio_tagging_loss=0.007808, over 15587.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09013, pruned_loss=0.01313, audio_tagging_loss=0.008995, over 3052442.19 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 03:54:59,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2669253.3333333335, ans=0.125 2023-11-24 03:55:04,833 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400400 2023-11-24 03:55:11,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2669320.0, ans=0.0 2023-11-24 03:55:26,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2669386.6666666665, ans=0.125 2023-11-24 03:55:39,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.18 vs. limit=15.0 2023-11-24 03:55:41,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.33 vs. limit=15.0 2023-11-24 03:55:52,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2669520.0, ans=0.0 2023-11-24 03:55:53,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2669586.6666666665, ans=0.125 2023-11-24 03:55:54,321 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.332e+01 8.867e+01 9.623e+01 1.551e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-24 03:55:54,365 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3650, loss[loss=0.06234, simple_loss=0.08612, pruned_loss=0.009541, audio_tagging_loss=0.00974, over 14467.00 frames. ], tot_loss[loss=0.06665, simple_loss=0.08952, pruned_loss=0.01301, audio_tagging_loss=0.008873, over 3045241.74 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:56:04,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2669586.6666666665, ans=0.1 2023-11-24 03:56:04,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.07 vs. limit=15.0 2023-11-24 03:56:06,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400450 2023-11-24 03:56:11,288 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.93 vs. limit=12.0 2023-11-24 03:56:20,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2669720.0, ans=0.0 2023-11-24 03:56:26,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2669720.0, ans=0.0 2023-11-24 03:56:34,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2669786.6666666665, ans=0.125 2023-11-24 03:56:45,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2669853.3333333335, ans=0.1 2023-11-24 03:56:54,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2669853.3333333335, ans=0.0 2023-11-24 03:56:56,227 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3700, loss[loss=0.07146, simple_loss=0.1097, pruned_loss=0.01175, audio_tagging_loss=0.004853, over 15630.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09067, pruned_loss=0.01316, audio_tagging_loss=0.00888, over 3050790.02 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:56:57,440 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.22 vs. limit=15.0 2023-11-24 03:56:59,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2669920.0, ans=0.125 2023-11-24 03:57:10,548 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400500 2023-11-24 03:57:12,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2669986.6666666665, ans=0.125 2023-11-24 03:57:20,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2669986.6666666665, ans=0.0 2023-11-24 03:57:29,585 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:57:34,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2670120.0, ans=0.1 2023-11-24 03:57:37,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2670120.0, ans=0.125 2023-11-24 03:57:39,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2670120.0, ans=0.0 2023-11-24 03:58:00,090 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.459e+01 9.233e+01 9.964e+01 1.289e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 03:58:00,135 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3750, loss[loss=0.06146, simple_loss=0.08356, pruned_loss=0.01077, audio_tagging_loss=0.008905, over 15555.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09099, pruned_loss=0.01321, audio_tagging_loss=0.008932, over 3058428.49 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:58:11,985 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400550 2023-11-24 03:58:23,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.64 vs. limit=15.0 2023-11-24 03:58:26,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2670386.6666666665, ans=0.125 2023-11-24 03:58:33,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2670386.6666666665, ans=0.025 2023-11-24 03:58:34,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2670386.6666666665, ans=0.125 2023-11-24 03:58:35,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2670453.3333333335, ans=0.0 2023-11-24 03:58:41,616 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 03:58:44,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2670453.3333333335, ans=0.2 2023-11-24 03:58:55,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2670520.0, ans=10.0 2023-11-24 03:59:01,178 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3800, loss[loss=0.05834, simple_loss=0.07759, pruned_loss=0.009758, audio_tagging_loss=0.009785, over 15462.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09155, pruned_loss=0.01328, audio_tagging_loss=0.008978, over 3058683.24 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 03:59:08,827 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:59:10,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.91 vs. limit=6.0 2023-11-24 03:59:12,309 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 03:59:13,384 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400600 2023-11-24 03:59:49,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2670786.6666666665, ans=0.2 2023-11-24 03:59:53,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2670853.3333333335, ans=0.125 2023-11-24 03:59:54,248 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.19 vs. limit=15.0 2023-11-24 03:59:59,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2670853.3333333335, ans=0.125 2023-11-24 04:00:03,044 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.597e+01 8.605e+01 9.126e+01 9.740e+01 1.269e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-24 04:00:03,089 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3850, loss[loss=0.0553, simple_loss=0.06986, pruned_loss=0.008732, audio_tagging_loss=0.01163, over 14686.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09126, pruned_loss=0.01319, audio_tagging_loss=0.009074, over 3059982.29 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:00:16,292 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400650 2023-11-24 04:00:51,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2671186.6666666665, ans=0.125 2023-11-24 04:01:06,130 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3900, loss[loss=0.06097, simple_loss=0.08094, pruned_loss=0.01088, audio_tagging_loss=0.009617, over 14625.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09249, pruned_loss=0.01351, audio_tagging_loss=0.009087, over 3054814.33 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:01:18,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400700 2023-11-24 04:01:31,387 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:01:38,715 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.33 vs. limit=15.0 2023-11-24 04:01:56,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2671520.0, ans=0.125 2023-11-24 04:02:08,916 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.467e+01 9.035e+01 9.770e+01 1.225e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 04:02:08,961 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 3950, loss[loss=0.08572, simple_loss=0.1197, pruned_loss=0.01603, audio_tagging_loss=0.009867, over 14349.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09185, pruned_loss=0.01335, audio_tagging_loss=0.00929, over 3048624.35 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:02:20,950 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400750 2023-11-24 04:02:24,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2671653.3333333335, ans=0.125 2023-11-24 04:02:30,434 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=15.0 2023-11-24 04:02:31,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2671653.3333333335, ans=0.125 2023-11-24 04:02:36,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2671720.0, ans=0.0 2023-11-24 04:03:10,953 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4000, loss[loss=0.06967, simple_loss=0.1015, pruned_loss=0.01304, audio_tagging_loss=0.005877, over 14172.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09175, pruned_loss=0.01339, audio_tagging_loss=0.009246, over 3050205.70 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:03:24,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400800 2023-11-24 04:03:39,925 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.34 vs. limit=15.0 2023-11-24 04:03:44,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2672053.3333333335, ans=0.125 2023-11-24 04:04:13,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2672253.3333333335, ans=0.125 2023-11-24 04:04:13,887 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.813e+01 8.478e+01 9.170e+01 1.003e+02 1.153e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-24 04:04:13,933 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4050, loss[loss=0.0545, simple_loss=0.07441, pruned_loss=0.007247, audio_tagging_loss=0.01004, over 14915.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09178, pruned_loss=0.01337, audio_tagging_loss=0.009294, over 3047972.20 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:04:16,820 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 04:04:17,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2672253.3333333335, ans=0.125 2023-11-24 04:04:26,463 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400850 2023-11-24 04:04:54,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2672453.3333333335, ans=0.125 2023-11-24 04:04:59,441 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0 2023-11-24 04:05:04,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2672520.0, ans=0.125 2023-11-24 04:05:05,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.65 vs. limit=15.0 2023-11-24 04:05:09,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.29 vs. limit=15.0 2023-11-24 04:05:15,959 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4100, loss[loss=0.06481, simple_loss=0.09355, pruned_loss=0.00984, audio_tagging_loss=0.008197, over 15895.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09202, pruned_loss=0.01333, audio_tagging_loss=0.009296, over 3047328.72 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:05:28,574 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400900 2023-11-24 04:05:29,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.25 vs. limit=22.5 2023-11-24 04:05:35,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2672653.3333333335, ans=0.125 2023-11-24 04:05:50,397 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:06:11,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2672853.3333333335, ans=0.125 2023-11-24 04:06:12,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.23 vs. limit=15.0 2023-11-24 04:06:18,034 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4150, loss[loss=0.06133, simple_loss=0.08179, pruned_loss=0.01247, audio_tagging_loss=0.007964, over 14719.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09216, pruned_loss=0.01348, audio_tagging_loss=0.009159, over 3046031.30 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:06:19,138 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.718e+01 9.320e+01 1.030e+02 1.270e+02, threshold=1.864e+02, percent-clipped=0.0 2023-11-24 04:06:31,268 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 400950 2023-11-24 04:06:43,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2673053.3333333335, ans=0.125 2023-11-24 04:06:50,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2673053.3333333335, ans=0.125 2023-11-24 04:06:52,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2673053.3333333335, ans=0.125 2023-11-24 04:07:00,693 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.43 vs. limit=22.5 2023-11-24 04:07:02,450 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 04:07:11,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2673186.6666666665, ans=0.2 2023-11-24 04:07:12,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2673186.6666666665, ans=0.0 2023-11-24 04:07:14,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2673186.6666666665, ans=0.2 2023-11-24 04:07:15,442 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.94 vs. limit=10.0 2023-11-24 04:07:21,669 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4200, loss[loss=0.07932, simple_loss=0.0992, pruned_loss=0.02138, audio_tagging_loss=0.008343, over 14061.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09214, pruned_loss=0.01358, audio_tagging_loss=0.00911, over 3044702.55 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:07:34,247 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401000 2023-11-24 04:07:43,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2673320.0, ans=0.035 2023-11-24 04:07:50,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2673386.6666666665, ans=0.125 2023-11-24 04:07:50,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.81 vs. limit=10.0 2023-11-24 04:07:55,512 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.32 vs. limit=15.0 2023-11-24 04:08:08,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2673453.3333333335, ans=0.125 2023-11-24 04:08:09,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2673453.3333333335, ans=0.125 2023-11-24 04:08:09,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2673453.3333333335, ans=0.125 2023-11-24 04:08:16,323 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2673520.0, ans=0.125 2023-11-24 04:08:24,423 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4250, loss[loss=0.06182, simple_loss=0.08798, pruned_loss=0.01009, audio_tagging_loss=0.007739, over 15706.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09293, pruned_loss=0.01368, audio_tagging_loss=0.008906, over 3054882.30 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:08:24,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2673586.6666666665, ans=0.1 2023-11-24 04:08:25,551 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.323e+01 9.029e+01 9.665e+01 1.222e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 04:08:28,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2673586.6666666665, ans=0.125 2023-11-24 04:08:36,850 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401050 2023-11-24 04:08:39,640 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.18 vs. limit=22.5 2023-11-24 04:08:53,367 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.30 vs. limit=15.0 2023-11-24 04:08:58,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2673720.0, ans=0.125 2023-11-24 04:09:03,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2673786.6666666665, ans=0.2 2023-11-24 04:09:26,470 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4300, loss[loss=0.05379, simple_loss=0.07294, pruned_loss=0.007611, audio_tagging_loss=0.009712, over 14934.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09328, pruned_loss=0.01372, audio_tagging_loss=0.008885, over 3055970.79 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:09:27,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2673920.0, ans=0.125 2023-11-24 04:09:29,511 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.16 vs. limit=15.0 2023-11-24 04:09:33,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2673920.0, ans=0.125 2023-11-24 04:09:38,411 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401100 2023-11-24 04:09:53,095 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:09:56,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2674053.3333333335, ans=0.125 2023-11-24 04:10:09,283 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.89 vs. limit=15.0 2023-11-24 04:10:11,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2674120.0, ans=0.2 2023-11-24 04:10:20,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2674186.6666666665, ans=0.0 2023-11-24 04:10:21,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2674186.6666666665, ans=0.0 2023-11-24 04:10:28,556 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4350, loss[loss=0.07359, simple_loss=0.09651, pruned_loss=0.01602, audio_tagging_loss=0.009312, over 14878.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09372, pruned_loss=0.01388, audio_tagging_loss=0.008771, over 3049984.85 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:10:30,156 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.375e+01 8.839e+01 9.580e+01 1.285e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-24 04:10:32,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2674253.3333333335, ans=0.0 2023-11-24 04:10:38,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2674253.3333333335, ans=0.1 2023-11-24 04:10:41,794 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401150 2023-11-24 04:10:46,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2674320.0, ans=0.125 2023-11-24 04:10:56,671 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.04 vs. limit=15.0 2023-11-24 04:11:03,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2674386.6666666665, ans=0.0 2023-11-24 04:11:05,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2674453.3333333335, ans=0.0 2023-11-24 04:11:31,219 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4400, loss[loss=0.05503, simple_loss=0.07074, pruned_loss=0.01021, audio_tagging_loss=0.00945, over 15443.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09378, pruned_loss=0.01391, audio_tagging_loss=0.008729, over 3047814.52 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:11:37,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2674586.6666666665, ans=0.125 2023-11-24 04:11:43,186 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401200 2023-11-24 04:12:07,093 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.66 vs. limit=15.0 2023-11-24 04:12:33,405 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4450, loss[loss=0.05862, simple_loss=0.07555, pruned_loss=0.01019, audio_tagging_loss=0.01064, over 15259.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09224, pruned_loss=0.01359, audio_tagging_loss=0.008891, over 3053834.38 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:12:34,505 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.511e+01 8.725e+01 9.329e+01 1.003e+02 1.264e+02, threshold=1.866e+02, percent-clipped=0.0 2023-11-24 04:12:43,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2674920.0, ans=0.1 2023-11-24 04:12:45,931 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401250 2023-11-24 04:12:51,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2674986.6666666665, ans=0.125 2023-11-24 04:12:54,375 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.03 vs. limit=15.0 2023-11-24 04:13:05,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2675053.3333333335, ans=0.1 2023-11-24 04:13:09,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2675053.3333333335, ans=0.125 2023-11-24 04:13:35,767 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4500, loss[loss=0.07498, simple_loss=0.1071, pruned_loss=0.01281, audio_tagging_loss=0.008593, over 14806.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09213, pruned_loss=0.01362, audio_tagging_loss=0.008918, over 3053211.67 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:13:47,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2675253.3333333335, ans=0.04949747468305833 2023-11-24 04:13:49,254 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401300 2023-11-24 04:14:34,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2675520.0, ans=0.125 2023-11-24 04:14:38,771 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4550, loss[loss=0.07344, simple_loss=0.1014, pruned_loss=0.01366, audio_tagging_loss=0.009109, over 15163.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09225, pruned_loss=0.01373, audio_tagging_loss=0.008926, over 3043076.85 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:14:39,936 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.652e+01 8.786e+01 9.235e+01 9.923e+01 1.429e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 04:14:50,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401350 2023-11-24 04:14:59,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2675653.3333333335, ans=0.125 2023-11-24 04:15:17,600 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2675786.6666666665, ans=0.125 2023-11-24 04:15:17,736 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.86 vs. limit=15.0 2023-11-24 04:15:24,800 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 04:15:39,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2675920.0, ans=0.95 2023-11-24 04:15:40,267 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4600, loss[loss=0.04835, simple_loss=0.05858, pruned_loss=0.0112, audio_tagging_loss=0.007857, over 15183.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09185, pruned_loss=0.01364, audio_tagging_loss=0.009041, over 3042699.63 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:15:47,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2675920.0, ans=0.1 2023-11-24 04:15:52,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401400 2023-11-24 04:16:16,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2676053.3333333335, ans=0.025 2023-11-24 04:16:25,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2676120.0, ans=0.125 2023-11-24 04:16:28,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2676120.0, ans=0.0 2023-11-24 04:16:29,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2676186.6666666665, ans=0.125 2023-11-24 04:16:33,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=18.16 vs. limit=22.5 2023-11-24 04:16:37,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2676186.6666666665, ans=0.125 2023-11-24 04:16:38,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2676186.6666666665, ans=0.2 2023-11-24 04:16:42,195 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4650, loss[loss=0.08743, simple_loss=0.1223, pruned_loss=0.01897, audio_tagging_loss=0.007312, over 15898.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09106, pruned_loss=0.01352, audio_tagging_loss=0.009201, over 3044336.64 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:16:43,856 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.311e+01 8.909e+01 9.483e+01 1.432e+02, threshold=1.782e+02, percent-clipped=0.0 2023-11-24 04:16:44,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2676253.3333333335, ans=0.2 2023-11-24 04:16:44,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2676253.3333333335, ans=0.1 2023-11-24 04:16:45,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2676253.3333333335, ans=0.125 2023-11-24 04:16:55,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401450 2023-11-24 04:17:08,872 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.31 vs. limit=15.0 2023-11-24 04:17:45,189 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4700, loss[loss=0.05324, simple_loss=0.07386, pruned_loss=0.008351, audio_tagging_loss=0.007958, over 15084.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09103, pruned_loss=0.01353, audio_tagging_loss=0.009189, over 3043540.38 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:17:49,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2676586.6666666665, ans=0.125 2023-11-24 04:17:57,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401500 2023-11-24 04:18:19,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2676720.0, ans=0.0 2023-11-24 04:18:19,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2676720.0, ans=0.125 2023-11-24 04:18:28,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.85 vs. limit=22.5 2023-11-24 04:18:35,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2676853.3333333335, ans=0.125 2023-11-24 04:18:46,872 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4750, loss[loss=0.06923, simple_loss=0.0906, pruned_loss=0.01348, audio_tagging_loss=0.01045, over 14141.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09109, pruned_loss=0.01335, audio_tagging_loss=0.009304, over 3041943.94 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:18:47,987 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.351e+01 9.004e+01 9.816e+01 1.181e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-24 04:18:58,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401550 2023-11-24 04:19:15,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2677053.3333333335, ans=0.125 2023-11-24 04:19:21,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2677053.3333333335, ans=0.125 2023-11-24 04:19:24,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2677120.0, ans=0.1 2023-11-24 04:19:47,775 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4800, loss[loss=0.05351, simple_loss=0.06981, pruned_loss=0.00871, audio_tagging_loss=0.009888, over 14565.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09012, pruned_loss=0.01312, audio_tagging_loss=0.009476, over 3042331.62 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:20:01,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401600 2023-11-24 04:20:29,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2677453.3333333335, ans=10.0 2023-11-24 04:20:32,141 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:20:52,267 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4850, loss[loss=0.06838, simple_loss=0.09502, pruned_loss=0.01078, audio_tagging_loss=0.01009, over 15021.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09021, pruned_loss=0.01308, audio_tagging_loss=0.009441, over 3041234.90 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:20:53,334 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.256e+01 8.375e+01 9.054e+01 9.821e+01 1.202e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-24 04:20:57,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2677586.6666666665, ans=0.1 2023-11-24 04:20:57,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2677586.6666666665, ans=0.2 2023-11-24 04:20:58,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2677586.6666666665, ans=0.125 2023-11-24 04:20:59,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2677586.6666666665, ans=0.125 2023-11-24 04:21:01,215 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.68 vs. limit=22.5 2023-11-24 04:21:04,069 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401650 2023-11-24 04:21:13,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2677653.3333333335, ans=0.1 2023-11-24 04:21:43,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2677853.3333333335, ans=0.125 2023-11-24 04:21:53,714 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4900, loss[loss=0.07162, simple_loss=0.1029, pruned_loss=0.01395, audio_tagging_loss=0.00623, over 14956.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.091, pruned_loss=0.01307, audio_tagging_loss=0.009415, over 3035190.85 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:22:05,638 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401700 2023-11-24 04:22:25,558 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.34 vs. limit=22.5 2023-11-24 04:22:28,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2678053.3333333335, ans=0.125 2023-11-24 04:22:32,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2678120.0, ans=0.125 2023-11-24 04:22:34,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2678120.0, ans=0.0 2023-11-24 04:22:40,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2678120.0, ans=0.125 2023-11-24 04:22:49,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2678186.6666666665, ans=0.125 2023-11-24 04:22:55,425 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 4950, loss[loss=0.08026, simple_loss=0.1073, pruned_loss=0.01876, audio_tagging_loss=0.007846, over 15183.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09058, pruned_loss=0.01303, audio_tagging_loss=0.009298, over 3036457.18 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:22:56,554 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.172e+01 8.560e+01 8.976e+01 9.753e+01 1.389e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-24 04:23:03,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2678253.3333333335, ans=0.125 2023-11-24 04:23:08,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401750 2023-11-24 04:23:14,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2678320.0, ans=0.125 2023-11-24 04:23:46,181 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:23:48,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2678520.0, ans=0.125 2023-11-24 04:23:57,870 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5000, loss[loss=0.06192, simple_loss=0.0808, pruned_loss=0.01347, audio_tagging_loss=0.008049, over 14700.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09055, pruned_loss=0.01296, audio_tagging_loss=0.009079, over 3033175.28 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:23:59,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2678586.6666666665, ans=0.1 2023-11-24 04:24:05,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2678586.6666666665, ans=0.125 2023-11-24 04:24:07,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2678586.6666666665, ans=0.0 2023-11-24 04:24:10,924 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401800 2023-11-24 04:24:27,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.24 vs. limit=15.0 2023-11-24 04:24:38,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2678786.6666666665, ans=0.2 2023-11-24 04:25:00,944 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5050, loss[loss=0.09432, simple_loss=0.1294, pruned_loss=0.02431, audio_tagging_loss=0.005309, over 15458.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09128, pruned_loss=0.0131, audio_tagging_loss=0.008971, over 3037264.44 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:25:02,054 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.608e+01 9.238e+01 9.938e+01 4.414e+02, threshold=1.848e+02, percent-clipped=1.0 2023-11-24 04:25:08,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2678920.0, ans=0.125 2023-11-24 04:25:09,940 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.63 vs. limit=10.0 2023-11-24 04:25:12,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401850 2023-11-24 04:26:02,691 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5100, loss[loss=0.05183, simple_loss=0.06663, pruned_loss=0.007604, audio_tagging_loss=0.01091, over 15367.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09067, pruned_loss=0.01307, audio_tagging_loss=0.008971, over 3038237.38 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:26:04,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2679253.3333333335, ans=0.2 2023-11-24 04:26:16,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401900 2023-11-24 04:26:26,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2679320.0, ans=0.0 2023-11-24 04:26:43,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.96 vs. limit=10.0 2023-11-24 04:27:05,524 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5150, loss[loss=0.08804, simple_loss=0.1125, pruned_loss=0.02306, audio_tagging_loss=0.008732, over 15187.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09048, pruned_loss=0.01296, audio_tagging_loss=0.008947, over 3039708.37 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:27:07,944 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.413e+01 8.884e+01 9.698e+01 1.235e+02, threshold=1.777e+02, percent-clipped=0.0 2023-11-24 04:27:18,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 401950 2023-11-24 04:27:18,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2679653.3333333335, ans=0.125 2023-11-24 04:27:30,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2679720.0, ans=0.125 2023-11-24 04:27:49,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2679786.6666666665, ans=0.0 2023-11-24 04:28:03,079 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.20 vs. limit=15.0 2023-11-24 04:28:07,491 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0 2023-11-24 04:28:08,612 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5200, loss[loss=0.07426, simple_loss=0.107, pruned_loss=0.01375, audio_tagging_loss=0.007008, over 15577.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09079, pruned_loss=0.01295, audio_tagging_loss=0.008924, over 3037619.37 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:28:10,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2679920.0, ans=0.07 2023-11-24 04:28:20,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402000 2023-11-24 04:28:30,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2679986.6666666665, ans=0.5 2023-11-24 04:29:03,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2680186.6666666665, ans=0.2 2023-11-24 04:29:07,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2680186.6666666665, ans=0.125 2023-11-24 04:29:09,998 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5250, loss[loss=0.08373, simple_loss=0.107, pruned_loss=0.02226, audio_tagging_loss=0.007982, over 14150.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09134, pruned_loss=0.01319, audio_tagging_loss=0.008889, over 3034856.87 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0 2023-11-24 04:29:11,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2680253.3333333335, ans=0.1 2023-11-24 04:29:12,361 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.228e+01 8.916e+01 9.618e+01 1.278e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-24 04:29:21,914 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402050 2023-11-24 04:29:36,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2680386.6666666665, ans=0.125 2023-11-24 04:30:01,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2680520.0, ans=0.1 2023-11-24 04:30:12,378 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5300, loss[loss=0.08184, simple_loss=0.1061, pruned_loss=0.02192, audio_tagging_loss=0.00689, over 14875.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09266, pruned_loss=0.01348, audio_tagging_loss=0.008812, over 3034143.83 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:30:25,286 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402100 2023-11-24 04:30:35,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.96 vs. limit=10.0 2023-11-24 04:30:38,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2680720.0, ans=0.0 2023-11-24 04:30:54,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2680786.6666666665, ans=0.09899494936611666 2023-11-24 04:30:56,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.23 vs. limit=6.0 2023-11-24 04:31:11,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2680853.3333333335, ans=0.125 2023-11-24 04:31:15,250 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5350, loss[loss=0.05301, simple_loss=0.07059, pruned_loss=0.008904, audio_tagging_loss=0.008811, over 15901.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09255, pruned_loss=0.01339, audio_tagging_loss=0.008872, over 3035796.67 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:31:15,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2680920.0, ans=0.0 2023-11-24 04:31:18,680 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.481e+01 8.724e+01 9.285e+01 9.851e+01 1.330e+02, threshold=1.857e+02, percent-clipped=0.0 2023-11-24 04:31:27,826 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402150 2023-11-24 04:32:16,955 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5400, loss[loss=0.05334, simple_loss=0.06682, pruned_loss=0.007786, audio_tagging_loss=0.01215, over 16221.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09243, pruned_loss=0.01358, audio_tagging_loss=0.008901, over 3032760.87 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:32:28,786 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402200 2023-11-24 04:32:29,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.04 vs. limit=15.0 2023-11-24 04:32:32,681 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.13 vs. limit=12.0 2023-11-24 04:32:34,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.56 vs. limit=15.0 2023-11-24 04:32:49,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2681386.6666666665, ans=0.125 2023-11-24 04:32:49,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2681386.6666666665, ans=0.125 2023-11-24 04:32:58,465 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.36 vs. limit=15.0 2023-11-24 04:32:59,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2681453.3333333335, ans=0.0 2023-11-24 04:33:18,583 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5450, loss[loss=0.05088, simple_loss=0.06643, pruned_loss=0.00795, audio_tagging_loss=0.009717, over 15350.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09212, pruned_loss=0.01348, audio_tagging_loss=0.008973, over 3035418.51 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 8.0 2023-11-24 04:33:21,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2681586.6666666665, ans=0.04949747468305833 2023-11-24 04:33:24,396 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.363e+01 9.005e+01 9.739e+01 1.241e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-24 04:33:29,873 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.41 vs. limit=22.5 2023-11-24 04:33:31,646 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402250 2023-11-24 04:33:33,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2681653.3333333335, ans=0.0 2023-11-24 04:33:54,597 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.13 vs. limit=15.0 2023-11-24 04:34:21,301 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5500, loss[loss=0.06726, simple_loss=0.09519, pruned_loss=0.009312, audio_tagging_loss=0.01035, over 15085.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09244, pruned_loss=0.01346, audio_tagging_loss=0.008962, over 3037681.50 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 8.0 2023-11-24 04:34:23,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.45 vs. limit=22.5 2023-11-24 04:34:33,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402300 2023-11-24 04:34:33,304 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:34:49,071 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:35:11,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2682186.6666666665, ans=0.125 2023-11-24 04:35:22,504 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5550, loss[loss=0.08837, simple_loss=0.1198, pruned_loss=0.01935, audio_tagging_loss=0.009121, over 16427.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09279, pruned_loss=0.01353, audio_tagging_loss=0.008995, over 3039908.75 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 8.0 2023-11-24 04:35:27,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.398e+01 8.518e+01 9.178e+01 1.002e+02 1.571e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-24 04:35:34,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402350 2023-11-24 04:35:41,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2682320.0, ans=0.125 2023-11-24 04:35:53,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2682386.6666666665, ans=0.125 2023-11-24 04:36:09,878 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.97 vs. limit=22.5 2023-11-24 04:36:15,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2682520.0, ans=0.2 2023-11-24 04:36:24,099 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5600, loss[loss=0.06144, simple_loss=0.07669, pruned_loss=0.01093, audio_tagging_loss=0.01216, over 14615.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09213, pruned_loss=0.01337, audio_tagging_loss=0.009077, over 3037785.75 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0 2023-11-24 04:36:37,242 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402400 2023-11-24 04:37:07,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2682786.6666666665, ans=0.05 2023-11-24 04:37:08,392 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 04:37:08,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2682786.6666666665, ans=0.0 2023-11-24 04:37:27,405 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5650, loss[loss=0.0568, simple_loss=0.06851, pruned_loss=0.01106, audio_tagging_loss=0.01149, over 15175.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09274, pruned_loss=0.01352, audio_tagging_loss=0.009177, over 3045562.81 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:37:33,009 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.272e+01 8.812e+01 9.562e+01 1.491e+02, threshold=1.762e+02, percent-clipped=0.0 2023-11-24 04:37:36,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2682920.0, ans=0.1 2023-11-24 04:37:40,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402450 2023-11-24 04:37:44,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2682986.6666666665, ans=0.09899494936611666 2023-11-24 04:37:53,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2683053.3333333335, ans=0.125 2023-11-24 04:37:54,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2683053.3333333335, ans=0.125 2023-11-24 04:37:55,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2683053.3333333335, ans=0.07 2023-11-24 04:37:58,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2683053.3333333335, ans=0.05 2023-11-24 04:38:31,391 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5700, loss[loss=0.05958, simple_loss=0.08047, pruned_loss=0.01198, audio_tagging_loss=0.007362, over 14990.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09262, pruned_loss=0.01362, audio_tagging_loss=0.009093, over 3045993.79 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:38:36,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2683253.3333333335, ans=0.1 2023-11-24 04:38:37,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2683253.3333333335, ans=0.0 2023-11-24 04:38:43,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402500 2023-11-24 04:38:50,564 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.49 vs. limit=15.0 2023-11-24 04:38:58,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2683386.6666666665, ans=0.0 2023-11-24 04:39:02,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2683386.6666666665, ans=0.125 2023-11-24 04:39:06,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2683386.6666666665, ans=0.0 2023-11-24 04:39:33,795 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5750, loss[loss=0.07651, simple_loss=0.1003, pruned_loss=0.0171, audio_tagging_loss=0.009262, over 15304.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09294, pruned_loss=0.01376, audio_tagging_loss=0.008936, over 3047588.00 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:39:39,183 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.554e+01 9.219e+01 1.000e+02 1.500e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 04:39:46,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2683653.3333333335, ans=0.125 2023-11-24 04:39:47,596 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402550 2023-11-24 04:39:48,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2683653.3333333335, ans=0.0 2023-11-24 04:39:56,101 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.48 vs. limit=15.0 2023-11-24 04:39:56,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2683653.3333333335, ans=0.1 2023-11-24 04:40:01,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2683720.0, ans=0.2 2023-11-24 04:40:15,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2683786.6666666665, ans=0.5 2023-11-24 04:40:22,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2683853.3333333335, ans=0.125 2023-11-24 04:40:23,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.70 vs. limit=15.0 2023-11-24 04:40:30,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2683853.3333333335, ans=0.0 2023-11-24 04:40:35,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2683853.3333333335, ans=0.05 2023-11-24 04:40:37,110 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5800, loss[loss=0.06817, simple_loss=0.08797, pruned_loss=0.01676, audio_tagging_loss=0.007427, over 15240.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09287, pruned_loss=0.01374, audio_tagging_loss=0.00886, over 3052511.38 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:40:38,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2683920.0, ans=0.2 2023-11-24 04:40:42,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2683920.0, ans=0.1 2023-11-24 04:40:42,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2683920.0, ans=0.125 2023-11-24 04:40:49,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402600 2023-11-24 04:41:06,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2684053.3333333335, ans=0.125 2023-11-24 04:41:10,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_na.min_abs, batch_count=2684053.3333333335, ans=0.02 2023-11-24 04:41:13,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2684120.0, ans=0.0 2023-11-24 04:41:21,616 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.02 vs. limit=22.5 2023-11-24 04:41:24,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2684120.0, ans=0.0 2023-11-24 04:41:39,347 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5850, loss[loss=0.06374, simple_loss=0.08386, pruned_loss=0.0126, audio_tagging_loss=0.0092, over 15184.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.0926, pruned_loss=0.0136, audio_tagging_loss=0.008824, over 3045879.02 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:41:39,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2684253.3333333335, ans=0.125 2023-11-24 04:41:44,140 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 8.321e+01 9.009e+01 9.790e+01 1.176e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-24 04:41:51,404 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402650 2023-11-24 04:41:51,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2684320.0, ans=0.1 2023-11-24 04:41:53,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2684320.0, ans=0.2 2023-11-24 04:42:17,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2684453.3333333335, ans=0.125 2023-11-24 04:42:41,230 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5900, loss[loss=0.07373, simple_loss=0.09741, pruned_loss=0.01462, audio_tagging_loss=0.01041, over 16462.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09319, pruned_loss=0.01372, audio_tagging_loss=0.008772, over 3053998.83 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:42:54,546 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402700 2023-11-24 04:43:23,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2684786.6666666665, ans=0.125 2023-11-24 04:43:24,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2684786.6666666665, ans=0.1 2023-11-24 04:43:27,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2684786.6666666665, ans=0.09899494936611666 2023-11-24 04:43:32,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2684853.3333333335, ans=0.0 2023-11-24 04:43:32,176 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:43:37,988 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=7.15 vs. limit=10.0 2023-11-24 04:43:44,347 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 5950, loss[loss=0.04654, simple_loss=0.05103, pruned_loss=0.009928, audio_tagging_loss=0.01109, over 13990.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09248, pruned_loss=0.01355, audio_tagging_loss=0.008803, over 3060579.04 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:43:49,677 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.371e+01 8.328e+01 9.136e+01 9.988e+01 1.334e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 04:43:52,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=2684920.0, ans=0.02 2023-11-24 04:43:57,012 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402750 2023-11-24 04:43:57,554 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=6.85 vs. limit=12.0 2023-11-24 04:44:03,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2684986.6666666665, ans=0.125 2023-11-24 04:44:35,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.78 vs. limit=22.5 2023-11-24 04:44:36,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2685186.6666666665, ans=0.125 2023-11-24 04:44:45,816 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6000, loss[loss=0.0708, simple_loss=0.09799, pruned_loss=0.01451, audio_tagging_loss=0.007303, over 17265.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.0931, pruned_loss=0.01369, audio_tagging_loss=0.008829, over 3053107.09 frames. ], batch size: 62, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:44:45,817 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 04:45:26,681 INFO [train_asr.py:1253] (3/4) Epoch 34, validation: loss=0.05772, simple_loss=0.05082, pruned_loss=0.005023, audio_tagging_loss=0.02728, over 4681554.00 frames. 2023-11-24 04:45:26,683 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 04:45:33,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2685253.3333333335, ans=0.0 2023-11-24 04:45:40,312 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402800 2023-11-24 04:46:09,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2685453.3333333335, ans=0.0 2023-11-24 04:46:11,982 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 04:46:25,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2685520.0, ans=0.0 2023-11-24 04:46:29,981 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6050, loss[loss=0.07702, simple_loss=0.1113, pruned_loss=0.01498, audio_tagging_loss=0.006388, over 15001.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09309, pruned_loss=0.01368, audio_tagging_loss=0.008802, over 3057001.84 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:46:30,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2685586.6666666665, ans=0.2 2023-11-24 04:46:35,237 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.324e+01 9.101e+01 9.713e+01 1.173e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-24 04:46:42,475 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402850 2023-11-24 04:46:43,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2685653.3333333335, ans=0.0 2023-11-24 04:46:50,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2685653.3333333335, ans=0.125 2023-11-24 04:46:55,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2685720.0, ans=0.025 2023-11-24 04:46:59,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2685720.0, ans=0.1 2023-11-24 04:47:13,579 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.50 vs. limit=22.5 2023-11-24 04:47:27,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2685853.3333333335, ans=0.1 2023-11-24 04:47:31,920 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6100, loss[loss=0.06197, simple_loss=0.08097, pruned_loss=0.01207, audio_tagging_loss=0.009414, over 14834.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09181, pruned_loss=0.01344, audio_tagging_loss=0.008809, over 3052078.84 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:47:43,833 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402900 2023-11-24 04:48:08,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.88 vs. limit=15.0 2023-11-24 04:48:16,171 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=31.12 vs. limit=22.5 2023-11-24 04:48:19,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2686120.0, ans=0.125 2023-11-24 04:48:33,315 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6150, loss[loss=0.0593, simple_loss=0.07783, pruned_loss=0.01137, audio_tagging_loss=0.009007, over 16958.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09195, pruned_loss=0.0135, audio_tagging_loss=0.008801, over 3054090.39 frames. ], batch size: 67, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:48:33,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2686253.3333333335, ans=0.125 2023-11-24 04:48:38,029 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.476e+01 9.051e+01 9.662e+01 1.133e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 04:48:40,801 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.48 vs. limit=15.0 2023-11-24 04:48:43,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten.whitening_limit, batch_count=2686253.3333333335, ans=22.5 2023-11-24 04:48:46,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 402950 2023-11-24 04:48:52,421 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.18 vs. limit=15.0 2023-11-24 04:49:05,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2686386.6666666665, ans=0.125 2023-11-24 04:49:32,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2686520.0, ans=0.125 2023-11-24 04:49:36,253 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6200, loss[loss=0.0658, simple_loss=0.09181, pruned_loss=0.01338, audio_tagging_loss=0.006522, over 15324.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09176, pruned_loss=0.01345, audio_tagging_loss=0.008896, over 3052219.69 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:49:49,226 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403000 2023-11-24 04:49:52,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2686653.3333333335, ans=0.1 2023-11-24 04:49:54,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2686653.3333333335, ans=0.0 2023-11-24 04:50:00,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2686720.0, ans=0.0 2023-11-24 04:50:10,293 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:50:16,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2686786.6666666665, ans=0.125 2023-11-24 04:50:25,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2686853.3333333335, ans=0.0 2023-11-24 04:50:27,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2686853.3333333335, ans=0.125 2023-11-24 04:50:39,706 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6250, loss[loss=0.06995, simple_loss=0.08449, pruned_loss=0.01376, audio_tagging_loss=0.01395, over 15056.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09147, pruned_loss=0.01345, audio_tagging_loss=0.00913, over 3058879.48 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:50:45,705 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.512e+01 9.019e+01 9.887e+01 1.410e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-24 04:50:46,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2686920.0, ans=0.2 2023-11-24 04:50:47,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2686920.0, ans=0.125 2023-11-24 04:50:51,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403050 2023-11-24 04:51:00,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2686986.6666666665, ans=0.0 2023-11-24 04:51:09,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2687053.3333333335, ans=0.0 2023-11-24 04:51:12,278 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:51:18,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2687120.0, ans=0.0 2023-11-24 04:51:22,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2687120.0, ans=0.2 2023-11-24 04:51:41,178 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6300, loss[loss=0.05567, simple_loss=0.07682, pruned_loss=0.008062, audio_tagging_loss=0.009196, over 16215.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09136, pruned_loss=0.01338, audio_tagging_loss=0.009136, over 3053317.82 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:51:53,709 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403100 2023-11-24 04:51:56,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2687320.0, ans=0.125 2023-11-24 04:52:14,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2687386.6666666665, ans=0.125 2023-11-24 04:52:16,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2687386.6666666665, ans=0.0 2023-11-24 04:52:17,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2687453.3333333335, ans=0.1 2023-11-24 04:52:43,752 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6350, loss[loss=0.06046, simple_loss=0.07934, pruned_loss=0.009634, audio_tagging_loss=0.01116, over 14260.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09061, pruned_loss=0.01318, audio_tagging_loss=0.009217, over 3043677.17 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:52:50,173 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.641e+01 8.312e+01 8.895e+01 9.882e+01 1.344e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-24 04:52:56,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403150 2023-11-24 04:53:03,039 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.70 vs. limit=15.0 2023-11-24 04:53:07,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2687720.0, ans=0.125 2023-11-24 04:53:33,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2687853.3333333335, ans=0.125 2023-11-24 04:53:46,054 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6400, loss[loss=0.06972, simple_loss=0.1006, pruned_loss=0.01097, audio_tagging_loss=0.008426, over 15638.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09069, pruned_loss=0.01321, audio_tagging_loss=0.009308, over 3044549.28 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:53:57,945 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403200 2023-11-24 04:54:11,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2688053.3333333335, ans=0.125 2023-11-24 04:54:28,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2688120.0, ans=0.2 2023-11-24 04:54:30,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2688120.0, ans=0.125 2023-11-24 04:54:33,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2688120.0, ans=0.125 2023-11-24 04:54:40,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2688186.6666666665, ans=0.95 2023-11-24 04:54:41,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2688186.6666666665, ans=0.0 2023-11-24 04:54:44,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2688186.6666666665, ans=0.2 2023-11-24 04:54:45,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2688186.6666666665, ans=0.0 2023-11-24 04:54:47,663 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6450, loss[loss=0.06735, simple_loss=0.09016, pruned_loss=0.01203, audio_tagging_loss=0.01024, over 15479.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.08958, pruned_loss=0.0131, audio_tagging_loss=0.009468, over 3037289.95 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:54:53,478 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.209e+01 8.936e+01 9.468e+01 2.072e+02, threshold=1.787e+02, percent-clipped=1.0 2023-11-24 04:54:56,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2688253.3333333335, ans=0.0 2023-11-24 04:55:00,084 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403250 2023-11-24 04:55:05,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2688320.0, ans=0.0 2023-11-24 04:55:14,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=22.5 2023-11-24 04:55:49,153 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.59 vs. limit=15.0 2023-11-24 04:55:49,658 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6500, loss[loss=0.071, simple_loss=0.1065, pruned_loss=0.01005, audio_tagging_loss=0.007705, over 15448.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09072, pruned_loss=0.01336, audio_tagging_loss=0.009325, over 3044623.74 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 04:55:56,702 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0 2023-11-24 04:56:02,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403300 2023-11-24 04:56:04,370 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.76 vs. limit=22.5 2023-11-24 04:56:11,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2688653.3333333335, ans=0.1 2023-11-24 04:56:21,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2688720.0, ans=0.1 2023-11-24 04:56:52,705 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6550, loss[loss=0.07377, simple_loss=0.09148, pruned_loss=0.018, audio_tagging_loss=0.01003, over 13705.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09065, pruned_loss=0.01344, audio_tagging_loss=0.009188, over 3046454.67 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:56:56,918 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.51 vs. limit=15.0 2023-11-24 04:56:57,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2688920.0, ans=0.2 2023-11-24 04:56:59,702 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.455e+01 8.633e+01 9.315e+01 9.907e+01 1.273e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 04:57:05,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403350 2023-11-24 04:57:13,941 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.84 vs. limit=15.0 2023-11-24 04:57:31,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2689120.0, ans=0.1 2023-11-24 04:57:40,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2689120.0, ans=0.2 2023-11-24 04:57:54,963 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6600, loss[loss=0.07207, simple_loss=0.105, pruned_loss=0.01151, audio_tagging_loss=0.00807, over 15780.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09098, pruned_loss=0.01345, audio_tagging_loss=0.009089, over 3047894.54 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:58:06,789 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403400 2023-11-24 04:58:11,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2689320.0, ans=0.0 2023-11-24 04:58:53,910 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0 2023-11-24 04:58:57,518 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6650, loss[loss=0.04284, simple_loss=0.04946, pruned_loss=0.00704, audio_tagging_loss=0.01107, over 15708.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09085, pruned_loss=0.01336, audio_tagging_loss=0.008964, over 3050486.82 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 04:58:57,791 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 04:59:05,515 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.526e+01 8.351e+01 9.016e+01 9.664e+01 1.302e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-24 04:59:11,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403450 2023-11-24 04:59:37,021 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.26 vs. limit=22.5 2023-11-24 04:59:40,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2689786.6666666665, ans=0.0 2023-11-24 04:59:46,832 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2689853.3333333335, ans=0.0 2023-11-24 05:00:00,601 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6700, loss[loss=0.06842, simple_loss=0.08931, pruned_loss=0.01414, audio_tagging_loss=0.009626, over 14810.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09018, pruned_loss=0.01309, audio_tagging_loss=0.009006, over 3048258.68 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:00:10,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2689920.0, ans=0.1 2023-11-24 05:00:12,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403500 2023-11-24 05:00:22,131 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.59 vs. limit=15.0 2023-11-24 05:00:34,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=2690053.3333333335, ans=15.0 2023-11-24 05:00:48,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2690120.0, ans=0.2 2023-11-24 05:01:01,985 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6750, loss[loss=0.06097, simple_loss=0.08097, pruned_loss=0.01134, audio_tagging_loss=0.009151, over 14392.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09071, pruned_loss=0.01316, audio_tagging_loss=0.008946, over 3043704.05 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:01:09,700 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.241e+01 8.716e+01 9.523e+01 2.058e+02, threshold=1.743e+02, percent-clipped=1.0 2023-11-24 05:01:10,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2690253.3333333335, ans=0.125 2023-11-24 05:01:14,523 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403550 2023-11-24 05:01:28,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2690386.6666666665, ans=0.125 2023-11-24 05:01:45,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2690453.3333333335, ans=0.0 2023-11-24 05:02:00,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2690520.0, ans=0.2 2023-11-24 05:02:04,022 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6800, loss[loss=0.08929, simple_loss=0.1268, pruned_loss=0.01937, audio_tagging_loss=0.006506, over 14237.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09128, pruned_loss=0.01328, audio_tagging_loss=0.008911, over 3038330.74 frames. ], batch size: 52, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:02:17,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403600 2023-11-24 05:03:07,295 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6850, loss[loss=0.05039, simple_loss=0.06702, pruned_loss=0.007196, audio_tagging_loss=0.00968, over 15587.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09104, pruned_loss=0.0133, audio_tagging_loss=0.008915, over 3041077.95 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:03:07,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2690920.0, ans=0.125 2023-11-24 05:03:15,560 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.436e+01 8.418e+01 8.943e+01 9.846e+01 1.358e+02, threshold=1.789e+02, percent-clipped=0.0 2023-11-24 05:03:19,219 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403650 2023-11-24 05:03:35,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2691053.3333333335, ans=0.125 2023-11-24 05:03:54,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.57 vs. limit=15.0 2023-11-24 05:03:55,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2691186.6666666665, ans=0.125 2023-11-24 05:04:01,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2691186.6666666665, ans=0.125 2023-11-24 05:04:08,463 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6900, loss[loss=0.07607, simple_loss=0.1017, pruned_loss=0.01389, audio_tagging_loss=0.01133, over 17027.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09209, pruned_loss=0.01362, audio_tagging_loss=0.008828, over 3050188.58 frames. ], batch size: 63, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:04:11,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2691253.3333333335, ans=0.04949747468305833 2023-11-24 05:04:15,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2691253.3333333335, ans=0.125 2023-11-24 05:04:20,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403700 2023-11-24 05:04:34,571 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.11 vs. limit=22.5 2023-11-24 05:04:54,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2691453.3333333335, ans=0.125 2023-11-24 05:04:55,834 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 05:05:09,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2691586.6666666665, ans=0.125 2023-11-24 05:05:10,071 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 6950, loss[loss=0.07965, simple_loss=0.1054, pruned_loss=0.0172, audio_tagging_loss=0.009761, over 16082.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09241, pruned_loss=0.01356, audio_tagging_loss=0.0089, over 3056488.22 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:05:19,310 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.551e+01 9.218e+01 9.903e+01 1.445e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 05:05:23,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403750 2023-11-24 05:05:33,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2691653.3333333335, ans=0.125 2023-11-24 05:05:50,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2691786.6666666665, ans=0.0 2023-11-24 05:06:02,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2691853.3333333335, ans=0.125 2023-11-24 05:06:10,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2691853.3333333335, ans=0.0 2023-11-24 05:06:13,672 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7000, loss[loss=0.05325, simple_loss=0.07263, pruned_loss=0.008011, audio_tagging_loss=0.008927, over 15809.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09202, pruned_loss=0.01336, audio_tagging_loss=0.008858, over 3048710.87 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:06:21,396 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.63 vs. limit=15.0 2023-11-24 05:06:25,488 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403800 2023-11-24 05:06:33,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2691986.6666666665, ans=0.125 2023-11-24 05:06:34,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2691986.6666666665, ans=0.2 2023-11-24 05:06:46,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2692053.3333333335, ans=0.125 2023-11-24 05:06:52,288 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.79 vs. limit=15.0 2023-11-24 05:06:58,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2692120.0, ans=0.125 2023-11-24 05:07:03,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2692186.6666666665, ans=0.125 2023-11-24 05:07:15,352 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7050, loss[loss=0.06119, simple_loss=0.07328, pruned_loss=0.01386, audio_tagging_loss=0.0107, over 15610.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09126, pruned_loss=0.01325, audio_tagging_loss=0.008978, over 3050634.32 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:07:15,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2692253.3333333335, ans=0.0 2023-11-24 05:07:23,615 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.305e+01 8.871e+01 9.954e+01 1.371e+02, threshold=1.774e+02, percent-clipped=0.0 2023-11-24 05:07:27,366 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403850 2023-11-24 05:07:48,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2692386.6666666665, ans=0.125 2023-11-24 05:07:55,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2692453.3333333335, ans=0.0 2023-11-24 05:07:57,364 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.22 vs. limit=15.0 2023-11-24 05:07:59,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2692453.3333333335, ans=0.125 2023-11-24 05:08:03,011 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.20 vs. limit=15.0 2023-11-24 05:08:16,600 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7100, loss[loss=0.06504, simple_loss=0.09238, pruned_loss=0.01254, audio_tagging_loss=0.006312, over 15210.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09169, pruned_loss=0.0133, audio_tagging_loss=0.008944, over 3049011.45 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:08:28,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2692653.3333333335, ans=0.2 2023-11-24 05:08:30,255 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403900 2023-11-24 05:08:33,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2692653.3333333335, ans=0.0 2023-11-24 05:08:38,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2692653.3333333335, ans=0.125 2023-11-24 05:08:43,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.86 vs. limit=15.0 2023-11-24 05:08:45,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2692720.0, ans=0.2 2023-11-24 05:08:45,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2692720.0, ans=0.0 2023-11-24 05:08:46,181 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.48 vs. limit=22.5 2023-11-24 05:09:20,771 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7150, loss[loss=0.05835, simple_loss=0.07939, pruned_loss=0.009966, audio_tagging_loss=0.008689, over 15222.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09136, pruned_loss=0.01338, audio_tagging_loss=0.009034, over 3045975.45 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:09:25,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2692920.0, ans=0.04949747468305833 2023-11-24 05:09:28,788 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:09:29,598 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 8.812e+01 9.268e+01 1.001e+02 1.464e+02, threshold=1.854e+02, percent-clipped=0.0 2023-11-24 05:09:33,379 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 403950 2023-11-24 05:09:34,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2692986.6666666665, ans=0.125 2023-11-24 05:09:58,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.46 vs. limit=15.0 2023-11-24 05:10:04,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2693120.0, ans=0.0 2023-11-24 05:10:07,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2693120.0, ans=0.95 2023-11-24 05:10:13,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2693186.6666666665, ans=0.125 2023-11-24 05:10:19,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2693186.6666666665, ans=0.125 2023-11-24 05:10:22,857 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7200, loss[loss=0.07761, simple_loss=0.1056, pruned_loss=0.018, audio_tagging_loss=0.006815, over 15248.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09213, pruned_loss=0.01336, audio_tagging_loss=0.009104, over 3041022.53 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:10:24,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2693253.3333333335, ans=0.125 2023-11-24 05:10:27,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2693253.3333333335, ans=0.125 2023-11-24 05:10:29,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2693253.3333333335, ans=0.07 2023-11-24 05:10:33,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2693320.0, ans=0.0 2023-11-24 05:10:34,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404000 2023-11-24 05:10:41,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2693320.0, ans=0.125 2023-11-24 05:10:48,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2693320.0, ans=0.125 2023-11-24 05:10:58,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2693386.6666666665, ans=0.0 2023-11-24 05:11:02,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2693386.6666666665, ans=0.0 2023-11-24 05:11:28,070 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7250, loss[loss=0.07758, simple_loss=0.103, pruned_loss=0.01458, audio_tagging_loss=0.01151, over 15307.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09208, pruned_loss=0.01333, audio_tagging_loss=0.009131, over 3041955.86 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:11:32,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.75 vs. limit=15.0 2023-11-24 05:11:32,099 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.50 vs. limit=12.0 2023-11-24 05:11:33,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2693586.6666666665, ans=0.0 2023-11-24 05:11:36,163 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.320e+01 8.579e+01 9.139e+01 1.008e+02 1.154e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 05:11:40,487 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404050 2023-11-24 05:12:08,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2693786.6666666665, ans=0.125 2023-11-24 05:12:14,062 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.44 vs. limit=6.0 2023-11-24 05:12:25,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2693853.3333333335, ans=0.125 2023-11-24 05:12:30,883 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7300, loss[loss=0.05341, simple_loss=0.07559, pruned_loss=0.007368, audio_tagging_loss=0.008244, over 15185.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.0922, pruned_loss=0.01324, audio_tagging_loss=0.00899, over 3041260.33 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:12:43,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404100 2023-11-24 05:12:54,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2694053.3333333335, ans=0.125 2023-11-24 05:13:03,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2694053.3333333335, ans=0.125 2023-11-24 05:13:07,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2694120.0, ans=0.125 2023-11-24 05:13:10,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2694120.0, ans=0.125 2023-11-24 05:13:12,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2694120.0, ans=0.0 2023-11-24 05:13:27,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2694186.6666666665, ans=0.0 2023-11-24 05:13:32,970 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7350, loss[loss=0.06827, simple_loss=0.09125, pruned_loss=0.01219, audio_tagging_loss=0.01045, over 16321.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09221, pruned_loss=0.01331, audio_tagging_loss=0.008943, over 3037959.30 frames. ], batch size: 62, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:13:33,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.04 vs. limit=22.5 2023-11-24 05:13:35,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2694253.3333333335, ans=0.125 2023-11-24 05:13:35,876 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=22.5 2023-11-24 05:13:39,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2694253.3333333335, ans=0.125 2023-11-24 05:13:43,685 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.435e+01 9.060e+01 9.696e+01 1.244e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 05:13:45,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404150 2023-11-24 05:13:48,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2694320.0, ans=0.09899494936611666 2023-11-24 05:13:52,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2694320.0, ans=0.1 2023-11-24 05:14:01,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2694386.6666666665, ans=0.125 2023-11-24 05:14:18,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2694453.3333333335, ans=0.125 2023-11-24 05:14:34,634 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7400, loss[loss=0.06861, simple_loss=0.09173, pruned_loss=0.01407, audio_tagging_loss=0.008676, over 15936.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09159, pruned_loss=0.01331, audio_tagging_loss=0.008892, over 3045923.11 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:14:44,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2694586.6666666665, ans=0.125 2023-11-24 05:14:47,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404200 2023-11-24 05:15:09,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2694720.0, ans=0.125 2023-11-24 05:15:11,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2694786.6666666665, ans=0.025 2023-11-24 05:15:12,024 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2694786.6666666665, ans=0.125 2023-11-24 05:15:26,474 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:15:30,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2694853.3333333335, ans=0.125 2023-11-24 05:15:37,380 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7450, loss[loss=0.07246, simple_loss=0.09794, pruned_loss=0.01433, audio_tagging_loss=0.009162, over 15158.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09231, pruned_loss=0.01347, audio_tagging_loss=0.008868, over 3045233.95 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:15:47,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2694920.0, ans=0.0 2023-11-24 05:15:48,987 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.292e+01 8.446e+01 9.170e+01 9.700e+01 1.341e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-24 05:15:50,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404250 2023-11-24 05:15:53,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2694986.6666666665, ans=0.0 2023-11-24 05:16:05,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2695053.3333333335, ans=0.2 2023-11-24 05:16:07,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0 2023-11-24 05:16:11,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2695053.3333333335, ans=0.1 2023-11-24 05:16:17,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2695120.0, ans=0.125 2023-11-24 05:16:19,875 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0 2023-11-24 05:16:32,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2695186.6666666665, ans=0.0 2023-11-24 05:16:37,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2695186.6666666665, ans=0.1 2023-11-24 05:16:40,531 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7500, loss[loss=0.06534, simple_loss=0.09094, pruned_loss=0.01263, audio_tagging_loss=0.007241, over 14546.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09281, pruned_loss=0.01348, audio_tagging_loss=0.008805, over 3045634.13 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:16:42,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2695253.3333333335, ans=0.1 2023-11-24 05:16:45,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2695253.3333333335, ans=0.125 2023-11-24 05:16:52,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404300 2023-11-24 05:17:07,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2695386.6666666665, ans=0.125 2023-11-24 05:17:24,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2695453.3333333335, ans=0.125 2023-11-24 05:17:24,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2695453.3333333335, ans=0.125 2023-11-24 05:17:39,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2695520.0, ans=0.2 2023-11-24 05:17:41,548 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7550, loss[loss=0.0664, simple_loss=0.09361, pruned_loss=0.01173, audio_tagging_loss=0.007865, over 14723.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09216, pruned_loss=0.01338, audio_tagging_loss=0.008838, over 3037135.75 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:17:51,951 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.237e+01 8.670e+01 9.167e+01 9.885e+01 1.310e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 05:17:53,356 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404350 2023-11-24 05:18:04,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2695653.3333333335, ans=0.2 2023-11-24 05:18:04,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2695653.3333333335, ans=0.125 2023-11-24 05:18:20,690 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:18:21,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2695786.6666666665, ans=0.0 2023-11-24 05:18:43,189 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7600, loss[loss=0.06021, simple_loss=0.07275, pruned_loss=0.01315, audio_tagging_loss=0.01069, over 14624.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09097, pruned_loss=0.01328, audio_tagging_loss=0.008937, over 3041567.50 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:18:49,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2695920.0, ans=0.07 2023-11-24 05:18:56,243 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404400 2023-11-24 05:19:00,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2695986.6666666665, ans=0.0 2023-11-24 05:19:07,494 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.97 vs. limit=15.0 2023-11-24 05:19:45,981 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7650, loss[loss=0.07089, simple_loss=0.1001, pruned_loss=0.01399, audio_tagging_loss=0.006854, over 15665.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09111, pruned_loss=0.01326, audio_tagging_loss=0.00883, over 3045325.98 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:19:49,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2696253.3333333335, ans=0.0 2023-11-24 05:19:51,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2696253.3333333335, ans=0.0 2023-11-24 05:19:57,213 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.265e+01 8.997e+01 9.695e+01 1.208e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-24 05:19:58,572 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404450 2023-11-24 05:20:04,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2696320.0, ans=0.0 2023-11-24 05:20:21,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2696453.3333333335, ans=0.2 2023-11-24 05:20:23,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2696453.3333333335, ans=0.125 2023-11-24 05:20:27,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.97 vs. limit=22.5 2023-11-24 05:20:30,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2696453.3333333335, ans=0.2 2023-11-24 05:20:34,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.39 vs. limit=22.5 2023-11-24 05:20:48,147 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7700, loss[loss=0.0441, simple_loss=0.05143, pruned_loss=0.007929, audio_tagging_loss=0.01046, over 15258.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09072, pruned_loss=0.01309, audio_tagging_loss=0.008929, over 3045398.84 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:21:00,134 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404500 2023-11-24 05:21:06,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.26 vs. limit=10.0 2023-11-24 05:21:49,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2696920.0, ans=0.1 2023-11-24 05:21:50,153 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7750, loss[loss=0.088, simple_loss=0.1163, pruned_loss=0.02141, audio_tagging_loss=0.008445, over 15763.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09081, pruned_loss=0.01318, audio_tagging_loss=0.00901, over 3038055.54 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:21:54,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2696920.0, ans=0.125 2023-11-24 05:22:01,916 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.444e+01 9.119e+01 9.673e+01 1.144e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-24 05:22:03,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404550 2023-11-24 05:22:46,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2697186.6666666665, ans=0.125 2023-11-24 05:22:50,324 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.37 vs. limit=15.0 2023-11-24 05:22:53,784 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7800, loss[loss=0.06041, simple_loss=0.08246, pruned_loss=0.01161, audio_tagging_loss=0.007568, over 15184.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09062, pruned_loss=0.01309, audio_tagging_loss=0.008922, over 3035708.46 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:23:05,798 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404600 2023-11-24 05:23:27,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2697386.6666666665, ans=0.2 2023-11-24 05:23:42,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2697453.3333333335, ans=0.0 2023-11-24 05:23:48,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2697520.0, ans=0.125 2023-11-24 05:23:56,293 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7850, loss[loss=0.07299, simple_loss=0.09726, pruned_loss=0.01402, audio_tagging_loss=0.01033, over 15194.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09093, pruned_loss=0.01316, audio_tagging_loss=0.008948, over 3043784.74 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:24:02,403 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.33 vs. limit=15.0 2023-11-24 05:24:05,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2697586.6666666665, ans=0.0 2023-11-24 05:24:07,514 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.405e+01 8.451e+01 9.106e+01 9.579e+01 1.135e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 05:24:08,843 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404650 2023-11-24 05:24:15,393 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.50 vs. limit=15.0 2023-11-24 05:24:17,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2697653.3333333335, ans=0.1 2023-11-24 05:24:29,867 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.50 vs. limit=15.0 2023-11-24 05:24:57,989 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7900, loss[loss=0.07213, simple_loss=0.1018, pruned_loss=0.01291, audio_tagging_loss=0.008306, over 15369.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09166, pruned_loss=0.01329, audio_tagging_loss=0.008953, over 3043219.79 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:25:11,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404700 2023-11-24 05:25:22,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2698053.3333333335, ans=0.0 2023-11-24 05:26:01,544 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 7950, loss[loss=0.08009, simple_loss=0.1096, pruned_loss=0.01698, audio_tagging_loss=0.008301, over 15567.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09141, pruned_loss=0.01321, audio_tagging_loss=0.009124, over 3043413.16 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:26:09,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2698253.3333333335, ans=0.07 2023-11-24 05:26:11,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2698253.3333333335, ans=0.1 2023-11-24 05:26:12,224 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.346e+01 8.927e+01 9.659e+01 1.276e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-24 05:26:13,540 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404750 2023-11-24 05:26:15,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff3.min_abs, batch_count=2698320.0, ans=0.2 2023-11-24 05:26:15,939 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 05:26:29,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.31 vs. limit=22.5 2023-11-24 05:26:32,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2698386.6666666665, ans=0.125 2023-11-24 05:26:34,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.99 vs. limit=22.5 2023-11-24 05:27:03,770 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8000, loss[loss=0.08182, simple_loss=0.1147, pruned_loss=0.01409, audio_tagging_loss=0.01038, over 16541.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09192, pruned_loss=0.01333, audio_tagging_loss=0.009171, over 3049302.61 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:27:05,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2698586.6666666665, ans=0.2 2023-11-24 05:27:16,451 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404800 2023-11-24 05:27:21,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2698653.3333333335, ans=0.0 2023-11-24 05:27:44,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2698786.6666666665, ans=0.1 2023-11-24 05:27:46,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2698786.6666666665, ans=0.125 2023-11-24 05:28:06,018 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8050, loss[loss=0.06228, simple_loss=0.07832, pruned_loss=0.01248, audio_tagging_loss=0.01064, over 16468.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.0918, pruned_loss=0.01329, audio_tagging_loss=0.009214, over 3056320.06 frames. ], batch size: 63, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:28:17,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.661e+01 8.621e+01 9.093e+01 9.694e+01 1.272e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 05:28:19,191 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404850 2023-11-24 05:28:19,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2698986.6666666665, ans=0.125 2023-11-24 05:28:36,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2699053.3333333335, ans=0.0 2023-11-24 05:28:42,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2699120.0, ans=0.125 2023-11-24 05:28:53,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2699120.0, ans=0.2 2023-11-24 05:28:58,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2699186.6666666665, ans=0.0 2023-11-24 05:28:59,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2699186.6666666665, ans=0.125 2023-11-24 05:29:08,653 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8100, loss[loss=0.07144, simple_loss=0.09465, pruned_loss=0.01571, audio_tagging_loss=0.008403, over 14138.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09125, pruned_loss=0.01313, audio_tagging_loss=0.009143, over 3051141.18 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:29:14,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2699253.3333333335, ans=0.0 2023-11-24 05:29:15,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2699253.3333333335, ans=0.125 2023-11-24 05:29:21,387 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404900 2023-11-24 05:29:43,654 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.04 vs. limit=22.5 2023-11-24 05:29:58,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2699520.0, ans=0.2 2023-11-24 05:30:10,918 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8150, loss[loss=0.05632, simple_loss=0.07267, pruned_loss=0.007194, audio_tagging_loss=0.01279, over 15728.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09209, pruned_loss=0.01333, audio_tagging_loss=0.008901, over 3052547.46 frames. ], batch size: 62, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:30:19,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2699586.6666666665, ans=0.125 2023-11-24 05:30:22,676 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.549e+01 9.218e+01 9.858e+01 1.332e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 05:30:22,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 404950 2023-11-24 05:30:32,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2699653.3333333335, ans=0.1 2023-11-24 05:30:45,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2699720.0, ans=0.0 2023-11-24 05:30:55,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2699786.6666666665, ans=0.125 2023-11-24 05:31:01,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2699853.3333333335, ans=0.125 2023-11-24 05:31:01,931 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.25 vs. limit=22.5 2023-11-24 05:31:12,013 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8200, loss[loss=0.07417, simple_loss=0.1005, pruned_loss=0.01461, audio_tagging_loss=0.009294, over 14808.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09209, pruned_loss=0.01333, audio_tagging_loss=0.008849, over 3050703.32 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:31:12,068 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 05:31:24,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405000 2023-11-24 05:31:26,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2699986.6666666665, ans=0.2 2023-11-24 05:31:28,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2699986.6666666665, ans=0.2 2023-11-24 05:31:34,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2699986.6666666665, ans=0.2 2023-11-24 05:31:38,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=2700053.3333333335, ans=0.1 2023-11-24 05:31:51,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2700120.0, ans=0.09899494936611666 2023-11-24 05:31:58,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2700120.0, ans=0.2 2023-11-24 05:32:14,718 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8250, loss[loss=0.04429, simple_loss=0.05624, pruned_loss=0.006594, audio_tagging_loss=0.009576, over 14176.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09114, pruned_loss=0.0132, audio_tagging_loss=0.008801, over 3047325.26 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:32:27,221 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405050 2023-11-24 05:32:28,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.524e+01 9.156e+01 9.753e+01 1.627e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-24 05:32:28,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.06 vs. limit=6.0 2023-11-24 05:32:47,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.67 vs. limit=15.0 2023-11-24 05:32:55,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2700453.3333333335, ans=0.1 2023-11-24 05:33:12,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=2700520.0, ans=22.5 2023-11-24 05:33:16,932 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8300, loss[loss=0.0638, simple_loss=0.08715, pruned_loss=0.01275, audio_tagging_loss=0.007471, over 15046.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09099, pruned_loss=0.01322, audio_tagging_loss=0.008871, over 3053824.62 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:33:28,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405100 2023-11-24 05:33:54,685 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.05 vs. limit=6.0 2023-11-24 05:34:18,255 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8350, loss[loss=0.04915, simple_loss=0.06875, pruned_loss=0.006117, audio_tagging_loss=0.008655, over 14570.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09156, pruned_loss=0.01326, audio_tagging_loss=0.008785, over 3053395.57 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0 2023-11-24 05:34:21,251 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.14 vs. limit=15.0 2023-11-24 05:34:29,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2700986.6666666665, ans=0.2 2023-11-24 05:34:29,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2700986.6666666665, ans=0.04949747468305833 2023-11-24 05:34:30,197 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405150 2023-11-24 05:34:31,208 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.407e+01 9.166e+01 9.742e+01 1.258e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 05:34:37,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2700986.6666666665, ans=0.07 2023-11-24 05:34:55,908 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.62 vs. limit=22.5 2023-11-24 05:35:14,794 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.14 vs. limit=6.0 2023-11-24 05:35:19,474 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8400, loss[loss=0.06341, simple_loss=0.09099, pruned_loss=0.01134, audio_tagging_loss=0.00657, over 15434.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09144, pruned_loss=0.01328, audio_tagging_loss=0.008842, over 3051536.88 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:35:32,228 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405200 2023-11-24 05:35:49,599 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0 2023-11-24 05:35:51,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2701386.6666666665, ans=0.1 2023-11-24 05:35:52,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2701386.6666666665, ans=0.0 2023-11-24 05:35:54,939 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2701453.3333333335, ans=0.125 2023-11-24 05:36:21,622 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8450, loss[loss=0.06275, simple_loss=0.08766, pruned_loss=0.01136, audio_tagging_loss=0.007564, over 16706.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09074, pruned_loss=0.01321, audio_tagging_loss=0.008921, over 3056029.42 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:36:22,504 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.28 vs. limit=15.0 2023-11-24 05:36:26,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.76 vs. limit=15.0 2023-11-24 05:36:33,797 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405250 2023-11-24 05:36:34,768 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.098e+01 8.659e+01 9.489e+01 1.219e+02, threshold=1.732e+02, percent-clipped=0.0 2023-11-24 05:36:45,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2701720.0, ans=0.125 2023-11-24 05:36:59,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2701786.6666666665, ans=0.0 2023-11-24 05:37:23,348 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8500, loss[loss=0.07349, simple_loss=0.1101, pruned_loss=0.01146, audio_tagging_loss=0.006986, over 16009.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09134, pruned_loss=0.0133, audio_tagging_loss=0.008881, over 3061173.83 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:37:25,134 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.50 vs. limit=12.0 2023-11-24 05:37:27,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=2701920.0, ans=0.02 2023-11-24 05:37:28,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2701920.0, ans=0.125 2023-11-24 05:37:28,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2701920.0, ans=0.2 2023-11-24 05:37:35,325 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405300 2023-11-24 05:38:08,836 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5 2023-11-24 05:38:24,782 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8550, loss[loss=0.04537, simple_loss=0.05939, pruned_loss=0.007664, audio_tagging_loss=0.008014, over 14425.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09011, pruned_loss=0.013, audio_tagging_loss=0.00899, over 3055965.13 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:38:25,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2702253.3333333335, ans=0.125 2023-11-24 05:38:29,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2702253.3333333335, ans=0.2 2023-11-24 05:38:39,290 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405350 2023-11-24 05:38:39,773 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.13 vs. limit=15.0 2023-11-24 05:38:40,297 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.730e+01 8.556e+01 9.186e+01 9.838e+01 1.237e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-24 05:38:43,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2702320.0, ans=0.125 2023-11-24 05:38:44,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2702320.0, ans=0.0 2023-11-24 05:38:44,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2702320.0, ans=0.125 2023-11-24 05:38:47,884 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2702320.0, ans=0.1 2023-11-24 05:38:54,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2702386.6666666665, ans=0.1 2023-11-24 05:39:02,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2702453.3333333335, ans=0.125 2023-11-24 05:39:10,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.23 vs. limit=6.0 2023-11-24 05:39:25,838 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=22.5 2023-11-24 05:39:28,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2702586.6666666665, ans=0.0 2023-11-24 05:39:29,209 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8600, loss[loss=0.05011, simple_loss=0.06209, pruned_loss=0.007591, audio_tagging_loss=0.01147, over 14928.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09018, pruned_loss=0.01311, audio_tagging_loss=0.008984, over 3050195.54 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:39:41,052 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405400 2023-11-24 05:39:49,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2702653.3333333335, ans=0.125 2023-11-24 05:39:53,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2702720.0, ans=0.0 2023-11-24 05:39:54,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2702720.0, ans=0.0 2023-11-24 05:40:02,757 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.38 vs. limit=22.5 2023-11-24 05:40:23,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2702853.3333333335, ans=0.1 2023-11-24 05:40:31,244 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8650, loss[loss=0.06374, simple_loss=0.09162, pruned_loss=0.01057, audio_tagging_loss=0.007366, over 16376.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09083, pruned_loss=0.01307, audio_tagging_loss=0.008945, over 3055020.22 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:40:32,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2702920.0, ans=0.09899494936611666 2023-11-24 05:40:40,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.14 vs. limit=15.0 2023-11-24 05:40:41,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2702920.0, ans=0.1 2023-11-24 05:40:43,350 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405450 2023-11-24 05:40:44,410 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.340e+01 8.964e+01 9.438e+01 1.251e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 05:41:15,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2703120.0, ans=0.125 2023-11-24 05:41:30,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2703186.6666666665, ans=0.125 2023-11-24 05:41:33,424 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8700, loss[loss=0.09272, simple_loss=0.1291, pruned_loss=0.01841, audio_tagging_loss=0.009779, over 15607.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09106, pruned_loss=0.01323, audio_tagging_loss=0.00906, over 3053485.30 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:41:46,887 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405500 2023-11-24 05:41:46,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2703320.0, ans=0.125 2023-11-24 05:41:48,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2703320.0, ans=0.2 2023-11-24 05:41:53,074 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.32 vs. limit=15.0 2023-11-24 05:42:02,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2703386.6666666665, ans=0.2 2023-11-24 05:42:32,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2703520.0, ans=0.125 2023-11-24 05:42:37,533 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8750, loss[loss=0.07795, simple_loss=0.1107, pruned_loss=0.01245, audio_tagging_loss=0.01016, over 15310.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09246, pruned_loss=0.01348, audio_tagging_loss=0.009082, over 3060030.57 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:42:50,135 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405550 2023-11-24 05:42:51,165 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.699e+01 9.632e+01 1.055e+02 1.285e+02, threshold=1.926e+02, percent-clipped=0.0 2023-11-24 05:42:58,811 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.68 vs. limit=22.5 2023-11-24 05:42:59,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2703653.3333333335, ans=0.125 2023-11-24 05:43:29,894 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.12 vs. limit=15.0 2023-11-24 05:43:29,968 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.87 vs. limit=15.0 2023-11-24 05:43:39,334 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8800, loss[loss=0.07965, simple_loss=0.1039, pruned_loss=0.01746, audio_tagging_loss=0.01024, over 14933.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09392, pruned_loss=0.0139, audio_tagging_loss=0.009067, over 3065492.24 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:43:42,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.64 vs. limit=22.5 2023-11-24 05:43:46,860 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:43:51,391 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405600 2023-11-24 05:44:02,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.43 vs. limit=22.5 2023-11-24 05:44:36,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2704186.6666666665, ans=0.035 2023-11-24 05:44:41,164 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8850, loss[loss=0.09148, simple_loss=0.1209, pruned_loss=0.02334, audio_tagging_loss=0.00768, over 14624.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09308, pruned_loss=0.01372, audio_tagging_loss=0.009108, over 3060999.07 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:44:49,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2704253.3333333335, ans=0.125 2023-11-24 05:44:53,064 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 05:44:54,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405650 2023-11-24 05:44:54,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2704320.0, ans=0.1 2023-11-24 05:44:55,471 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.409e+01 8.541e+01 8.982e+01 9.482e+01 1.517e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-24 05:45:26,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2704453.3333333335, ans=0.125 2023-11-24 05:45:27,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2704453.3333333335, ans=0.125 2023-11-24 05:45:28,088 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.79 vs. limit=15.0 2023-11-24 05:45:32,424 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:45:44,003 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8900, loss[loss=0.03595, simple_loss=0.03794, pruned_loss=0.005298, audio_tagging_loss=0.01168, over 14371.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09173, pruned_loss=0.01355, audio_tagging_loss=0.009052, over 3060966.78 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:45:56,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405700 2023-11-24 05:46:05,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2704653.3333333335, ans=0.125 2023-11-24 05:46:10,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2704720.0, ans=0.0 2023-11-24 05:46:18,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2704720.0, ans=0.04949747468305833 2023-11-24 05:46:45,642 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 8950, loss[loss=0.07325, simple_loss=0.09403, pruned_loss=0.01519, audio_tagging_loss=0.01105, over 15405.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09183, pruned_loss=0.01351, audio_tagging_loss=0.009012, over 3059559.65 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:46:51,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2704920.0, ans=0.0 2023-11-24 05:46:58,112 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405750 2023-11-24 05:46:59,192 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.112e+01 8.729e+01 9.252e+01 9.908e+01 1.235e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 05:47:26,313 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.89 vs. limit=22.5 2023-11-24 05:47:29,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=12.0 2023-11-24 05:47:33,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2705186.6666666665, ans=0.0 2023-11-24 05:47:47,505 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9000, loss[loss=0.07818, simple_loss=0.1035, pruned_loss=0.01888, audio_tagging_loss=0.007563, over 14050.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09188, pruned_loss=0.01357, audio_tagging_loss=0.00899, over 3056538.36 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:47:47,506 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 05:48:27,881 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6103, 3.7904, 3.9682, 3.4668], device='cuda:3') 2023-11-24 05:48:30,289 INFO [train_asr.py:1253] (3/4) Epoch 34, validation: loss=0.05898, simple_loss=0.05084, pruned_loss=0.005097, audio_tagging_loss=0.02846, over 4681554.00 frames. 2023-11-24 05:48:30,290 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 05:48:34,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2705253.3333333335, ans=0.04949747468305833 2023-11-24 05:48:42,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405800 2023-11-24 05:48:49,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2705320.0, ans=0.025 2023-11-24 05:48:52,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2705320.0, ans=0.125 2023-11-24 05:48:58,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2705386.6666666665, ans=0.2 2023-11-24 05:48:59,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2705386.6666666665, ans=0.1 2023-11-24 05:49:06,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2705453.3333333335, ans=0.0 2023-11-24 05:49:18,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2705453.3333333335, ans=0.0 2023-11-24 05:49:32,030 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9050, loss[loss=0.06828, simple_loss=0.09877, pruned_loss=0.01029, audio_tagging_loss=0.008612, over 16364.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09199, pruned_loss=0.01351, audio_tagging_loss=0.008905, over 3057389.14 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:49:44,533 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405850 2023-11-24 05:49:45,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2705653.3333333335, ans=0.125 2023-11-24 05:49:46,750 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.178e+01 8.606e+01 9.162e+01 9.789e+01 1.610e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 05:50:00,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2705720.0, ans=0.0 2023-11-24 05:50:03,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2705720.0, ans=0.1 2023-11-24 05:50:05,591 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2705720.0, ans=0.025 2023-11-24 05:50:34,167 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.46 vs. limit=15.0 2023-11-24 05:50:34,568 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9100, loss[loss=0.05553, simple_loss=0.07755, pruned_loss=0.008348, audio_tagging_loss=0.008407, over 15430.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09164, pruned_loss=0.01344, audio_tagging_loss=0.008873, over 3054415.02 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:50:38,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.48 vs. limit=15.0 2023-11-24 05:50:47,521 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405900 2023-11-24 05:51:28,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2706186.6666666665, ans=0.0 2023-11-24 05:51:31,469 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.09 vs. limit=15.0 2023-11-24 05:51:37,134 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9150, loss[loss=0.07541, simple_loss=0.0995, pruned_loss=0.01707, audio_tagging_loss=0.008589, over 14836.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09212, pruned_loss=0.01361, audio_tagging_loss=0.008879, over 3047787.48 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 05:51:39,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2706253.3333333335, ans=0.2 2023-11-24 05:51:49,174 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 405950 2023-11-24 05:51:51,503 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.335e+01 8.866e+01 9.361e+01 1.025e+02 1.380e+02, threshold=1.872e+02, percent-clipped=0.0 2023-11-24 05:52:00,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2706386.6666666665, ans=0.125 2023-11-24 05:52:14,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2706453.3333333335, ans=0.125 2023-11-24 05:52:19,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2706453.3333333335, ans=0.1 2023-11-24 05:52:39,405 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9200, loss[loss=0.07022, simple_loss=0.09749, pruned_loss=0.01285, audio_tagging_loss=0.008626, over 14192.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09096, pruned_loss=0.01337, audio_tagging_loss=0.008874, over 3047109.71 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:52:51,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406000 2023-11-24 05:52:52,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2706653.3333333335, ans=0.125 2023-11-24 05:52:56,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2706653.3333333335, ans=0.125 2023-11-24 05:53:21,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2706786.6666666665, ans=0.125 2023-11-24 05:53:21,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2706786.6666666665, ans=0.1 2023-11-24 05:53:41,423 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9250, loss[loss=0.04389, simple_loss=0.05631, pruned_loss=0.007136, audio_tagging_loss=0.008595, over 14533.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09057, pruned_loss=0.01324, audio_tagging_loss=0.00887, over 3043864.66 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:53:54,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406050 2023-11-24 05:53:57,353 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 8.286e+01 8.929e+01 9.652e+01 1.621e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 05:53:57,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2706986.6666666665, ans=0.125 2023-11-24 05:54:00,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2706986.6666666665, ans=0.0 2023-11-24 05:54:08,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2707053.3333333335, ans=0.1 2023-11-24 05:54:43,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2707253.3333333335, ans=0.2 2023-11-24 05:54:45,407 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9300, loss[loss=0.07632, simple_loss=0.09548, pruned_loss=0.01805, audio_tagging_loss=0.01053, over 14855.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09087, pruned_loss=0.01327, audio_tagging_loss=0.008868, over 3046785.06 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:54:47,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2707253.3333333335, ans=0.125 2023-11-24 05:54:57,426 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.45 vs. limit=10.0 2023-11-24 05:54:57,989 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406100 2023-11-24 05:55:17,335 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.01 vs. limit=15.0 2023-11-24 05:55:19,591 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.45 vs. limit=15.0 2023-11-24 05:55:22,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2707453.3333333335, ans=0.125 2023-11-24 05:55:23,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2707453.3333333335, ans=0.125 2023-11-24 05:55:25,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2707453.3333333335, ans=0.125 2023-11-24 05:55:33,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2707453.3333333335, ans=0.0 2023-11-24 05:55:34,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2707520.0, ans=0.0 2023-11-24 05:55:35,743 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:55:36,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2707520.0, ans=0.125 2023-11-24 05:55:37,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2707520.0, ans=10.0 2023-11-24 05:55:40,469 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:55:45,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2707520.0, ans=0.0 2023-11-24 05:55:47,268 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9350, loss[loss=0.06401, simple_loss=0.08645, pruned_loss=0.01295, audio_tagging_loss=0.007831, over 14529.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09098, pruned_loss=0.01345, audio_tagging_loss=0.008958, over 3047394.12 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:55:54,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2707586.6666666665, ans=0.0 2023-11-24 05:55:59,138 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406150 2023-11-24 05:56:01,327 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.542e+01 9.315e+01 1.015e+02 1.329e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 05:56:02,785 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 05:56:13,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2707720.0, ans=0.125 2023-11-24 05:56:15,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2707720.0, ans=0.09899494936611666 2023-11-24 05:56:15,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2707720.0, ans=0.125 2023-11-24 05:56:33,377 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.58 vs. limit=15.0 2023-11-24 05:56:45,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2707853.3333333335, ans=0.0 2023-11-24 05:56:49,159 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9400, loss[loss=0.05571, simple_loss=0.06854, pruned_loss=0.01066, audio_tagging_loss=0.01078, over 15446.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09067, pruned_loss=0.0135, audio_tagging_loss=0.009065, over 3049088.88 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:56:49,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2707920.0, ans=0.0 2023-11-24 05:57:02,299 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406200 2023-11-24 05:57:09,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2707986.6666666665, ans=0.125 2023-11-24 05:57:36,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=2708120.0, ans=0.1 2023-11-24 05:57:39,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2708186.6666666665, ans=0.09899494936611666 2023-11-24 05:57:50,056 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 05:57:50,616 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0 2023-11-24 05:57:52,356 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9450, loss[loss=0.06659, simple_loss=0.08658, pruned_loss=0.01275, audio_tagging_loss=0.01056, over 16334.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09125, pruned_loss=0.01358, audio_tagging_loss=0.009185, over 3062518.08 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:57:59,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2708253.3333333335, ans=0.125 2023-11-24 05:58:05,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406250 2023-11-24 05:58:07,679 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.368e+01 8.774e+01 9.342e+01 1.287e+02, threshold=1.755e+02, percent-clipped=0.0 2023-11-24 05:58:15,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2708320.0, ans=0.125 2023-11-24 05:58:16,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2708386.6666666665, ans=0.05 2023-11-24 05:58:32,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2708453.3333333335, ans=22.5 2023-11-24 05:58:33,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.17 vs. limit=10.0 2023-11-24 05:58:41,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2708520.0, ans=0.0 2023-11-24 05:58:55,174 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9500, loss[loss=0.09861, simple_loss=0.1249, pruned_loss=0.02848, audio_tagging_loss=0.007677, over 16155.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09099, pruned_loss=0.0137, audio_tagging_loss=0.009231, over 3064347.32 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 05:58:57,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2708586.6666666665, ans=0.09899494936611666 2023-11-24 05:59:01,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2708586.6666666665, ans=0.0 2023-11-24 05:59:07,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406300 2023-11-24 05:59:07,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2708653.3333333335, ans=0.125 2023-11-24 05:59:07,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2708653.3333333335, ans=0.125 2023-11-24 05:59:24,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2708720.0, ans=10.0 2023-11-24 05:59:56,986 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9550, loss[loss=0.0806, simple_loss=0.1158, pruned_loss=0.01591, audio_tagging_loss=0.006793, over 15668.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09093, pruned_loss=0.01364, audio_tagging_loss=0.009291, over 3055916.80 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0 2023-11-24 06:00:09,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406350 2023-11-24 06:00:13,200 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.403e+01 9.161e+01 9.740e+01 1.180e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 06:00:24,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2709053.3333333335, ans=0.125 2023-11-24 06:00:24,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2709053.3333333335, ans=0.1 2023-11-24 06:00:27,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2709053.3333333335, ans=0.125 2023-11-24 06:00:30,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.whiten.whitening_limit, batch_count=2709053.3333333335, ans=12.0 2023-11-24 06:00:40,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2709120.0, ans=0.125 2023-11-24 06:00:51,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0 2023-11-24 06:00:53,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2709186.6666666665, ans=0.0 2023-11-24 06:00:57,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2709186.6666666665, ans=0.2 2023-11-24 06:00:59,408 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9600, loss[loss=0.04453, simple_loss=0.05603, pruned_loss=0.008657, audio_tagging_loss=0.007858, over 14637.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09108, pruned_loss=0.01352, audio_tagging_loss=0.009273, over 3060593.36 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 06:01:13,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406400 2023-11-24 06:01:14,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2709320.0, ans=0.0 2023-11-24 06:01:20,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.35 vs. limit=15.0 2023-11-24 06:01:21,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2709320.0, ans=0.1 2023-11-24 06:01:47,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2709453.3333333335, ans=0.125 2023-11-24 06:01:55,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2709520.0, ans=0.0 2023-11-24 06:02:00,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2709520.0, ans=0.0 2023-11-24 06:02:03,407 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9650, loss[loss=0.08065, simple_loss=0.115, pruned_loss=0.01725, audio_tagging_loss=0.005888, over 16757.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09145, pruned_loss=0.01348, audio_tagging_loss=0.009149, over 3060414.86 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 32.0 2023-11-24 06:02:10,081 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.88 vs. limit=10.0 2023-11-24 06:02:12,363 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.02 vs. limit=15.0 2023-11-24 06:02:15,215 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406450 2023-11-24 06:02:18,592 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.885e+01 8.449e+01 8.956e+01 9.585e+01 1.411e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-24 06:02:32,802 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.82 vs. limit=15.0 2023-11-24 06:02:48,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2709786.6666666665, ans=0.5 2023-11-24 06:03:04,826 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9700, loss[loss=0.06683, simple_loss=0.08292, pruned_loss=0.01659, audio_tagging_loss=0.008777, over 14995.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09159, pruned_loss=0.01341, audio_tagging_loss=0.008956, over 3057790.43 frames. ], batch size: 61, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:03:12,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2709920.0, ans=0.125 2023-11-24 06:03:16,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406500 2023-11-24 06:03:23,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2709986.6666666665, ans=0.0 2023-11-24 06:03:23,383 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.39 vs. limit=22.5 2023-11-24 06:03:46,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2710120.0, ans=0.0 2023-11-24 06:04:01,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2710186.6666666665, ans=0.1 2023-11-24 06:04:06,203 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9750, loss[loss=0.07687, simple_loss=0.1073, pruned_loss=0.01427, audio_tagging_loss=0.008928, over 13906.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09094, pruned_loss=0.01318, audio_tagging_loss=0.00893, over 3055164.55 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:04:19,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406550 2023-11-24 06:04:21,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2710320.0, ans=0.0 2023-11-24 06:04:24,474 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.74 vs. limit=10.0 2023-11-24 06:04:24,968 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.565e+01 9.211e+01 9.883e+01 1.176e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-24 06:04:37,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2710386.6666666665, ans=0.125 2023-11-24 06:04:50,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2710453.3333333335, ans=0.125 2023-11-24 06:04:50,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.81 vs. limit=15.0 2023-11-24 06:04:55,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.73 vs. limit=15.0 2023-11-24 06:04:57,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2710520.0, ans=0.025 2023-11-24 06:05:09,514 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9800, loss[loss=0.07485, simple_loss=0.09994, pruned_loss=0.01646, audio_tagging_loss=0.008424, over 14888.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09048, pruned_loss=0.01313, audio_tagging_loss=0.008962, over 3045931.89 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:05:22,228 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406600 2023-11-24 06:05:23,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2710653.3333333335, ans=0.125 2023-11-24 06:05:32,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2710653.3333333335, ans=0.125 2023-11-24 06:05:34,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2710720.0, ans=0.0 2023-11-24 06:05:48,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2710786.6666666665, ans=10.0 2023-11-24 06:05:54,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2710786.6666666665, ans=0.0 2023-11-24 06:06:04,972 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 06:06:12,010 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9850, loss[loss=0.07665, simple_loss=0.1046, pruned_loss=0.01516, audio_tagging_loss=0.009201, over 15115.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.0917, pruned_loss=0.01346, audio_tagging_loss=0.008849, over 3046478.42 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:06:23,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406650 2023-11-24 06:06:27,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2710986.6666666665, ans=0.0 2023-11-24 06:06:28,339 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.400e+01 8.541e+01 9.131e+01 9.880e+01 1.241e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 06:06:35,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2711053.3333333335, ans=0.1 2023-11-24 06:06:43,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2711053.3333333335, ans=0.125 2023-11-24 06:06:48,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2711120.0, ans=0.2 2023-11-24 06:06:50,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2711120.0, ans=0.125 2023-11-24 06:06:52,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2711120.0, ans=0.0 2023-11-24 06:07:13,494 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9900, loss[loss=0.06429, simple_loss=0.09055, pruned_loss=0.01017, audio_tagging_loss=0.008842, over 15279.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09182, pruned_loss=0.01339, audio_tagging_loss=0.008776, over 3044375.82 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:07:13,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2711253.3333333335, ans=0.2 2023-11-24 06:07:27,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406700 2023-11-24 06:07:49,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.09 vs. limit=15.0 2023-11-24 06:08:03,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2711520.0, ans=0.0 2023-11-24 06:08:03,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2711520.0, ans=0.125 2023-11-24 06:08:16,820 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 9950, loss[loss=0.07763, simple_loss=0.103, pruned_loss=0.01764, audio_tagging_loss=0.00848, over 14847.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09202, pruned_loss=0.01344, audio_tagging_loss=0.008768, over 3047057.09 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:08:25,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.30 vs. limit=10.0 2023-11-24 06:08:29,175 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406750 2023-11-24 06:08:34,883 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.047e+01 8.279e+01 9.180e+01 9.821e+01 1.550e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-24 06:09:01,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2711786.6666666665, ans=0.0 2023-11-24 06:09:11,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2711853.3333333335, ans=0.125 2023-11-24 06:09:13,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2711853.3333333335, ans=0.2 2023-11-24 06:09:18,683 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10000, loss[loss=0.05567, simple_loss=0.07029, pruned_loss=0.01234, audio_tagging_loss=0.008182, over 15253.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09095, pruned_loss=0.0132, audio_tagging_loss=0.008805, over 3049762.40 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:09:30,659 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406800 2023-11-24 06:09:48,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2712053.3333333335, ans=0.125 2023-11-24 06:10:04,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2712120.0, ans=0.1 2023-11-24 06:10:09,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2712186.6666666665, ans=0.0 2023-11-24 06:10:13,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2712186.6666666665, ans=0.1 2023-11-24 06:10:20,490 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10050, loss[loss=0.08641, simple_loss=0.1228, pruned_loss=0.02012, audio_tagging_loss=0.004891, over 15917.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09197, pruned_loss=0.01333, audio_tagging_loss=0.008755, over 3049727.31 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:10:27,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2712253.3333333335, ans=0.0 2023-11-24 06:10:33,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406850 2023-11-24 06:10:39,684 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.233e+01 8.347e+01 9.098e+01 9.653e+01 1.255e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-24 06:10:53,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.32 vs. limit=15.0 2023-11-24 06:10:55,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2712386.6666666665, ans=0.125 2023-11-24 06:11:17,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2712520.0, ans=0.0 2023-11-24 06:11:23,347 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10100, loss[loss=0.08862, simple_loss=0.1155, pruned_loss=0.02358, audio_tagging_loss=0.007271, over 15547.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09196, pruned_loss=0.01338, audio_tagging_loss=0.008774, over 3050343.62 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:11:35,122 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406900 2023-11-24 06:11:45,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2712653.3333333335, ans=0.0 2023-11-24 06:11:48,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2712720.0, ans=0.125 2023-11-24 06:11:55,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2712720.0, ans=0.0 2023-11-24 06:12:12,380 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 06:12:24,238 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10150, loss[loss=0.06085, simple_loss=0.08819, pruned_loss=0.00887, audio_tagging_loss=0.007887, over 16515.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09219, pruned_loss=0.01353, audio_tagging_loss=0.008823, over 3057346.44 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:12:31,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2712920.0, ans=0.125 2023-11-24 06:12:36,678 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 406950 2023-11-24 06:12:42,365 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.240e+01 8.502e+01 9.089e+01 9.719e+01 1.256e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-24 06:12:51,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2713053.3333333335, ans=0.125 2023-11-24 06:12:52,476 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 06:13:25,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.34 vs. limit=10.0 2023-11-24 06:13:26,448 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10200, loss[loss=0.07885, simple_loss=0.1097, pruned_loss=0.01698, audio_tagging_loss=0.007023, over 15536.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09223, pruned_loss=0.01351, audio_tagging_loss=0.008849, over 3056253.02 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:13:31,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2713253.3333333335, ans=0.125 2023-11-24 06:13:31,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2713253.3333333335, ans=0.0 2023-11-24 06:13:39,168 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407000 2023-11-24 06:13:42,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2713320.0, ans=0.125 2023-11-24 06:13:44,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2713320.0, ans=0.125 2023-11-24 06:13:51,003 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 06:13:53,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2713386.6666666665, ans=0.1 2023-11-24 06:14:29,222 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10250, loss[loss=0.08817, simple_loss=0.1266, pruned_loss=0.01742, audio_tagging_loss=0.007449, over 16778.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09193, pruned_loss=0.01349, audio_tagging_loss=0.008896, over 3055022.63 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:14:41,637 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407050 2023-11-24 06:14:43,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2713653.3333333335, ans=0.125 2023-11-24 06:14:45,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2713653.3333333335, ans=0.125 2023-11-24 06:14:47,355 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.429e+01 8.707e+01 9.386e+01 1.043e+02 1.340e+02, threshold=1.877e+02, percent-clipped=0.0 2023-11-24 06:15:01,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2713720.0, ans=0.125 2023-11-24 06:15:14,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2713786.6666666665, ans=0.125 2023-11-24 06:15:30,888 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10300, loss[loss=0.08084, simple_loss=0.117, pruned_loss=0.01471, audio_tagging_loss=0.007601, over 16758.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09333, pruned_loss=0.01387, audio_tagging_loss=0.00885, over 3056710.71 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:15:42,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407100 2023-11-24 06:15:55,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2714053.3333333335, ans=0.1 2023-11-24 06:16:07,706 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:16:14,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2714120.0, ans=0.1 2023-11-24 06:16:16,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2714120.0, ans=0.125 2023-11-24 06:16:31,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2714253.3333333335, ans=0.125 2023-11-24 06:16:32,185 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10350, loss[loss=0.08289, simple_loss=0.1184, pruned_loss=0.0158, audio_tagging_loss=0.007872, over 16519.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09316, pruned_loss=0.01373, audio_tagging_loss=0.008952, over 3059748.11 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:16:45,347 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407150 2023-11-24 06:16:52,216 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.280e+01 8.459e+01 9.063e+01 9.492e+01 1.258e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-24 06:16:58,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2714386.6666666665, ans=0.1 2023-11-24 06:16:58,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2714386.6666666665, ans=0.05 2023-11-24 06:17:07,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2714386.6666666665, ans=0.0 2023-11-24 06:17:17,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.00 vs. limit=10.0 2023-11-24 06:17:21,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2714520.0, ans=0.0 2023-11-24 06:17:25,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2714520.0, ans=0.125 2023-11-24 06:17:35,063 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10400, loss[loss=0.06736, simple_loss=0.08564, pruned_loss=0.01208, audio_tagging_loss=0.01246, over 15281.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09229, pruned_loss=0.01345, audio_tagging_loss=0.009152, over 3054618.70 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:17:47,887 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407200 2023-11-24 06:17:54,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2714653.3333333335, ans=0.1 2023-11-24 06:18:03,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2714720.0, ans=0.0 2023-11-24 06:18:10,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten.whitening_limit, batch_count=2714720.0, ans=15.0 2023-11-24 06:18:27,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2714853.3333333335, ans=0.0 2023-11-24 06:18:38,040 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10450, loss[loss=0.06703, simple_loss=0.0901, pruned_loss=0.01302, audio_tagging_loss=0.008957, over 15072.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09287, pruned_loss=0.01358, audio_tagging_loss=0.009128, over 3050244.32 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:18:49,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407250 2023-11-24 06:18:52,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2714986.6666666665, ans=0.125 2023-11-24 06:18:55,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.63 vs. limit=10.0 2023-11-24 06:18:55,553 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.473e+01 9.221e+01 1.007e+02 1.315e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 06:19:19,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2715120.0, ans=0.1 2023-11-24 06:19:38,982 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10500, loss[loss=0.06435, simple_loss=0.08115, pruned_loss=0.01214, audio_tagging_loss=0.01164, over 15061.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09339, pruned_loss=0.01367, audio_tagging_loss=0.008977, over 3055007.60 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:19:39,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.90 vs. limit=15.0 2023-11-24 06:19:40,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=6.97 vs. limit=15.0 2023-11-24 06:19:44,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2715253.3333333335, ans=0.0 2023-11-24 06:19:51,565 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407300 2023-11-24 06:19:55,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2715320.0, ans=0.0 2023-11-24 06:20:11,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.68 vs. limit=22.5 2023-11-24 06:20:19,440 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.55 vs. limit=15.0 2023-11-24 06:20:41,123 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10550, loss[loss=0.07297, simple_loss=0.09493, pruned_loss=0.01804, audio_tagging_loss=0.007468, over 15821.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09324, pruned_loss=0.01376, audio_tagging_loss=0.00884, over 3050414.44 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:20:41,849 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0 2023-11-24 06:20:43,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2715586.6666666665, ans=0.125 2023-11-24 06:20:54,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407350 2023-11-24 06:20:58,496 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.68 vs. limit=15.0 2023-11-24 06:21:01,302 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.501e+01 9.231e+01 9.962e+01 1.175e+02, threshold=1.846e+02, percent-clipped=0.0 2023-11-24 06:21:08,055 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=22.5 2023-11-24 06:21:21,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.99 vs. limit=12.0 2023-11-24 06:21:22,064 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.53 vs. limit=12.0 2023-11-24 06:21:41,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2715853.3333333335, ans=0.125 2023-11-24 06:21:41,343 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.36 vs. limit=10.0 2023-11-24 06:21:43,201 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10600, loss[loss=0.07908, simple_loss=0.1104, pruned_loss=0.01625, audio_tagging_loss=0.007628, over 15048.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.0931, pruned_loss=0.01364, audio_tagging_loss=0.008785, over 3054341.95 frames. ], batch size: 53, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:21:44,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2715920.0, ans=0.125 2023-11-24 06:21:49,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2715920.0, ans=0.125 2023-11-24 06:21:55,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407400 2023-11-24 06:21:58,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2715986.6666666665, ans=0.125 2023-11-24 06:22:04,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2715986.6666666665, ans=0.125 2023-11-24 06:22:33,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2716186.6666666665, ans=0.125 2023-11-24 06:22:45,106 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10650, loss[loss=0.07326, simple_loss=0.09932, pruned_loss=0.01462, audio_tagging_loss=0.008984, over 15168.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09281, pruned_loss=0.01356, audio_tagging_loss=0.00877, over 3055911.34 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:22:45,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=12.0 2023-11-24 06:22:56,893 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407450 2023-11-24 06:23:04,957 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.300e+01 8.906e+01 9.684e+01 1.178e+02, threshold=1.781e+02, percent-clipped=0.0 2023-11-24 06:23:08,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2716320.0, ans=0.0 2023-11-24 06:23:12,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2716386.6666666665, ans=0.0 2023-11-24 06:23:25,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2716453.3333333335, ans=0.0 2023-11-24 06:23:46,565 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10700, loss[loss=0.07341, simple_loss=0.0995, pruned_loss=0.01439, audio_tagging_loss=0.009277, over 15623.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09252, pruned_loss=0.01349, audio_tagging_loss=0.008765, over 3048593.76 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:24:00,625 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407500 2023-11-24 06:24:05,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2716653.3333333335, ans=0.05 2023-11-24 06:24:09,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2716653.3333333335, ans=0.5 2023-11-24 06:24:11,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2716720.0, ans=0.125 2023-11-24 06:24:17,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2716720.0, ans=0.0 2023-11-24 06:24:17,923 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0 2023-11-24 06:24:30,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2716786.6666666665, ans=0.125 2023-11-24 06:24:33,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2716786.6666666665, ans=0.04949747468305833 2023-11-24 06:24:36,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.08 vs. limit=22.5 2023-11-24 06:24:38,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2716853.3333333335, ans=0.1 2023-11-24 06:24:45,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2716853.3333333335, ans=0.0 2023-11-24 06:24:50,039 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10750, loss[loss=0.06167, simple_loss=0.08207, pruned_loss=0.01173, audio_tagging_loss=0.008907, over 14758.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09191, pruned_loss=0.01328, audio_tagging_loss=0.008816, over 3051953.67 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:24:57,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2716920.0, ans=0.125 2023-11-24 06:25:02,077 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407550 2023-11-24 06:25:04,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2716986.6666666665, ans=0.2 2023-11-24 06:25:09,203 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.785e+01 8.501e+01 8.981e+01 9.807e+01 1.307e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-24 06:25:15,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2717053.3333333335, ans=0.0 2023-11-24 06:25:21,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2717053.3333333335, ans=0.2 2023-11-24 06:25:25,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2717120.0, ans=0.0 2023-11-24 06:25:36,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2717120.0, ans=0.2 2023-11-24 06:25:39,574 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.17 vs. limit=15.0 2023-11-24 06:25:51,846 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10800, loss[loss=0.04793, simple_loss=0.05978, pruned_loss=0.007846, audio_tagging_loss=0.01019, over 15081.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09198, pruned_loss=0.01338, audio_tagging_loss=0.008848, over 3052233.88 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:25:57,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2717253.3333333335, ans=0.125 2023-11-24 06:26:04,033 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407600 2023-11-24 06:26:06,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2717320.0, ans=0.125 2023-11-24 06:26:13,788 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.95 vs. limit=15.0 2023-11-24 06:26:19,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2717386.6666666665, ans=0.09899494936611666 2023-11-24 06:26:46,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2717520.0, ans=0.125 2023-11-24 06:26:49,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2717520.0, ans=0.1 2023-11-24 06:26:54,167 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10850, loss[loss=0.07337, simple_loss=0.09855, pruned_loss=0.01366, audio_tagging_loss=0.01043, over 15315.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09158, pruned_loss=0.01331, audio_tagging_loss=0.008885, over 3048587.24 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:26:59,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.62 vs. limit=15.0 2023-11-24 06:27:07,785 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407650 2023-11-24 06:27:10,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.62 vs. limit=10.0 2023-11-24 06:27:12,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.04 vs. limit=12.0 2023-11-24 06:27:15,321 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.695e+01 9.228e+01 1.009e+02 1.373e+02, threshold=1.846e+02, percent-clipped=0.0 2023-11-24 06:27:24,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2717720.0, ans=0.125 2023-11-24 06:27:24,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2717720.0, ans=0.0 2023-11-24 06:27:38,730 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.30 vs. limit=10.0 2023-11-24 06:27:44,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2717853.3333333335, ans=0.125 2023-11-24 06:27:44,826 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.62 vs. limit=22.5 2023-11-24 06:27:54,165 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 06:27:57,723 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10900, loss[loss=0.09133, simple_loss=0.1212, pruned_loss=0.02269, audio_tagging_loss=0.008045, over 15244.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09132, pruned_loss=0.01337, audio_tagging_loss=0.008919, over 3046322.64 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:28:10,246 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407700 2023-11-24 06:28:25,970 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.13 vs. limit=22.5 2023-11-24 06:28:28,323 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.01 vs. limit=15.0 2023-11-24 06:28:29,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2718053.3333333335, ans=0.09899494936611666 2023-11-24 06:28:36,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2718120.0, ans=0.125 2023-11-24 06:28:42,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2718120.0, ans=0.125 2023-11-24 06:28:55,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2718186.6666666665, ans=0.125 2023-11-24 06:28:59,462 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 10950, loss[loss=0.05694, simple_loss=0.08226, pruned_loss=0.007352, audio_tagging_loss=0.008456, over 15997.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09194, pruned_loss=0.01355, audio_tagging_loss=0.009009, over 3054159.73 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:29:11,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407750 2023-11-24 06:29:13,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2718320.0, ans=0.0 2023-11-24 06:29:21,253 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.403e+01 8.864e+01 9.655e+01 1.476e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-24 06:29:44,702 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:29:47,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2718453.3333333335, ans=0.07 2023-11-24 06:29:49,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2718520.0, ans=0.1 2023-11-24 06:29:56,854 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.25 vs. limit=22.5 2023-11-24 06:30:00,916 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11000, loss[loss=0.07217, simple_loss=0.101, pruned_loss=0.01301, audio_tagging_loss=0.008658, over 16201.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09157, pruned_loss=0.01344, audio_tagging_loss=0.009028, over 3050751.39 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:30:11,001 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 06:30:14,005 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407800 2023-11-24 06:30:19,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.98 vs. limit=15.0 2023-11-24 06:30:19,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.29 vs. limit=22.5 2023-11-24 06:30:33,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2718720.0, ans=0.0 2023-11-24 06:30:40,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2718786.6666666665, ans=0.125 2023-11-24 06:30:40,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2718786.6666666665, ans=0.125 2023-11-24 06:30:52,906 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.89 vs. limit=10.0 2023-11-24 06:30:56,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2718853.3333333335, ans=0.125 2023-11-24 06:30:58,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2718853.3333333335, ans=0.1 2023-11-24 06:30:59,509 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.24 vs. limit=15.0 2023-11-24 06:31:04,754 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11050, loss[loss=0.07276, simple_loss=0.09661, pruned_loss=0.01412, audio_tagging_loss=0.01034, over 15334.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09167, pruned_loss=0.01348, audio_tagging_loss=0.009074, over 3049522.41 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:31:17,396 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407850 2023-11-24 06:31:26,801 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.514e+01 8.990e+01 9.675e+01 1.640e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-24 06:31:29,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2719053.3333333335, ans=0.1 2023-11-24 06:32:04,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2719186.6666666665, ans=0.07 2023-11-24 06:32:06,875 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11100, loss[loss=0.06908, simple_loss=0.1001, pruned_loss=0.01041, audio_tagging_loss=0.008633, over 16297.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09142, pruned_loss=0.01331, audio_tagging_loss=0.009206, over 3047463.35 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:32:10,717 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2719253.3333333335, ans=0.125 2023-11-24 06:32:11,034 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.50 vs. limit=15.0 2023-11-24 06:32:19,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407900 2023-11-24 06:32:22,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2719320.0, ans=0.125 2023-11-24 06:32:38,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2719386.6666666665, ans=0.1 2023-11-24 06:32:47,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2719453.3333333335, ans=0.0 2023-11-24 06:32:47,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2719453.3333333335, ans=0.2 2023-11-24 06:33:02,660 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:33:08,626 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11150, loss[loss=0.05932, simple_loss=0.07285, pruned_loss=0.01237, audio_tagging_loss=0.01053, over 14754.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09212, pruned_loss=0.01347, audio_tagging_loss=0.009246, over 3049251.07 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:33:11,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2719586.6666666665, ans=0.125 2023-11-24 06:33:17,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2719586.6666666665, ans=0.125 2023-11-24 06:33:21,124 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 407950 2023-11-24 06:33:31,069 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.64 vs. limit=15.0 2023-11-24 06:33:31,714 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.657e+01 9.225e+01 9.905e+01 1.226e+02, threshold=1.845e+02, percent-clipped=0.0 2023-11-24 06:33:34,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2719720.0, ans=0.125 2023-11-24 06:33:43,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2719720.0, ans=0.05 2023-11-24 06:33:51,410 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:33:59,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2719853.3333333335, ans=0.0 2023-11-24 06:34:01,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.43 vs. limit=15.0 2023-11-24 06:34:10,892 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11200, loss[loss=0.04551, simple_loss=0.05253, pruned_loss=0.007084, audio_tagging_loss=0.01216, over 14724.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09168, pruned_loss=0.01328, audio_tagging_loss=0.009304, over 3050008.85 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:34:22,574 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.55 vs. limit=15.0 2023-11-24 06:34:23,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408000 2023-11-24 06:34:51,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2720120.0, ans=0.025 2023-11-24 06:34:58,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2720120.0, ans=0.125 2023-11-24 06:34:58,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2720120.0, ans=0.2 2023-11-24 06:35:11,120 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:35:16,696 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11250, loss[loss=0.04646, simple_loss=0.06073, pruned_loss=0.006145, audio_tagging_loss=0.009948, over 14475.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09056, pruned_loss=0.01311, audio_tagging_loss=0.009421, over 3052671.05 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:35:29,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408050 2023-11-24 06:35:30,399 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:35:35,581 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0 2023-11-24 06:35:39,644 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.153e+01 8.539e+01 9.181e+01 9.755e+01 1.709e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-24 06:35:44,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2720386.6666666665, ans=0.125 2023-11-24 06:35:45,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2720386.6666666665, ans=0.125 2023-11-24 06:35:56,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.76 vs. limit=15.0 2023-11-24 06:35:56,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.whiten.whitening_limit, batch_count=2720453.3333333335, ans=15.0 2023-11-24 06:36:07,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2720520.0, ans=0.1 2023-11-24 06:36:07,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2720520.0, ans=0.0 2023-11-24 06:36:08,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.41 vs. limit=15.0 2023-11-24 06:36:18,128 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11300, loss[loss=0.07315, simple_loss=0.09233, pruned_loss=0.01666, audio_tagging_loss=0.01033, over 15007.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09144, pruned_loss=0.01335, audio_tagging_loss=0.009154, over 3049058.13 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:36:28,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.39 vs. limit=15.0 2023-11-24 06:36:30,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408100 2023-11-24 06:36:43,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2720720.0, ans=0.04949747468305833 2023-11-24 06:37:06,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2720853.3333333335, ans=0.0 2023-11-24 06:37:19,932 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11350, loss[loss=0.07093, simple_loss=0.09132, pruned_loss=0.01706, audio_tagging_loss=0.008206, over 14626.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09102, pruned_loss=0.01322, audio_tagging_loss=0.009018, over 3047351.62 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:37:33,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408150 2023-11-24 06:37:41,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2720986.6666666665, ans=0.0 2023-11-24 06:37:43,673 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.373e+01 9.046e+01 9.724e+01 1.113e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-24 06:37:46,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2721053.3333333335, ans=0.0 2023-11-24 06:37:54,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2721053.3333333335, ans=0.2 2023-11-24 06:38:05,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2721120.0, ans=0.05 2023-11-24 06:38:06,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.04 vs. limit=22.5 2023-11-24 06:38:12,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2721186.6666666665, ans=0.125 2023-11-24 06:38:13,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2721186.6666666665, ans=0.125 2023-11-24 06:38:22,791 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11400, loss[loss=0.07217, simple_loss=0.0983, pruned_loss=0.01283, audio_tagging_loss=0.01019, over 14747.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.0916, pruned_loss=0.01334, audio_tagging_loss=0.008946, over 3046185.28 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:38:31,751 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.73 vs. limit=22.5 2023-11-24 06:38:34,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408200 2023-11-24 06:38:43,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2721320.0, ans=0.125 2023-11-24 06:38:46,924 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:38:55,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2721386.6666666665, ans=0.125 2023-11-24 06:39:10,699 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.49 vs. limit=15.0 2023-11-24 06:39:17,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2721520.0, ans=0.125 2023-11-24 06:39:19,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2721520.0, ans=0.125 2023-11-24 06:39:20,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2721520.0, ans=0.0 2023-11-24 06:39:24,333 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11450, loss[loss=0.07759, simple_loss=0.1138, pruned_loss=0.01331, audio_tagging_loss=0.007396, over 15845.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09205, pruned_loss=0.01352, audio_tagging_loss=0.008856, over 3039889.33 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:39:36,977 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408250 2023-11-24 06:39:41,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2721653.3333333335, ans=0.025 2023-11-24 06:39:48,648 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.611e+01 9.148e+01 9.766e+01 1.313e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 06:40:03,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=2721786.6666666665, ans=0.02 2023-11-24 06:40:15,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.26 vs. limit=22.5 2023-11-24 06:40:25,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2721920.0, ans=0.5 2023-11-24 06:40:26,821 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11500, loss[loss=0.06482, simple_loss=0.08424, pruned_loss=0.01409, audio_tagging_loss=0.008604, over 15006.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09317, pruned_loss=0.01381, audio_tagging_loss=0.00879, over 3041461.26 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:40:30,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2721920.0, ans=0.1 2023-11-24 06:40:33,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.78 vs. limit=15.0 2023-11-24 06:40:39,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408300 2023-11-24 06:41:04,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2722120.0, ans=0.125 2023-11-24 06:41:15,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2722186.6666666665, ans=0.125 2023-11-24 06:41:16,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2722186.6666666665, ans=0.125 2023-11-24 06:41:16,870 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.79 vs. limit=15.0 2023-11-24 06:41:23,554 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.55 vs. limit=15.0 2023-11-24 06:41:28,747 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11550, loss[loss=0.09003, simple_loss=0.1185, pruned_loss=0.02175, audio_tagging_loss=0.009051, over 13779.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09244, pruned_loss=0.01368, audio_tagging_loss=0.008854, over 3038412.99 frames. ], batch size: 53, lr: 1.98e-03, grad_scale: 8.0 2023-11-24 06:41:36,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2722253.3333333335, ans=0.0 2023-11-24 06:41:40,681 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408350 2023-11-24 06:41:51,959 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.495e+01 8.414e+01 9.031e+01 9.585e+01 1.239e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 06:41:55,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2722386.6666666665, ans=0.5 2023-11-24 06:42:01,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2722386.6666666665, ans=0.0 2023-11-24 06:42:07,467 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 06:42:07,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2722453.3333333335, ans=0.0 2023-11-24 06:42:19,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2722520.0, ans=0.015 2023-11-24 06:42:20,850 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.84 vs. limit=15.0 2023-11-24 06:42:28,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.40 vs. limit=22.5 2023-11-24 06:42:29,716 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11600, loss[loss=0.08156, simple_loss=0.1196, pruned_loss=0.01674, audio_tagging_loss=0.005033, over 14634.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09337, pruned_loss=0.01389, audio_tagging_loss=0.008867, over 3035730.07 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:42:42,244 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408400 2023-11-24 06:42:56,423 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.06 vs. limit=12.0 2023-11-24 06:43:21,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0 2023-11-24 06:43:31,829 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11650, loss[loss=0.08963, simple_loss=0.1236, pruned_loss=0.02015, audio_tagging_loss=0.007672, over 15924.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09294, pruned_loss=0.01376, audio_tagging_loss=0.008841, over 3036579.66 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:43:33,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2722920.0, ans=0.125 2023-11-24 06:43:42,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2722920.0, ans=0.1 2023-11-24 06:43:45,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408450 2023-11-24 06:43:47,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2722986.6666666665, ans=0.1 2023-11-24 06:43:50,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2722986.6666666665, ans=0.125 2023-11-24 06:43:55,840 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.490e+01 8.921e+01 9.548e+01 1.142e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-24 06:43:58,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2723053.3333333335, ans=0.0 2023-11-24 06:44:00,012 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.83 vs. limit=15.0 2023-11-24 06:44:02,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2723053.3333333335, ans=0.125 2023-11-24 06:44:10,637 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0 2023-11-24 06:44:34,538 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11700, loss[loss=0.06503, simple_loss=0.09166, pruned_loss=0.009118, audio_tagging_loss=0.01008, over 15582.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09309, pruned_loss=0.01371, audio_tagging_loss=0.008822, over 3047062.03 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:44:46,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408500 2023-11-24 06:44:53,809 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:44:54,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2723320.0, ans=0.0 2023-11-24 06:45:00,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=2723386.6666666665, ans=0.02 2023-11-24 06:45:01,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2723386.6666666665, ans=0.0 2023-11-24 06:45:25,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2723520.0, ans=0.125 2023-11-24 06:45:33,247 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.06 vs. limit=22.5 2023-11-24 06:45:35,968 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11750, loss[loss=0.06484, simple_loss=0.09306, pruned_loss=0.008264, audio_tagging_loss=0.01005, over 14677.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09248, pruned_loss=0.01363, audio_tagging_loss=0.008934, over 3040359.89 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:45:47,377 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.09 vs. limit=15.0 2023-11-24 06:45:48,013 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408550 2023-11-24 06:45:49,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2723653.3333333335, ans=0.125 2023-11-24 06:45:54,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2723653.3333333335, ans=0.0 2023-11-24 06:45:59,492 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.522e+01 8.698e+01 9.400e+01 1.049e+02 1.274e+02, threshold=1.880e+02, percent-clipped=0.0 2023-11-24 06:46:12,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2723786.6666666665, ans=0.0 2023-11-24 06:46:21,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2723786.6666666665, ans=0.1 2023-11-24 06:46:28,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.99 vs. limit=22.5 2023-11-24 06:46:36,805 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11800, loss[loss=0.08762, simple_loss=0.113, pruned_loss=0.02151, audio_tagging_loss=0.009603, over 15340.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09256, pruned_loss=0.01359, audio_tagging_loss=0.008982, over 3054113.21 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:46:47,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2723920.0, ans=0.125 2023-11-24 06:46:51,029 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408600 2023-11-24 06:46:51,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.40 vs. limit=22.5 2023-11-24 06:46:54,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2723986.6666666665, ans=0.2 2023-11-24 06:46:56,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2723986.6666666665, ans=0.1 2023-11-24 06:46:57,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2723986.6666666665, ans=0.125 2023-11-24 06:47:32,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2724186.6666666665, ans=0.95 2023-11-24 06:47:40,779 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11850, loss[loss=0.07992, simple_loss=0.1107, pruned_loss=0.01809, audio_tagging_loss=0.006464, over 16273.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09214, pruned_loss=0.01345, audio_tagging_loss=0.009042, over 3046954.87 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:47:52,794 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408650 2023-11-24 06:48:03,387 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.550e+01 9.032e+01 9.774e+01 1.343e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 06:48:25,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2724453.3333333335, ans=0.0 2023-11-24 06:48:39,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2724520.0, ans=0.2 2023-11-24 06:48:42,010 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11900, loss[loss=0.07491, simple_loss=0.09474, pruned_loss=0.01647, audio_tagging_loss=0.01108, over 16310.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09304, pruned_loss=0.01366, audio_tagging_loss=0.009023, over 3049810.02 frames. ], batch size: 62, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:48:53,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2724653.3333333335, ans=0.125 2023-11-24 06:48:54,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408700 2023-11-24 06:49:03,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2724653.3333333335, ans=0.1 2023-11-24 06:49:08,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0 2023-11-24 06:49:08,994 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.17 vs. limit=15.0 2023-11-24 06:49:18,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2724786.6666666665, ans=0.1 2023-11-24 06:49:27,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2724786.6666666665, ans=0.1 2023-11-24 06:49:30,296 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:49:42,888 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 11950, loss[loss=0.07464, simple_loss=0.1056, pruned_loss=0.01381, audio_tagging_loss=0.008044, over 15578.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.0911, pruned_loss=0.01327, audio_tagging_loss=0.009202, over 3053159.48 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0 2023-11-24 06:49:56,359 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408750 2023-11-24 06:50:01,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.42 vs. limit=15.0 2023-11-24 06:50:07,408 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.035e+01 8.293e+01 8.993e+01 9.649e+01 1.275e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-24 06:50:14,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2725053.3333333335, ans=0.09899494936611666 2023-11-24 06:50:22,629 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:50:26,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2725120.0, ans=0.1 2023-11-24 06:50:27,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2725120.0, ans=0.0 2023-11-24 06:50:27,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2725120.0, ans=0.125 2023-11-24 06:50:31,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2725186.6666666665, ans=0.125 2023-11-24 06:50:36,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2725186.6666666665, ans=0.125 2023-11-24 06:50:38,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2725186.6666666665, ans=0.95 2023-11-24 06:50:38,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2725186.6666666665, ans=0.125 2023-11-24 06:50:42,954 INFO [train_asr.py:1221] (3/4) Epoch 34, batch 12000, loss[loss=0.05494, simple_loss=0.07778, pruned_loss=0.006742, audio_tagging_loss=0.009308, over 15434.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09164, pruned_loss=0.01335, audio_tagging_loss=0.009216, over 3057077.83 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 32.0 2023-11-24 06:50:42,954 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 06:51:11,179 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9660, 3.2148, 2.9785, 3.1921, 3.4126, 2.7329, 3.3882, 2.5604], device='cuda:3') 2023-11-24 06:51:25,769 INFO [train_asr.py:1253] (3/4) Epoch 34, validation: loss=0.05837, simple_loss=0.05087, pruned_loss=0.005158, audio_tagging_loss=0.02778, over 4681554.00 frames. 2023-11-24 06:51:25,770 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 06:51:26,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2725253.3333333335, ans=0.0 2023-11-24 06:51:29,329 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:51:29,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.29 vs. limit=12.0 2023-11-24 06:51:36,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408800 2023-11-24 06:52:25,020 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 0, loss[loss=0.09287, simple_loss=0.1141, pruned_loss=0.02096, audio_tagging_loss=0.01487, over 16097.00 frames. ], tot_loss[loss=0.09287, simple_loss=0.1141, pruned_loss=0.02096, audio_tagging_loss=0.01487, over 16097.00 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 06:52:25,020 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 06:53:00,541 INFO [train_asr.py:1253] (3/4) Epoch 35, validation: loss=0.05805, simple_loss=0.05089, pruned_loss=0.005144, audio_tagging_loss=0.02746, over 4681554.00 frames. 2023-11-24 06:53:00,542 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 06:53:15,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2725473.3333333335, ans=0.125 2023-11-24 06:53:47,139 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408850 2023-11-24 06:53:47,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2725606.6666666665, ans=0.125 2023-11-24 06:53:57,723 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.642e+01 8.972e+01 1.010e+02 1.109e+02 1.515e+02, threshold=2.019e+02, percent-clipped=0.0 2023-11-24 06:54:03,118 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 50, loss[loss=0.08271, simple_loss=0.1104, pruned_loss=0.01251, audio_tagging_loss=0.01502, over 15754.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.0905, pruned_loss=0.01345, audio_tagging_loss=0.01782, over 686038.11 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 06:54:11,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2725740.0, ans=0.125 2023-11-24 06:54:15,638 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.38 vs. limit=15.0 2023-11-24 06:54:50,019 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408900 2023-11-24 06:55:03,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2726006.6666666665, ans=0.0 2023-11-24 06:55:04,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2726006.6666666665, ans=0.1 2023-11-24 06:55:06,616 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 100, loss[loss=0.07128, simple_loss=0.08868, pruned_loss=0.01021, audio_tagging_loss=0.01673, over 15090.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.08973, pruned_loss=0.01261, audio_tagging_loss=0.01681, over 1211018.24 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 06:55:19,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.85 vs. limit=15.0 2023-11-24 06:55:53,049 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 408950 2023-11-24 06:56:02,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=14.12 vs. limit=15.0 2023-11-24 06:56:04,090 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.733e+01 9.169e+01 9.656e+01 1.031e+02 1.256e+02, threshold=1.931e+02, percent-clipped=0.0 2023-11-24 06:56:08,796 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 150, loss[loss=0.09163, simple_loss=0.1358, pruned_loss=0.01741, audio_tagging_loss=0.006333, over 14917.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.09362, pruned_loss=0.01327, audio_tagging_loss=0.01484, over 1619455.22 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 06:56:11,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2726406.6666666665, ans=0.125 2023-11-24 06:56:12,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.86 vs. limit=22.5 2023-11-24 06:56:23,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2726473.3333333335, ans=0.125 2023-11-24 06:56:26,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2726473.3333333335, ans=0.125 2023-11-24 06:56:35,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2726540.0, ans=0.0 2023-11-24 06:56:36,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2726540.0, ans=0.125 2023-11-24 06:56:44,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0 2023-11-24 06:56:46,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.62 vs. limit=22.5 2023-11-24 06:56:54,758 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409000 2023-11-24 06:56:58,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2726673.3333333335, ans=0.1 2023-11-24 06:56:59,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.54 vs. limit=22.5 2023-11-24 06:57:04,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2726673.3333333335, ans=0.125 2023-11-24 06:57:05,978 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:57:11,059 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 200, loss[loss=0.09235, simple_loss=0.1217, pruned_loss=0.02305, audio_tagging_loss=0.008441, over 14698.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09334, pruned_loss=0.01336, audio_tagging_loss=0.01306, over 1936160.82 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 06:57:13,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2726740.0, ans=0.125 2023-11-24 06:57:19,026 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 06:57:24,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2726806.6666666665, ans=0.125 2023-11-24 06:57:34,926 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.75 vs. limit=15.0 2023-11-24 06:57:51,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2726940.0, ans=0.1 2023-11-24 06:57:52,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2726940.0, ans=0.2 2023-11-24 06:57:52,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2726940.0, ans=0.125 2023-11-24 06:57:56,220 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409050 2023-11-24 06:58:09,528 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.892e+01 8.427e+01 9.100e+01 9.790e+01 1.266e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-24 06:58:13,114 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 250, loss[loss=0.06104, simple_loss=0.07074, pruned_loss=0.01422, audio_tagging_loss=0.01145, over 16066.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.0939, pruned_loss=0.01348, audio_tagging_loss=0.01168, over 2191323.16 frames. ], batch size: 61, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 06:58:25,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2727140.0, ans=0.0 2023-11-24 06:58:46,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2727206.6666666665, ans=0.0 2023-11-24 06:58:59,003 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409100 2023-11-24 06:59:14,735 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 300, loss[loss=0.07689, simple_loss=0.1022, pruned_loss=0.01484, audio_tagging_loss=0.01096, over 14608.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09503, pruned_loss=0.0137, audio_tagging_loss=0.01081, over 2386141.07 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 06:59:38,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.00 vs. limit=22.5 2023-11-24 06:59:42,850 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.46 vs. limit=22.5 2023-11-24 07:00:00,038 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409150 2023-11-24 07:00:03,772 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.60 vs. limit=15.0 2023-11-24 07:00:07,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2727673.3333333335, ans=0.0 2023-11-24 07:00:13,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.99 vs. limit=12.0 2023-11-24 07:00:13,704 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.065e+01 8.645e+01 9.268e+01 9.915e+01 1.259e+02, threshold=1.854e+02, percent-clipped=0.0 2023-11-24 07:00:16,047 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 350, loss[loss=0.06032, simple_loss=0.08453, pruned_loss=0.01026, audio_tagging_loss=0.007793, over 14439.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09408, pruned_loss=0.01352, audio_tagging_loss=0.01017, over 2530847.97 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 8.0 2023-11-24 07:00:20,765 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0 2023-11-24 07:00:39,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2727806.6666666665, ans=0.125 2023-11-24 07:00:40,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2727873.3333333335, ans=0.0 2023-11-24 07:01:02,304 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409200 2023-11-24 07:01:03,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2727940.0, ans=0.125 2023-11-24 07:01:19,547 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 400, loss[loss=0.06463, simple_loss=0.08542, pruned_loss=0.01465, audio_tagging_loss=0.007268, over 14300.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09476, pruned_loss=0.01366, audio_tagging_loss=0.009734, over 2642061.24 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:01:22,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2728073.3333333335, ans=0.125 2023-11-24 07:01:26,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2728073.3333333335, ans=0.1 2023-11-24 07:02:04,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2728273.3333333335, ans=0.04949747468305833 2023-11-24 07:02:05,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409250 2023-11-24 07:02:16,305 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.24 vs. limit=15.0 2023-11-24 07:02:18,962 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.492e+01 8.964e+01 9.627e+01 1.454e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 07:02:21,382 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 450, loss[loss=0.0602, simple_loss=0.08038, pruned_loss=0.007392, audio_tagging_loss=0.01261, over 14627.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09368, pruned_loss=0.01344, audio_tagging_loss=0.009464, over 2734623.01 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:02:38,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2728473.3333333335, ans=0.5 2023-11-24 07:03:03,682 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2728606.6666666665, ans=0.0 2023-11-24 07:03:08,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409300 2023-11-24 07:03:11,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2728673.3333333335, ans=15.0 2023-11-24 07:03:24,183 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 500, loss[loss=0.05628, simple_loss=0.07063, pruned_loss=0.01213, audio_tagging_loss=0.008841, over 15517.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09328, pruned_loss=0.01346, audio_tagging_loss=0.009276, over 2806616.46 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:03:29,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2728740.0, ans=0.125 2023-11-24 07:03:44,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2728806.6666666665, ans=0.125 2023-11-24 07:03:48,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2728873.3333333335, ans=0.2 2023-11-24 07:03:55,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2728873.3333333335, ans=0.125 2023-11-24 07:04:01,726 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.43 vs. limit=15.0 2023-11-24 07:04:07,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2728940.0, ans=0.0 2023-11-24 07:04:10,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409350 2023-11-24 07:04:12,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2729006.6666666665, ans=0.2 2023-11-24 07:04:16,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2729006.6666666665, ans=0.125 2023-11-24 07:04:24,160 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.423e+01 9.068e+01 9.980e+01 1.380e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 07:04:27,229 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 550, loss[loss=0.05898, simple_loss=0.08217, pruned_loss=0.009954, audio_tagging_loss=0.007942, over 15625.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09243, pruned_loss=0.01336, audio_tagging_loss=0.009211, over 2857641.86 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:04:30,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2729073.3333333335, ans=0.125 2023-11-24 07:04:41,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2729140.0, ans=0.125 2023-11-24 07:04:43,033 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.31 vs. limit=15.0 2023-11-24 07:04:59,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2729206.6666666665, ans=0.125 2023-11-24 07:05:00,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2729206.6666666665, ans=0.1 2023-11-24 07:05:02,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2729273.3333333335, ans=0.0 2023-11-24 07:05:12,316 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409400 2023-11-24 07:05:23,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0 2023-11-24 07:05:28,487 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 600, loss[loss=0.06659, simple_loss=0.08639, pruned_loss=0.01412, audio_tagging_loss=0.009268, over 15008.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09186, pruned_loss=0.01317, audio_tagging_loss=0.0092, over 2896719.22 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:05:44,604 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.67 vs. limit=15.0 2023-11-24 07:05:53,002 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.13 vs. limit=12.0 2023-11-24 07:05:54,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0 2023-11-24 07:06:07,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2729606.6666666665, ans=0.1 2023-11-24 07:06:14,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409450 2023-11-24 07:06:16,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2729606.6666666665, ans=15.0 2023-11-24 07:06:23,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2729673.3333333335, ans=0.1 2023-11-24 07:06:27,508 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.229e+01 8.791e+01 9.631e+01 1.307e+02, threshold=1.758e+02, percent-clipped=0.0 2023-11-24 07:06:29,981 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 650, loss[loss=0.05976, simple_loss=0.0756, pruned_loss=0.008869, audio_tagging_loss=0.0131, over 15749.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09294, pruned_loss=0.01332, audio_tagging_loss=0.009097, over 2934521.12 frames. ], batch size: 61, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:06:38,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2729740.0, ans=0.0 2023-11-24 07:06:40,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2729740.0, ans=10.0 2023-11-24 07:06:59,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2729873.3333333335, ans=0.125 2023-11-24 07:07:12,237 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2729940.0, ans=0.1 2023-11-24 07:07:13,681 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2729940.0, ans=0.0 2023-11-24 07:07:14,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2729940.0, ans=0.2 2023-11-24 07:07:15,805 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409500 2023-11-24 07:07:32,858 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 700, loss[loss=0.08485, simple_loss=0.109, pruned_loss=0.02122, audio_tagging_loss=0.009123, over 14604.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.0916, pruned_loss=0.013, audio_tagging_loss=0.009162, over 2956762.23 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:07:44,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2730140.0, ans=0.025 2023-11-24 07:08:13,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2730273.3333333335, ans=0.125 2023-11-24 07:08:15,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2730273.3333333335, ans=0.1 2023-11-24 07:08:15,302 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=12.0 2023-11-24 07:08:19,032 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409550 2023-11-24 07:08:32,408 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.410e+01 9.143e+01 1.010e+02 1.194e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 07:08:32,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2730340.0, ans=0.1 2023-11-24 07:08:34,812 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 750, loss[loss=0.07069, simple_loss=0.09516, pruned_loss=0.01404, audio_tagging_loss=0.009068, over 15247.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09204, pruned_loss=0.01319, audio_tagging_loss=0.009174, over 2978423.50 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:08:35,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2730406.6666666665, ans=0.125 2023-11-24 07:08:42,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2730406.6666666665, ans=0.0 2023-11-24 07:09:00,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2730540.0, ans=0.0 2023-11-24 07:09:20,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409600 2023-11-24 07:09:30,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2730673.3333333335, ans=0.0 2023-11-24 07:09:36,491 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 800, loss[loss=0.06818, simple_loss=0.09713, pruned_loss=0.01128, audio_tagging_loss=0.008332, over 15155.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09178, pruned_loss=0.01314, audio_tagging_loss=0.009206, over 2996069.30 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 07:10:16,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.80 vs. limit=15.0 2023-11-24 07:10:22,551 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409650 2023-11-24 07:10:35,976 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.567e+01 9.382e+01 1.008e+02 1.693e+02, threshold=1.876e+02, percent-clipped=0.0 2023-11-24 07:10:38,334 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 850, loss[loss=0.05996, simple_loss=0.08588, pruned_loss=0.00842, audio_tagging_loss=0.008594, over 15119.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09125, pruned_loss=0.01302, audio_tagging_loss=0.009286, over 3007765.58 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 07:10:40,662 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.61 vs. limit=10.0 2023-11-24 07:10:50,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2731140.0, ans=0.0 2023-11-24 07:10:55,628 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.05 vs. limit=6.0 2023-11-24 07:11:24,104 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409700 2023-11-24 07:11:33,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2731340.0, ans=0.0 2023-11-24 07:11:40,599 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 900, loss[loss=0.06969, simple_loss=0.0984, pruned_loss=0.01393, audio_tagging_loss=0.006561, over 15318.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09108, pruned_loss=0.01325, audio_tagging_loss=0.009322, over 3012854.13 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 07:11:46,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2731406.6666666665, ans=0.0 2023-11-24 07:11:56,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2731473.3333333335, ans=0.125 2023-11-24 07:12:03,327 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:12:26,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409750 2023-11-24 07:12:28,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2731606.6666666665, ans=0.2 2023-11-24 07:12:31,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2731673.3333333335, ans=0.1 2023-11-24 07:12:39,637 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.786e+01 9.264e+01 9.713e+01 1.175e+02, threshold=1.853e+02, percent-clipped=0.0 2023-11-24 07:12:42,028 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 950, loss[loss=0.05595, simple_loss=0.07682, pruned_loss=0.008152, audio_tagging_loss=0.009384, over 15069.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09095, pruned_loss=0.01313, audio_tagging_loss=0.009249, over 3024126.50 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 07:12:45,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2731740.0, ans=0.035 2023-11-24 07:12:47,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2731740.0, ans=0.0 2023-11-24 07:13:09,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2731873.3333333335, ans=0.125 2023-11-24 07:13:22,914 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.62 vs. limit=15.0 2023-11-24 07:13:28,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409800 2023-11-24 07:13:36,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2732006.6666666665, ans=0.0 2023-11-24 07:13:36,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.whiten.whitening_limit, batch_count=2732006.6666666665, ans=12.0 2023-11-24 07:13:44,716 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1000, loss[loss=0.07134, simple_loss=0.1066, pruned_loss=0.01369, audio_tagging_loss=0.004381, over 15074.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09142, pruned_loss=0.01331, audio_tagging_loss=0.009, over 3026147.66 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:13:53,700 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.47 vs. limit=10.0 2023-11-24 07:14:02,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2732140.0, ans=0.0 2023-11-24 07:14:11,519 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 07:14:26,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.55 vs. limit=22.5 2023-11-24 07:14:27,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2732273.3333333335, ans=0.95 2023-11-24 07:14:31,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409850 2023-11-24 07:14:40,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2732340.0, ans=0.125 2023-11-24 07:14:46,887 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.432e+01 9.037e+01 9.609e+01 1.581e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 07:14:48,081 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1050, loss[loss=0.08684, simple_loss=0.1202, pruned_loss=0.02007, audio_tagging_loss=0.006651, over 14671.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09086, pruned_loss=0.01331, audio_tagging_loss=0.008913, over 3031359.30 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:15:15,715 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.24 vs. limit=10.0 2023-11-24 07:15:16,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2732540.0, ans=0.5 2023-11-24 07:15:17,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2732540.0, ans=0.0 2023-11-24 07:15:30,224 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:15:34,110 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409900 2023-11-24 07:15:36,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2732673.3333333335, ans=0.125 2023-11-24 07:15:49,675 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1100, loss[loss=0.0747, simple_loss=0.1006, pruned_loss=0.01588, audio_tagging_loss=0.008501, over 15210.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09089, pruned_loss=0.01332, audio_tagging_loss=0.008808, over 3035389.39 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:15:52,059 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 07:16:05,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2732806.6666666665, ans=0.2 2023-11-24 07:16:06,064 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.57 vs. limit=10.0 2023-11-24 07:16:06,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2732806.6666666665, ans=0.0 2023-11-24 07:16:13,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2732873.3333333335, ans=0.125 2023-11-24 07:16:25,619 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.60 vs. limit=12.0 2023-11-24 07:16:26,867 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=15.0 2023-11-24 07:16:31,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2732940.0, ans=0.0 2023-11-24 07:16:35,780 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 409950 2023-11-24 07:16:40,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2733006.6666666665, ans=10.0 2023-11-24 07:16:49,758 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.554e+01 8.985e+01 9.572e+01 1.203e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-24 07:16:50,946 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1150, loss[loss=0.08406, simple_loss=0.1103, pruned_loss=0.01776, audio_tagging_loss=0.01115, over 16326.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09099, pruned_loss=0.01327, audio_tagging_loss=0.008779, over 3037993.77 frames. ], batch size: 62, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:17:05,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2733140.0, ans=0.125 2023-11-24 07:17:17,104 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:17:21,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2733206.6666666665, ans=0.125 2023-11-24 07:17:24,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2733206.6666666665, ans=0.0 2023-11-24 07:17:37,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410000 2023-11-24 07:17:53,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2733406.6666666665, ans=0.04949747468305833 2023-11-24 07:17:54,384 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1200, loss[loss=0.05518, simple_loss=0.07375, pruned_loss=0.009973, audio_tagging_loss=0.008336, over 14489.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.091, pruned_loss=0.0132, audio_tagging_loss=0.008765, over 3045142.56 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 07:18:13,532 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.43 vs. limit=15.0 2023-11-24 07:18:17,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.82 vs. limit=22.5 2023-11-24 07:18:30,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2733606.6666666665, ans=0.125 2023-11-24 07:18:40,695 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410050 2023-11-24 07:18:46,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2733673.3333333335, ans=0.125 2023-11-24 07:18:53,835 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.56 vs. limit=22.5 2023-11-24 07:18:55,462 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.712e+01 8.610e+01 9.345e+01 9.890e+01 1.292e+02, threshold=1.869e+02, percent-clipped=0.0 2023-11-24 07:18:56,700 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1250, loss[loss=0.05415, simple_loss=0.07622, pruned_loss=0.008655, audio_tagging_loss=0.007388, over 15781.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09025, pruned_loss=0.01307, audio_tagging_loss=0.008732, over 3044444.70 frames. ], batch size: 61, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 07:18:59,384 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:19:23,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2733873.3333333335, ans=0.125 2023-11-24 07:19:30,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2733873.3333333335, ans=0.0 2023-11-24 07:19:43,159 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410100 2023-11-24 07:19:43,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2733940.0, ans=0.1 2023-11-24 07:19:49,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2734006.6666666665, ans=0.125 2023-11-24 07:19:49,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2734006.6666666665, ans=0.2 2023-11-24 07:19:58,438 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1300, loss[loss=0.06577, simple_loss=0.08528, pruned_loss=0.01153, audio_tagging_loss=0.0116, over 13660.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.08992, pruned_loss=0.01295, audio_tagging_loss=0.008807, over 3040008.49 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:20:14,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2734140.0, ans=0.125 2023-11-24 07:20:32,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2734206.6666666665, ans=0.1 2023-11-24 07:20:45,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410150 2023-11-24 07:20:57,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2734340.0, ans=0.07 2023-11-24 07:21:01,937 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.342e+01 8.312e+01 8.919e+01 9.626e+01 1.151e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-24 07:21:01,981 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1350, loss[loss=0.08147, simple_loss=0.1121, pruned_loss=0.01781, audio_tagging_loss=0.007639, over 15373.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09046, pruned_loss=0.01301, audio_tagging_loss=0.008876, over 3044454.44 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:21:25,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2734540.0, ans=0.05 2023-11-24 07:21:35,476 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.53 vs. limit=15.0 2023-11-24 07:21:47,028 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 07:21:48,344 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410200 2023-11-24 07:21:54,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2734673.3333333335, ans=0.125 2023-11-24 07:22:04,505 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1400, loss[loss=0.05179, simple_loss=0.0669, pruned_loss=0.0076, audio_tagging_loss=0.01074, over 14841.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.08989, pruned_loss=0.01293, audio_tagging_loss=0.008992, over 3044089.36 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:22:09,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2734740.0, ans=0.125 2023-11-24 07:22:11,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2734740.0, ans=0.0 2023-11-24 07:22:47,437 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.66 vs. limit=15.0 2023-11-24 07:22:50,213 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410250 2023-11-24 07:23:06,022 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.310e+01 8.227e+01 8.989e+01 9.587e+01 1.141e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-24 07:23:06,067 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1450, loss[loss=0.05142, simple_loss=0.06654, pruned_loss=0.01129, audio_tagging_loss=0.006852, over 14113.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.08989, pruned_loss=0.01308, audio_tagging_loss=0.009052, over 3044293.41 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:23:13,968 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:23:47,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2735273.3333333335, ans=0.125 2023-11-24 07:23:52,469 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410300 2023-11-24 07:24:09,059 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1500, loss[loss=0.04546, simple_loss=0.05771, pruned_loss=0.006478, audio_tagging_loss=0.01012, over 13706.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09094, pruned_loss=0.01333, audio_tagging_loss=0.009013, over 3045134.63 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:24:24,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2735473.3333333335, ans=0.1 2023-11-24 07:24:28,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2735473.3333333335, ans=0.1 2023-11-24 07:24:34,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2735540.0, ans=0.2 2023-11-24 07:24:55,006 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410350 2023-11-24 07:24:57,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2735673.3333333335, ans=0.125 2023-11-24 07:25:10,422 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.532e+01 9.153e+01 9.693e+01 1.540e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-24 07:25:10,467 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1550, loss[loss=0.05987, simple_loss=0.07872, pruned_loss=0.01275, audio_tagging_loss=0.007762, over 14390.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09108, pruned_loss=0.01337, audio_tagging_loss=0.009123, over 3047303.24 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:25:11,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2735740.0, ans=0.1 2023-11-24 07:25:28,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2735806.6666666665, ans=0.1 2023-11-24 07:25:40,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2735873.3333333335, ans=0.0 2023-11-24 07:25:43,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2735873.3333333335, ans=0.1 2023-11-24 07:25:45,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2735873.3333333335, ans=0.0 2023-11-24 07:25:57,320 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410400 2023-11-24 07:25:58,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2735940.0, ans=0.0 2023-11-24 07:26:09,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2736006.6666666665, ans=0.125 2023-11-24 07:26:12,716 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.08 vs. limit=22.5 2023-11-24 07:26:13,370 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1600, loss[loss=0.06279, simple_loss=0.08442, pruned_loss=0.01181, audio_tagging_loss=0.008776, over 15734.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09123, pruned_loss=0.01356, audio_tagging_loss=0.009101, over 3051355.66 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 32.0 2023-11-24 07:26:21,855 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.42 vs. limit=15.0 2023-11-24 07:26:50,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2736273.3333333335, ans=0.125 2023-11-24 07:26:59,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410450 2023-11-24 07:27:06,801 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2736340.0, ans=0.0 2023-11-24 07:27:14,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2736406.6666666665, ans=0.0 2023-11-24 07:27:15,439 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1650, loss[loss=0.0714, simple_loss=0.09928, pruned_loss=0.01396, audio_tagging_loss=0.007801, over 16447.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09094, pruned_loss=0.01336, audio_tagging_loss=0.009161, over 3058200.82 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:27:17,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.572e+01 9.096e+01 9.732e+01 1.252e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 07:27:24,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2736406.6666666665, ans=0.035 2023-11-24 07:27:30,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2736473.3333333335, ans=0.125 2023-11-24 07:27:38,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.60 vs. limit=15.0 2023-11-24 07:27:41,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.17 vs. limit=15.0 2023-11-24 07:27:59,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2736606.6666666665, ans=0.125 2023-11-24 07:28:01,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410500 2023-11-24 07:28:02,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2736606.6666666665, ans=0.0 2023-11-24 07:28:10,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2736673.3333333335, ans=0.2 2023-11-24 07:28:16,963 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1700, loss[loss=0.05096, simple_loss=0.05701, pruned_loss=0.00991, audio_tagging_loss=0.01254, over 14745.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09127, pruned_loss=0.0133, audio_tagging_loss=0.009244, over 3058983.79 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:28:33,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.29 vs. limit=15.0 2023-11-24 07:28:44,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2736873.3333333335, ans=0.125 2023-11-24 07:29:03,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410550 2023-11-24 07:29:18,498 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1750, loss[loss=0.07016, simple_loss=0.1019, pruned_loss=0.01507, audio_tagging_loss=0.004129, over 14626.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.0916, pruned_loss=0.01322, audio_tagging_loss=0.009164, over 3057389.91 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:29:19,631 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.653e+01 8.559e+01 9.195e+01 9.925e+01 1.188e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 07:29:24,589 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.01 vs. limit=15.0 2023-11-24 07:29:32,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2737140.0, ans=0.125 2023-11-24 07:29:44,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2737206.6666666665, ans=0.0 2023-11-24 07:29:45,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2737206.6666666665, ans=0.025 2023-11-24 07:29:51,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2737206.6666666665, ans=0.0 2023-11-24 07:29:55,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.14 vs. limit=15.0 2023-11-24 07:30:01,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2737273.3333333335, ans=0.2 2023-11-24 07:30:04,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410600 2023-11-24 07:30:05,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2737273.3333333335, ans=0.125 2023-11-24 07:30:21,245 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1800, loss[loss=0.05172, simple_loss=0.06475, pruned_loss=0.007993, audio_tagging_loss=0.01135, over 15564.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09112, pruned_loss=0.01301, audio_tagging_loss=0.009105, over 3056806.37 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 8.0 2023-11-24 07:30:32,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2737473.3333333335, ans=0.1 2023-11-24 07:30:54,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2737540.0, ans=0.125 2023-11-24 07:31:05,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2737606.6666666665, ans=0.125 2023-11-24 07:31:06,447 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410650 2023-11-24 07:31:12,397 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.69 vs. limit=12.0 2023-11-24 07:31:18,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2737673.3333333335, ans=0.035 2023-11-24 07:31:23,080 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1850, loss[loss=0.06073, simple_loss=0.07865, pruned_loss=0.01416, audio_tagging_loss=0.007241, over 14955.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09162, pruned_loss=0.0131, audio_tagging_loss=0.008988, over 3053957.86 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 8.0 2023-11-24 07:31:25,381 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.283e+01 8.932e+01 9.471e+01 1.128e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 07:31:26,008 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.86 vs. limit=15.0 2023-11-24 07:31:26,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2737740.0, ans=0.05 2023-11-24 07:31:31,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2737740.0, ans=0.125 2023-11-24 07:31:57,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2737873.3333333335, ans=0.0 2023-11-24 07:32:08,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2737940.0, ans=0.0 2023-11-24 07:32:09,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410700 2023-11-24 07:32:22,642 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.64 vs. limit=22.5 2023-11-24 07:32:24,339 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1900, loss[loss=0.06776, simple_loss=0.07918, pruned_loss=0.01731, audio_tagging_loss=0.01087, over 14443.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09079, pruned_loss=0.01302, audio_tagging_loss=0.009033, over 3047748.48 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 8.0 2023-11-24 07:32:47,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2738140.0, ans=0.125 2023-11-24 07:32:49,708 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.62 vs. limit=15.0 2023-11-24 07:32:59,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2738206.6666666665, ans=0.1 2023-11-24 07:33:02,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.30 vs. limit=12.0 2023-11-24 07:33:10,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410750 2023-11-24 07:33:10,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2738273.3333333335, ans=0.125 2023-11-24 07:33:10,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2738273.3333333335, ans=0.05 2023-11-24 07:33:13,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2738340.0, ans=0.1 2023-11-24 07:33:25,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2738406.6666666665, ans=0.0 2023-11-24 07:33:26,588 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 1950, loss[loss=0.07135, simple_loss=0.09415, pruned_loss=0.01613, audio_tagging_loss=0.008141, over 15301.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09107, pruned_loss=0.01312, audio_tagging_loss=0.008949, over 3055075.06 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 8.0 2023-11-24 07:33:28,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2738406.6666666665, ans=0.125 2023-11-24 07:33:28,936 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.505e+01 9.149e+01 9.609e+01 1.191e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 07:33:36,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2738406.6666666665, ans=0.2 2023-11-24 07:34:06,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2738606.6666666665, ans=0.125 2023-11-24 07:34:11,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2738606.6666666665, ans=0.125 2023-11-24 07:34:11,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2738606.6666666665, ans=0.2 2023-11-24 07:34:12,068 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410800 2023-11-24 07:34:24,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2738673.3333333335, ans=0.125 2023-11-24 07:34:28,849 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2000, loss[loss=0.07407, simple_loss=0.09477, pruned_loss=0.01407, audio_tagging_loss=0.01262, over 14658.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09143, pruned_loss=0.01339, audio_tagging_loss=0.008948, over 3049238.40 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:34:34,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2738740.0, ans=0.0 2023-11-24 07:34:36,194 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2738740.0, ans=0.125 2023-11-24 07:34:38,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2738740.0, ans=0.125 2023-11-24 07:35:10,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2738940.0, ans=0.1 2023-11-24 07:35:13,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2738940.0, ans=0.125 2023-11-24 07:35:15,229 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410850 2023-11-24 07:35:25,146 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.73 vs. limit=10.0 2023-11-24 07:35:25,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0 2023-11-24 07:35:30,568 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2050, loss[loss=0.06666, simple_loss=0.09816, pruned_loss=0.009959, audio_tagging_loss=0.007623, over 15432.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09129, pruned_loss=0.01336, audio_tagging_loss=0.008933, over 3050800.18 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:35:32,816 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.370e+01 9.057e+01 9.754e+01 1.235e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-24 07:35:36,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2739073.3333333335, ans=0.125 2023-11-24 07:36:03,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.38 vs. limit=15.0 2023-11-24 07:36:11,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2739273.3333333335, ans=0.125 2023-11-24 07:36:11,509 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.07 vs. limit=15.0 2023-11-24 07:36:14,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.24 vs. limit=15.0 2023-11-24 07:36:16,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410900 2023-11-24 07:36:16,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2739273.3333333335, ans=0.125 2023-11-24 07:36:31,817 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2100, loss[loss=0.0719, simple_loss=0.09828, pruned_loss=0.01349, audio_tagging_loss=0.009276, over 16047.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09166, pruned_loss=0.01333, audio_tagging_loss=0.008987, over 3052464.96 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:36:49,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2739473.3333333335, ans=0.125 2023-11-24 07:37:03,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2739540.0, ans=0.125 2023-11-24 07:37:03,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2739540.0, ans=0.1 2023-11-24 07:37:07,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2739540.0, ans=0.1 2023-11-24 07:37:18,324 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 410950 2023-11-24 07:37:35,229 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2150, loss[loss=0.07507, simple_loss=0.09565, pruned_loss=0.01808, audio_tagging_loss=0.009164, over 16983.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09258, pruned_loss=0.01358, audio_tagging_loss=0.008927, over 3055817.04 frames. ], batch size: 64, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:37:37,589 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.614e+01 8.763e+01 9.326e+01 1.012e+02 1.387e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 07:38:11,457 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 07:38:21,494 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411000 2023-11-24 07:38:21,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2739940.0, ans=0.1 2023-11-24 07:38:29,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2740006.6666666665, ans=0.0 2023-11-24 07:38:37,191 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2200, loss[loss=0.05459, simple_loss=0.06441, pruned_loss=0.01128, audio_tagging_loss=0.01111, over 15627.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09231, pruned_loss=0.01344, audio_tagging_loss=0.008896, over 3056483.60 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 16.0 2023-11-24 07:38:43,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2740073.3333333335, ans=0.2 2023-11-24 07:38:46,184 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0 2023-11-24 07:39:23,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411050 2023-11-24 07:39:24,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2740273.3333333335, ans=0.0 2023-11-24 07:39:32,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2740340.0, ans=0.125 2023-11-24 07:39:38,465 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2250, loss[loss=0.05514, simple_loss=0.06947, pruned_loss=0.01096, audio_tagging_loss=0.009447, over 15005.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09219, pruned_loss=0.01344, audio_tagging_loss=0.008855, over 3055797.25 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:39:40,800 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.314e+01 8.864e+01 9.346e+01 9.896e+01 1.400e+02, threshold=1.869e+02, percent-clipped=0.0 2023-11-24 07:40:12,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2740540.0, ans=0.125 2023-11-24 07:40:25,273 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411100 2023-11-24 07:40:42,286 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2300, loss[loss=0.06099, simple_loss=0.08072, pruned_loss=0.00982, audio_tagging_loss=0.0108, over 14696.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09163, pruned_loss=0.01328, audio_tagging_loss=0.008858, over 3053489.90 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:40:50,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2740740.0, ans=0.125 2023-11-24 07:40:57,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2740806.6666666665, ans=0.125 2023-11-24 07:41:04,728 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2740806.6666666665, ans=0.025 2023-11-24 07:41:28,776 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411150 2023-11-24 07:41:37,508 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 07:41:40,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2741006.6666666665, ans=0.125 2023-11-24 07:41:44,645 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2350, loss[loss=0.07566, simple_loss=0.1065, pruned_loss=0.01292, audio_tagging_loss=0.009503, over 14954.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09158, pruned_loss=0.01323, audio_tagging_loss=0.00902, over 3050890.68 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:41:47,091 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.561e+01 8.408e+01 9.223e+01 9.902e+01 1.512e+02, threshold=1.845e+02, percent-clipped=0.0 2023-11-24 07:41:53,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2741073.3333333335, ans=0.1 2023-11-24 07:42:04,147 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2741140.0, ans=0.125 2023-11-24 07:42:31,344 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411200 2023-11-24 07:42:34,603 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.19 vs. limit=15.0 2023-11-24 07:42:36,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2741340.0, ans=0.0 2023-11-24 07:42:43,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2741340.0, ans=0.5 2023-11-24 07:42:47,002 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2400, loss[loss=0.0619, simple_loss=0.07331, pruned_loss=0.01228, audio_tagging_loss=0.01296, over 15968.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09041, pruned_loss=0.01316, audio_tagging_loss=0.009178, over 3048899.49 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:42:56,784 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.24 vs. limit=15.0 2023-11-24 07:42:58,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2741473.3333333335, ans=0.0 2023-11-24 07:43:33,270 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411250 2023-11-24 07:43:49,970 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2450, loss[loss=0.07195, simple_loss=0.09387, pruned_loss=0.01471, audio_tagging_loss=0.01031, over 15431.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.08982, pruned_loss=0.01316, audio_tagging_loss=0.009226, over 3050802.76 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:43:52,873 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.523e+01 9.052e+01 9.834e+01 1.481e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 07:44:04,509 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:44:15,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2741873.3333333335, ans=0.1 2023-11-24 07:44:31,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2741940.0, ans=0.1 2023-11-24 07:44:36,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411300 2023-11-24 07:44:44,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2742006.6666666665, ans=0.125 2023-11-24 07:44:53,082 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2500, loss[loss=0.04821, simple_loss=0.06262, pruned_loss=0.006836, audio_tagging_loss=0.01006, over 15677.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09032, pruned_loss=0.01301, audio_tagging_loss=0.009214, over 3059834.59 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:44:54,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2742073.3333333335, ans=0.0 2023-11-24 07:45:30,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2742273.3333333335, ans=0.1 2023-11-24 07:45:39,711 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411350 2023-11-24 07:45:55,255 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2550, loss[loss=0.06669, simple_loss=0.08479, pruned_loss=0.01414, audio_tagging_loss=0.01015, over 15753.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08945, pruned_loss=0.01298, audio_tagging_loss=0.009203, over 3051633.61 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:45:57,675 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.339e+01 8.522e+01 9.145e+01 9.945e+01 1.198e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 07:46:07,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2742473.3333333335, ans=0.1 2023-11-24 07:46:30,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2742540.0, ans=10.0 2023-11-24 07:46:42,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411400 2023-11-24 07:46:58,264 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2600, loss[loss=0.08972, simple_loss=0.1272, pruned_loss=0.01691, audio_tagging_loss=0.009228, over 16372.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.08955, pruned_loss=0.01291, audio_tagging_loss=0.009055, over 3053969.90 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:47:00,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2742740.0, ans=0.125 2023-11-24 07:47:08,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2742740.0, ans=0.07 2023-11-24 07:47:10,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2742806.6666666665, ans=0.125 2023-11-24 07:47:44,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411450 2023-11-24 07:48:01,093 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2650, loss[loss=0.07502, simple_loss=0.1045, pruned_loss=0.01541, audio_tagging_loss=0.007342, over 15406.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.08951, pruned_loss=0.01297, audio_tagging_loss=0.008974, over 3047234.00 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:48:02,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2743073.3333333335, ans=0.2 2023-11-24 07:48:03,408 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.074e+01 8.476e+01 9.136e+01 9.874e+01 1.198e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 07:48:12,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2743140.0, ans=0.125 2023-11-24 07:48:33,556 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:48:39,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2743273.3333333335, ans=0.0 2023-11-24 07:48:47,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411500 2023-11-24 07:48:56,980 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.61 vs. limit=12.0 2023-11-24 07:49:03,307 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2700, loss[loss=0.07339, simple_loss=0.1008, pruned_loss=0.01351, audio_tagging_loss=0.009469, over 15112.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09033, pruned_loss=0.01308, audio_tagging_loss=0.008985, over 3050214.66 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:49:28,185 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.26 vs. limit=15.0 2023-11-24 07:49:30,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2743540.0, ans=10.0 2023-11-24 07:49:44,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2743606.6666666665, ans=0.125 2023-11-24 07:49:49,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411550 2023-11-24 07:49:49,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2743606.6666666665, ans=0.1 2023-11-24 07:50:05,638 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2750, loss[loss=0.05777, simple_loss=0.06766, pruned_loss=0.01182, audio_tagging_loss=0.01213, over 15401.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.08984, pruned_loss=0.01297, audio_tagging_loss=0.00898, over 3051437.14 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:50:09,751 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.955e+01 8.410e+01 9.303e+01 9.931e+01 1.664e+02, threshold=1.861e+02, percent-clipped=0.0 2023-11-24 07:50:51,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411600 2023-11-24 07:50:54,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2744006.6666666665, ans=0.125 2023-11-24 07:50:55,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2744006.6666666665, ans=0.125 2023-11-24 07:50:59,885 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 07:51:06,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2744006.6666666665, ans=0.125 2023-11-24 07:51:08,616 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2800, loss[loss=0.08149, simple_loss=0.1081, pruned_loss=0.01872, audio_tagging_loss=0.008712, over 15817.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09012, pruned_loss=0.01312, audio_tagging_loss=0.008953, over 3043802.08 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:51:15,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2744073.3333333335, ans=0.0 2023-11-24 07:51:23,049 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=10.45 vs. limit=10.0 2023-11-24 07:51:46,658 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0 2023-11-24 07:51:51,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2744273.3333333335, ans=0.0 2023-11-24 07:51:53,700 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:51:54,671 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411650 2023-11-24 07:51:57,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2744340.0, ans=0.125 2023-11-24 07:52:06,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.20 vs. limit=15.0 2023-11-24 07:52:10,222 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2850, loss[loss=0.05055, simple_loss=0.06597, pruned_loss=0.008521, audio_tagging_loss=0.009045, over 14530.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09009, pruned_loss=0.01305, audio_tagging_loss=0.008927, over 3044112.79 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:52:14,257 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.630e+01 8.559e+01 8.889e+01 9.637e+01 1.206e+02, threshold=1.778e+02, percent-clipped=0.0 2023-11-24 07:52:14,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2744406.6666666665, ans=0.2 2023-11-24 07:52:22,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2744473.3333333335, ans=0.125 2023-11-24 07:52:26,111 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0 2023-11-24 07:52:29,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2744473.3333333335, ans=0.125 2023-11-24 07:52:42,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2744540.0, ans=0.0 2023-11-24 07:52:56,681 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411700 2023-11-24 07:53:01,228 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2744673.3333333335, ans=0.5 2023-11-24 07:53:12,561 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2900, loss[loss=0.08099, simple_loss=0.1137, pruned_loss=0.01754, audio_tagging_loss=0.006578, over 15719.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09055, pruned_loss=0.01315, audio_tagging_loss=0.008828, over 3042323.00 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:53:15,901 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 07:53:23,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2744740.0, ans=0.125 2023-11-24 07:53:33,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.06 vs. limit=10.0 2023-11-24 07:53:39,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2744873.3333333335, ans=0.0 2023-11-24 07:53:45,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2744873.3333333335, ans=0.125 2023-11-24 07:53:52,448 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.99 vs. limit=15.0 2023-11-24 07:53:55,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2744940.0, ans=0.125 2023-11-24 07:53:59,246 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411750 2023-11-24 07:54:16,225 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 2950, loss[loss=0.07877, simple_loss=0.113, pruned_loss=0.01488, audio_tagging_loss=0.007405, over 15960.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09122, pruned_loss=0.01315, audio_tagging_loss=0.008894, over 3049834.23 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 07:54:19,690 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.555e+01 8.832e+01 9.434e+01 1.012e+02 1.234e+02, threshold=1.887e+02, percent-clipped=0.0 2023-11-24 07:54:27,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.86 vs. limit=15.0 2023-11-24 07:54:30,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2745140.0, ans=0.125 2023-11-24 07:54:39,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=14.33 vs. limit=15.0 2023-11-24 07:54:44,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2745206.6666666665, ans=0.0 2023-11-24 07:54:57,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0 2023-11-24 07:54:58,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2745273.3333333335, ans=0.0 2023-11-24 07:55:02,593 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411800 2023-11-24 07:55:14,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2745340.0, ans=0.125 2023-11-24 07:55:18,158 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3000, loss[loss=0.06091, simple_loss=0.08057, pruned_loss=0.01187, audio_tagging_loss=0.008758, over 14110.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09082, pruned_loss=0.0132, audio_tagging_loss=0.009026, over 3052208.88 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:55:18,158 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 07:55:43,549 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.2491, 4.9447, 4.4101, 4.8165], device='cuda:3') 2023-11-24 07:56:00,426 INFO [train_asr.py:1253] (3/4) Epoch 35, validation: loss=0.05789, simple_loss=0.05083, pruned_loss=0.005097, audio_tagging_loss=0.02738, over 4681554.00 frames. 2023-11-24 07:56:00,427 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 07:56:16,513 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.19 vs. limit=15.0 2023-11-24 07:56:31,414 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2745540.0, ans=0.5 2023-11-24 07:56:46,039 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411850 2023-11-24 07:56:49,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2745673.3333333335, ans=0.0 2023-11-24 07:57:02,435 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3050, loss[loss=0.06664, simple_loss=0.08854, pruned_loss=0.01381, audio_tagging_loss=0.008556, over 14713.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09126, pruned_loss=0.01336, audio_tagging_loss=0.009008, over 3054316.63 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:57:07,182 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.364e+01 8.656e+01 9.299e+01 1.004e+02 2.054e+02, threshold=1.860e+02, percent-clipped=1.0 2023-11-24 07:57:22,027 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2745806.6666666665, ans=0.0 2023-11-24 07:57:25,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2745873.3333333335, ans=0.2 2023-11-24 07:57:28,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2745873.3333333335, ans=0.1 2023-11-24 07:57:38,864 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 07:57:49,062 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411900 2023-11-24 07:58:04,304 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3100, loss[loss=0.06211, simple_loss=0.07955, pruned_loss=0.01206, audio_tagging_loss=0.01028, over 15396.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09126, pruned_loss=0.01347, audio_tagging_loss=0.009023, over 3048604.06 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:58:50,693 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 411950 2023-11-24 07:59:04,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2746406.6666666665, ans=0.1 2023-11-24 07:59:05,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2746406.6666666665, ans=0.2 2023-11-24 07:59:05,907 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3150, loss[loss=0.075, simple_loss=0.1072, pruned_loss=0.01095, audio_tagging_loss=0.01046, over 14988.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09152, pruned_loss=0.01326, audio_tagging_loss=0.009048, over 3054178.09 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 07:59:09,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2746406.6666666665, ans=0.0 2023-11-24 07:59:11,716 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.198e+01 8.533e+01 9.313e+01 1.001e+02 1.475e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 07:59:37,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2746540.0, ans=0.0 2023-11-24 07:59:43,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2746606.6666666665, ans=0.0 2023-11-24 07:59:47,813 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.44 vs. limit=15.0 2023-11-24 07:59:49,840 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2746606.6666666665, ans=0.125 2023-11-24 07:59:51,941 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412000 2023-11-24 08:00:09,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2746673.3333333335, ans=0.2 2023-11-24 08:00:12,756 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3200, loss[loss=0.06293, simple_loss=0.09028, pruned_loss=0.009871, audio_tagging_loss=0.007918, over 15578.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09094, pruned_loss=0.01316, audio_tagging_loss=0.009182, over 3053534.26 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:00:45,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2746873.3333333335, ans=0.1 2023-11-24 08:00:50,213 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.72 vs. limit=22.5 2023-11-24 08:00:58,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412050 2023-11-24 08:00:58,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2746940.0, ans=0.0 2023-11-24 08:01:05,058 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2747006.6666666665, ans=0.125 2023-11-24 08:01:12,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=12.0 2023-11-24 08:01:14,295 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3250, loss[loss=0.05321, simple_loss=0.07411, pruned_loss=0.008189, audio_tagging_loss=0.007968, over 15783.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09137, pruned_loss=0.01334, audio_tagging_loss=0.00916, over 3054634.48 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:01:20,018 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.341e+01 8.971e+01 9.631e+01 1.385e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-24 08:01:41,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2747206.6666666665, ans=10.0 2023-11-24 08:01:44,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.76 vs. limit=15.0 2023-11-24 08:01:46,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2747206.6666666665, ans=0.125 2023-11-24 08:01:50,267 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.38 vs. limit=6.0 2023-11-24 08:02:00,437 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412100 2023-11-24 08:02:06,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2747340.0, ans=0.125 2023-11-24 08:02:15,752 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3300, loss[loss=0.05352, simple_loss=0.07613, pruned_loss=0.008563, audio_tagging_loss=0.006888, over 15658.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09061, pruned_loss=0.013, audio_tagging_loss=0.009287, over 3061191.86 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:02:18,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2747406.6666666665, ans=0.2 2023-11-24 08:02:25,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2747406.6666666665, ans=0.1 2023-11-24 08:02:37,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2747473.3333333335, ans=0.2 2023-11-24 08:02:43,794 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0 2023-11-24 08:02:52,730 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.45 vs. limit=15.0 2023-11-24 08:03:01,669 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412150 2023-11-24 08:03:19,295 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3350, loss[loss=0.06911, simple_loss=0.09856, pruned_loss=0.0127, audio_tagging_loss=0.007128, over 15527.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09091, pruned_loss=0.01318, audio_tagging_loss=0.009206, over 3065382.01 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:03:25,133 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.567e+01 8.617e+01 9.240e+01 1.018e+02 1.398e+02, threshold=1.848e+02, percent-clipped=0.0 2023-11-24 08:03:28,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2747740.0, ans=0.2 2023-11-24 08:03:56,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2747940.0, ans=0.07 2023-11-24 08:03:57,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2747940.0, ans=0.0 2023-11-24 08:04:03,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2747940.0, ans=0.0 2023-11-24 08:04:04,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412200 2023-11-24 08:04:11,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2748006.6666666665, ans=0.1 2023-11-24 08:04:17,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2748006.6666666665, ans=0.0 2023-11-24 08:04:20,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2748073.3333333335, ans=0.07 2023-11-24 08:04:20,963 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3400, loss[loss=0.06905, simple_loss=0.08849, pruned_loss=0.01681, audio_tagging_loss=0.007993, over 15608.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09085, pruned_loss=0.01303, audio_tagging_loss=0.009094, over 3063982.65 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:04:35,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2748140.0, ans=0.1 2023-11-24 08:04:42,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2748140.0, ans=0.125 2023-11-24 08:04:52,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2748206.6666666665, ans=0.125 2023-11-24 08:05:07,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412250 2023-11-24 08:05:10,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2748340.0, ans=0.125 2023-11-24 08:05:10,349 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.47 vs. limit=15.0 2023-11-24 08:05:19,346 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2748340.0, ans=0.125 2023-11-24 08:05:22,632 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3450, loss[loss=0.06742, simple_loss=0.08643, pruned_loss=0.01515, audio_tagging_loss=0.009055, over 15311.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09098, pruned_loss=0.01311, audio_tagging_loss=0.008898, over 3061607.48 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:05:28,833 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 8.590e+01 9.237e+01 1.003e+02 1.214e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 08:05:48,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2748540.0, ans=0.2 2023-11-24 08:05:51,549 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.22 vs. limit=15.0 2023-11-24 08:06:08,560 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412300 2023-11-24 08:06:16,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2748673.3333333335, ans=0.125 2023-11-24 08:06:18,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2748673.3333333335, ans=0.125 2023-11-24 08:06:25,689 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3500, loss[loss=0.0536, simple_loss=0.06699, pruned_loss=0.009414, audio_tagging_loss=0.01069, over 16207.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09054, pruned_loss=0.01305, audio_tagging_loss=0.008936, over 3057803.97 frames. ], batch size: 63, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:06:34,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2748740.0, ans=0.125 2023-11-24 08:06:51,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.36 vs. limit=15.0 2023-11-24 08:06:52,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2748873.3333333335, ans=0.125 2023-11-24 08:06:57,204 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 08:07:12,220 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412350 2023-11-24 08:07:16,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.45 vs. limit=15.0 2023-11-24 08:07:17,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2749006.6666666665, ans=0.1 2023-11-24 08:07:27,848 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3550, loss[loss=0.07504, simple_loss=0.09749, pruned_loss=0.01367, audio_tagging_loss=0.01263, over 14985.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09055, pruned_loss=0.01308, audio_tagging_loss=0.00891, over 3056466.78 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:07:33,744 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.276e+01 8.953e+01 9.644e+01 1.310e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-24 08:07:42,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2749140.0, ans=0.2 2023-11-24 08:07:49,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2749140.0, ans=0.125 2023-11-24 08:08:14,269 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412400 2023-11-24 08:08:15,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2749273.3333333335, ans=0.125 2023-11-24 08:08:30,062 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3600, loss[loss=0.07805, simple_loss=0.1092, pruned_loss=0.01523, audio_tagging_loss=0.008234, over 15201.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09065, pruned_loss=0.01316, audio_tagging_loss=0.008896, over 3054740.58 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:08:38,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2749406.6666666665, ans=0.0 2023-11-24 08:09:03,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2749540.0, ans=0.1 2023-11-24 08:09:16,455 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412450 2023-11-24 08:09:24,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2749673.3333333335, ans=0.125 2023-11-24 08:09:30,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2749673.3333333335, ans=0.125 2023-11-24 08:09:33,404 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3650, loss[loss=0.04066, simple_loss=0.05091, pruned_loss=0.004856, audio_tagging_loss=0.01035, over 15403.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09053, pruned_loss=0.01321, audio_tagging_loss=0.008875, over 3059627.42 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:09:35,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2749740.0, ans=0.1 2023-11-24 08:09:39,299 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.298e+01 8.982e+01 9.715e+01 1.238e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-24 08:10:19,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412500 2023-11-24 08:10:35,368 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3700, loss[loss=0.07563, simple_loss=0.1055, pruned_loss=0.01541, audio_tagging_loss=0.00745, over 15625.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09103, pruned_loss=0.01318, audio_tagging_loss=0.008792, over 3054119.12 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:10:36,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2750073.3333333335, ans=0.0 2023-11-24 08:11:21,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412550 2023-11-24 08:11:25,340 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2750340.0, ans=0.0 2023-11-24 08:11:37,814 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3750, loss[loss=0.07793, simple_loss=0.1, pruned_loss=0.01745, audio_tagging_loss=0.01047, over 15570.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09204, pruned_loss=0.01314, audio_tagging_loss=0.008761, over 3057917.99 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:11:38,469 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.10 vs. limit=15.0 2023-11-24 08:11:44,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2750406.6666666665, ans=0.125 2023-11-24 08:11:44,955 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.608e+01 8.598e+01 9.101e+01 9.921e+01 1.299e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-24 08:11:49,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.30 vs. limit=15.0 2023-11-24 08:12:05,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.82 vs. limit=15.0 2023-11-24 08:12:08,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2750540.0, ans=0.0 2023-11-24 08:12:20,785 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 08:12:24,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412600 2023-11-24 08:12:40,939 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3800, loss[loss=0.05936, simple_loss=0.07605, pruned_loss=0.01238, audio_tagging_loss=0.00896, over 14074.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09318, pruned_loss=0.01355, audio_tagging_loss=0.008769, over 3052461.58 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:12:59,842 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.57 vs. limit=22.5 2023-11-24 08:13:08,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2750873.3333333335, ans=0.0 2023-11-24 08:13:15,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2750873.3333333335, ans=0.125 2023-11-24 08:13:27,035 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412650 2023-11-24 08:13:43,142 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3850, loss[loss=0.05437, simple_loss=0.08063, pruned_loss=0.008129, audio_tagging_loss=0.00592, over 15728.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09291, pruned_loss=0.01336, audio_tagging_loss=0.008786, over 3059439.84 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:13:43,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2751073.3333333335, ans=0.125 2023-11-24 08:13:50,160 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.566e+01 8.505e+01 9.405e+01 9.842e+01 1.302e+02, threshold=1.881e+02, percent-clipped=0.0 2023-11-24 08:14:09,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2751206.6666666665, ans=0.0 2023-11-24 08:14:18,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2751206.6666666665, ans=0.125 2023-11-24 08:14:29,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412700 2023-11-24 08:14:37,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2751340.0, ans=0.1 2023-11-24 08:14:45,020 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3900, loss[loss=0.07787, simple_loss=0.1088, pruned_loss=0.01652, audio_tagging_loss=0.006958, over 15115.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09252, pruned_loss=0.01341, audio_tagging_loss=0.008898, over 3053309.53 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:14:45,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2751406.6666666665, ans=0.0 2023-11-24 08:14:51,686 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.47 vs. limit=22.5 2023-11-24 08:15:10,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2751540.0, ans=0.2 2023-11-24 08:15:10,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2751540.0, ans=0.125 2023-11-24 08:15:15,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2751540.0, ans=0.125 2023-11-24 08:15:16,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2751540.0, ans=0.125 2023-11-24 08:15:20,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2751540.0, ans=10.0 2023-11-24 08:15:23,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2751606.6666666665, ans=0.125 2023-11-24 08:15:30,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412750 2023-11-24 08:15:41,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2751673.3333333335, ans=0.1 2023-11-24 08:15:43,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2751673.3333333335, ans=0.2 2023-11-24 08:15:47,557 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 3950, loss[loss=0.06293, simple_loss=0.08362, pruned_loss=0.01099, audio_tagging_loss=0.01012, over 14941.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09266, pruned_loss=0.0134, audio_tagging_loss=0.009031, over 3048990.83 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:15:55,234 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.568e+01 8.460e+01 9.070e+01 9.889e+01 1.315e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 08:15:57,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2751740.0, ans=0.0 2023-11-24 08:16:12,589 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.12 vs. limit=15.0 2023-11-24 08:16:18,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2751873.3333333335, ans=0.125 2023-11-24 08:16:19,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2751873.3333333335, ans=0.125 2023-11-24 08:16:32,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.74 vs. limit=12.0 2023-11-24 08:16:33,327 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412800 2023-11-24 08:16:36,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2752006.6666666665, ans=0.07 2023-11-24 08:16:37,994 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2752006.6666666665, ans=0.05 2023-11-24 08:16:38,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2752006.6666666665, ans=0.125 2023-11-24 08:16:47,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2752006.6666666665, ans=15.0 2023-11-24 08:16:47,296 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.16 vs. limit=15.0 2023-11-24 08:16:50,260 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4000, loss[loss=0.07737, simple_loss=0.1075, pruned_loss=0.0162, audio_tagging_loss=0.007445, over 15880.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09215, pruned_loss=0.01328, audio_tagging_loss=0.009161, over 3047592.64 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:16:51,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2752073.3333333335, ans=0.125 2023-11-24 08:17:31,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2752273.3333333335, ans=0.025 2023-11-24 08:17:36,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412850 2023-11-24 08:17:41,651 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 08:17:51,861 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4050, loss[loss=0.05936, simple_loss=0.07889, pruned_loss=0.01066, audio_tagging_loss=0.009253, over 13791.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09205, pruned_loss=0.01332, audio_tagging_loss=0.009236, over 3044627.41 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:17:53,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=12.0 2023-11-24 08:17:54,227 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 08:17:58,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2752406.6666666665, ans=0.2 2023-11-24 08:17:58,907 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.632e+01 8.603e+01 9.245e+01 9.992e+01 1.368e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-24 08:18:20,532 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2752540.0, ans=0.0 2023-11-24 08:18:32,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2752606.6666666665, ans=0.0 2023-11-24 08:18:38,172 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412900 2023-11-24 08:18:44,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2752673.3333333335, ans=0.125 2023-11-24 08:18:45,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2752673.3333333335, ans=0.125 2023-11-24 08:18:54,278 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4100, loss[loss=0.06474, simple_loss=0.08414, pruned_loss=0.01267, audio_tagging_loss=0.01, over 16012.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09139, pruned_loss=0.01317, audio_tagging_loss=0.009197, over 3039005.10 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:19:07,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2752806.6666666665, ans=0.125 2023-11-24 08:19:16,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.65 vs. limit=8.0 2023-11-24 08:19:27,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2752873.3333333335, ans=0.125 2023-11-24 08:19:27,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2752873.3333333335, ans=0.0 2023-11-24 08:19:34,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2752940.0, ans=0.125 2023-11-24 08:19:35,179 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.85 vs. limit=10.0 2023-11-24 08:19:41,109 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 412950 2023-11-24 08:19:44,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2753006.6666666665, ans=0.2 2023-11-24 08:19:57,716 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4150, loss[loss=0.06461, simple_loss=0.08355, pruned_loss=0.01376, audio_tagging_loss=0.009077, over 14877.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09116, pruned_loss=0.0132, audio_tagging_loss=0.009199, over 3032645.93 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:19:58,320 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0 2023-11-24 08:20:04,895 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.354e+01 9.149e+01 1.001e+02 1.406e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 08:20:05,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2753073.3333333335, ans=0.125 2023-11-24 08:20:20,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2753206.6666666665, ans=0.125 2023-11-24 08:20:26,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2753206.6666666665, ans=0.0 2023-11-24 08:20:31,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.11 vs. limit=22.5 2023-11-24 08:20:40,329 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.50 vs. limit=15.0 2023-11-24 08:20:43,167 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 08:20:43,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2753273.3333333335, ans=0.125 2023-11-24 08:20:44,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413000 2023-11-24 08:20:44,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2753273.3333333335, ans=0.07 2023-11-24 08:20:54,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2753340.0, ans=0.125 2023-11-24 08:21:00,135 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4200, loss[loss=0.06599, simple_loss=0.08718, pruned_loss=0.01391, audio_tagging_loss=0.008491, over 14712.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09152, pruned_loss=0.01328, audio_tagging_loss=0.009101, over 3045640.93 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:21:13,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2753473.3333333335, ans=0.125 2023-11-24 08:21:24,019 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.27 vs. limit=15.0 2023-11-24 08:21:29,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2753540.0, ans=0.0 2023-11-24 08:21:35,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2753540.0, ans=0.125 2023-11-24 08:21:39,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2753606.6666666665, ans=0.0 2023-11-24 08:21:46,638 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413050 2023-11-24 08:21:51,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.53 vs. limit=22.5 2023-11-24 08:21:54,170 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.83 vs. limit=15.0 2023-11-24 08:21:58,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2753673.3333333335, ans=0.0 2023-11-24 08:22:01,970 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4250, loss[loss=0.06876, simple_loss=0.08124, pruned_loss=0.01453, audio_tagging_loss=0.0136, over 15795.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09166, pruned_loss=0.01339, audio_tagging_loss=0.009021, over 3048996.54 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:22:11,989 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.341e+01 8.897e+01 9.785e+01 1.363e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-24 08:22:28,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.95 vs. limit=15.0 2023-11-24 08:22:37,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2753873.3333333335, ans=0.0 2023-11-24 08:22:43,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2753940.0, ans=0.1 2023-11-24 08:22:48,153 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413100 2023-11-24 08:22:53,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2754006.6666666665, ans=0.2 2023-11-24 08:23:05,760 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4300, loss[loss=0.06536, simple_loss=0.08655, pruned_loss=0.01431, audio_tagging_loss=0.007779, over 14892.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09175, pruned_loss=0.01333, audio_tagging_loss=0.008969, over 3048331.49 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:23:35,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2754206.6666666665, ans=0.0 2023-11-24 08:23:36,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2754206.6666666665, ans=0.125 2023-11-24 08:23:52,268 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413150 2023-11-24 08:23:54,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2754340.0, ans=0.1 2023-11-24 08:24:07,341 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4350, loss[loss=0.06685, simple_loss=0.09085, pruned_loss=0.01147, audio_tagging_loss=0.009959, over 14075.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09145, pruned_loss=0.01328, audio_tagging_loss=0.009012, over 3043260.55 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:24:15,742 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.072e+01 8.729e+01 9.185e+01 1.017e+02 1.222e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-24 08:24:25,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=2754473.3333333335, ans=0.2 2023-11-24 08:24:37,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2754540.0, ans=0.0 2023-11-24 08:24:41,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2754540.0, ans=0.0 2023-11-24 08:24:53,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413200 2023-11-24 08:24:58,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2754673.3333333335, ans=0.2 2023-11-24 08:25:09,267 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4400, loss[loss=0.06158, simple_loss=0.08567, pruned_loss=0.01136, audio_tagging_loss=0.007384, over 16064.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09091, pruned_loss=0.01321, audio_tagging_loss=0.009089, over 3038026.17 frames. ], batch size: 62, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:25:10,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2754740.0, ans=0.1 2023-11-24 08:25:29,580 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.10 vs. limit=15.0 2023-11-24 08:25:39,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2754873.3333333335, ans=0.125 2023-11-24 08:25:51,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2754940.0, ans=0.125 2023-11-24 08:25:55,016 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413250 2023-11-24 08:25:55,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2754940.0, ans=0.0 2023-11-24 08:25:59,003 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.70 vs. limit=15.0 2023-11-24 08:26:12,569 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4450, loss[loss=0.08751, simple_loss=0.1105, pruned_loss=0.02265, audio_tagging_loss=0.009625, over 15872.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09121, pruned_loss=0.01338, audio_tagging_loss=0.008901, over 3042380.27 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:26:20,941 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.416e+01 8.955e+01 9.921e+01 1.330e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-24 08:26:47,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2755273.3333333335, ans=0.125 2023-11-24 08:26:55,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2755273.3333333335, ans=0.2 2023-11-24 08:26:57,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2755273.3333333335, ans=0.125 2023-11-24 08:26:58,583 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413300 2023-11-24 08:27:13,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2755406.6666666665, ans=0.125 2023-11-24 08:27:14,340 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.92 vs. limit=22.5 2023-11-24 08:27:14,672 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4500, loss[loss=0.07479, simple_loss=0.09664, pruned_loss=0.0135, audio_tagging_loss=0.01297, over 15923.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09134, pruned_loss=0.0133, audio_tagging_loss=0.008828, over 3041571.48 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:27:16,287 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.29 vs. limit=10.0 2023-11-24 08:27:19,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2755406.6666666665, ans=0.1 2023-11-24 08:27:37,191 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.07 vs. limit=6.0 2023-11-24 08:27:48,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2755540.0, ans=0.0 2023-11-24 08:27:57,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.38 vs. limit=15.0 2023-11-24 08:27:59,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2755606.6666666665, ans=0.0 2023-11-24 08:28:00,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413350 2023-11-24 08:28:15,966 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4550, loss[loss=0.06418, simple_loss=0.08077, pruned_loss=0.01275, audio_tagging_loss=0.01105, over 15095.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.0909, pruned_loss=0.01341, audio_tagging_loss=0.008902, over 3037061.89 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:28:24,748 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.054e+01 8.588e+01 9.174e+01 1.003e+02 1.201e+02, threshold=1.835e+02, percent-clipped=0.0 2023-11-24 08:28:52,504 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.32 vs. limit=15.0 2023-11-24 08:28:56,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2755940.0, ans=0.04949747468305833 2023-11-24 08:29:02,803 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413400 2023-11-24 08:29:03,982 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 08:29:19,843 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4600, loss[loss=0.05016, simple_loss=0.06783, pruned_loss=0.004753, audio_tagging_loss=0.0115, over 14736.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.0903, pruned_loss=0.01343, audio_tagging_loss=0.008974, over 3042243.13 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:29:36,050 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.59 vs. limit=15.0 2023-11-24 08:29:37,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2756140.0, ans=0.0 2023-11-24 08:29:48,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2756206.6666666665, ans=0.0 2023-11-24 08:30:06,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413450 2023-11-24 08:30:14,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2756340.0, ans=0.125 2023-11-24 08:30:22,934 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4650, loss[loss=0.07437, simple_loss=0.1006, pruned_loss=0.01613, audio_tagging_loss=0.007939, over 15156.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09145, pruned_loss=0.0136, audio_tagging_loss=0.008943, over 3041995.61 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:30:24,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2756406.6666666665, ans=0.0 2023-11-24 08:30:30,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2756406.6666666665, ans=0.0 2023-11-24 08:30:31,177 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.223e+01 8.404e+01 8.988e+01 9.502e+01 1.265e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-24 08:30:37,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2756473.3333333335, ans=0.0 2023-11-24 08:30:49,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2756540.0, ans=0.1 2023-11-24 08:31:08,793 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413500 2023-11-24 08:31:14,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys.whitening_limit, batch_count=2756673.3333333335, ans=6.0 2023-11-24 08:31:24,028 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4700, loss[loss=0.06183, simple_loss=0.07823, pruned_loss=0.01405, audio_tagging_loss=0.008668, over 15567.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09125, pruned_loss=0.01333, audio_tagging_loss=0.009016, over 3047658.26 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:31:26,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2756740.0, ans=0.1 2023-11-24 08:31:31,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2756740.0, ans=0.125 2023-11-24 08:31:49,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2756873.3333333335, ans=0.0 2023-11-24 08:31:52,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2756873.3333333335, ans=0.0 2023-11-24 08:31:57,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2756873.3333333335, ans=0.125 2023-11-24 08:32:03,474 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.42 vs. limit=22.5 2023-11-24 08:32:10,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413550 2023-11-24 08:32:24,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2757006.6666666665, ans=0.2 2023-11-24 08:32:26,984 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4750, loss[loss=0.09104, simple_loss=0.1263, pruned_loss=0.01714, audio_tagging_loss=0.01075, over 16024.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.0911, pruned_loss=0.01321, audio_tagging_loss=0.009018, over 3046611.93 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 08:32:29,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2757073.3333333335, ans=0.1 2023-11-24 08:32:38,224 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.376e+01 8.531e+01 9.142e+01 9.868e+01 1.183e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 08:32:44,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2757140.0, ans=0.0 2023-11-24 08:32:55,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2757206.6666666665, ans=0.125 2023-11-24 08:33:02,813 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2757273.3333333335, ans=0.125 2023-11-24 08:33:10,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2757273.3333333335, ans=0.2 2023-11-24 08:33:13,242 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413600 2023-11-24 08:33:20,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2757340.0, ans=0.0 2023-11-24 08:33:29,358 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4800, loss[loss=0.04853, simple_loss=0.06517, pruned_loss=0.005867, audio_tagging_loss=0.01008, over 15705.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09084, pruned_loss=0.01305, audio_tagging_loss=0.009151, over 3049414.11 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:33:45,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.19 vs. limit=15.0 2023-11-24 08:33:52,208 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=3.93 vs. limit=6.0 2023-11-24 08:33:59,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2757540.0, ans=0.0 2023-11-24 08:34:14,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2757606.6666666665, ans=0.0 2023-11-24 08:34:15,817 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413650 2023-11-24 08:34:31,810 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4850, loss[loss=0.06822, simple_loss=0.08681, pruned_loss=0.01537, audio_tagging_loss=0.009441, over 14986.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09086, pruned_loss=0.01303, audio_tagging_loss=0.009224, over 3050872.45 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:34:41,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2757740.0, ans=0.125 2023-11-24 08:34:42,341 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.773e+01 9.268e+01 1.019e+02 1.463e+02, threshold=1.854e+02, percent-clipped=0.0 2023-11-24 08:35:08,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2757940.0, ans=0.125 2023-11-24 08:35:18,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413700 2023-11-24 08:35:18,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2757940.0, ans=0.125 2023-11-24 08:35:34,421 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4900, loss[loss=0.07763, simple_loss=0.1061, pruned_loss=0.01516, audio_tagging_loss=0.009431, over 14521.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09136, pruned_loss=0.01312, audio_tagging_loss=0.009101, over 3046682.62 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:36:19,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2758273.3333333335, ans=0.125 2023-11-24 08:36:20,953 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413750 2023-11-24 08:36:22,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2758273.3333333335, ans=0.125 2023-11-24 08:36:28,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2758340.0, ans=0.125 2023-11-24 08:36:37,225 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 4950, loss[loss=0.05751, simple_loss=0.07969, pruned_loss=0.01129, audio_tagging_loss=0.006373, over 14474.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09096, pruned_loss=0.01311, audio_tagging_loss=0.00899, over 3047496.36 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:36:38,561 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2758406.6666666665, ans=0.1 2023-11-24 08:36:47,869 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.724e+01 8.295e+01 8.995e+01 9.901e+01 1.240e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-24 08:36:49,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2758473.3333333335, ans=0.125 2023-11-24 08:37:04,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2758540.0, ans=0.95 2023-11-24 08:37:22,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2758606.6666666665, ans=0.1 2023-11-24 08:37:24,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413800 2023-11-24 08:37:25,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2758606.6666666665, ans=0.04949747468305833 2023-11-24 08:37:39,935 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5000, loss[loss=0.0811, simple_loss=0.121, pruned_loss=0.01414, audio_tagging_loss=0.006475, over 16375.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09085, pruned_loss=0.01297, audio_tagging_loss=0.008929, over 3045083.67 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:37:48,408 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.97 vs. limit=12.0 2023-11-24 08:38:25,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2758940.0, ans=22.5 2023-11-24 08:38:26,171 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413850 2023-11-24 08:38:35,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2759006.6666666665, ans=0.0 2023-11-24 08:38:42,461 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5050, loss[loss=0.06879, simple_loss=0.09439, pruned_loss=0.01174, audio_tagging_loss=0.009853, over 15328.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.0909, pruned_loss=0.013, audio_tagging_loss=0.008887, over 3044208.81 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:38:54,965 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.375e+01 9.094e+01 9.590e+01 1.351e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 08:38:57,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2759140.0, ans=0.0 2023-11-24 08:39:03,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2759140.0, ans=0.035 2023-11-24 08:39:14,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2759206.6666666665, ans=0.125 2023-11-24 08:39:24,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2759273.3333333335, ans=0.2 2023-11-24 08:39:25,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2759273.3333333335, ans=0.0 2023-11-24 08:39:26,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2759273.3333333335, ans=0.0 2023-11-24 08:39:28,560 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413900 2023-11-24 08:39:38,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2759340.0, ans=0.125 2023-11-24 08:39:45,561 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5100, loss[loss=0.05862, simple_loss=0.07791, pruned_loss=0.009397, audio_tagging_loss=0.01027, over 15241.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09113, pruned_loss=0.01313, audio_tagging_loss=0.008856, over 3044377.81 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:40:31,772 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 413950 2023-11-24 08:40:32,315 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.24 vs. limit=15.0 2023-11-24 08:40:34,696 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.27 vs. limit=15.0 2023-11-24 08:40:43,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2759673.3333333335, ans=0.1 2023-11-24 08:40:46,918 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5150, loss[loss=0.05731, simple_loss=0.08885, pruned_loss=0.00538, audio_tagging_loss=0.007502, over 14817.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09035, pruned_loss=0.01288, audio_tagging_loss=0.008858, over 3046193.79 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:40:53,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2759740.0, ans=0.125 2023-11-24 08:40:58,169 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.417e+01 9.000e+01 9.770e+01 1.432e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-24 08:40:59,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2759806.6666666665, ans=0.125 2023-11-24 08:41:03,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2759806.6666666665, ans=0.0 2023-11-24 08:41:23,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2759940.0, ans=0.1 2023-11-24 08:41:33,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414000 2023-11-24 08:41:34,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2759940.0, ans=0.0 2023-11-24 08:41:45,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2760006.6666666665, ans=0.125 2023-11-24 08:41:49,228 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5200, loss[loss=0.06986, simple_loss=0.09467, pruned_loss=0.01536, audio_tagging_loss=0.007176, over 15378.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09077, pruned_loss=0.01297, audio_tagging_loss=0.008843, over 3049792.18 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:42:10,789 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.86 vs. limit=15.0 2023-11-24 08:42:26,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2760273.3333333335, ans=0.1 2023-11-24 08:42:29,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2760273.3333333335, ans=0.0 2023-11-24 08:42:33,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2760273.3333333335, ans=0.0 2023-11-24 08:42:35,976 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414050 2023-11-24 08:42:41,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.44 vs. limit=15.0 2023-11-24 08:42:52,958 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5250, loss[loss=0.06552, simple_loss=0.09477, pruned_loss=0.01326, audio_tagging_loss=0.004874, over 14465.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09137, pruned_loss=0.01321, audio_tagging_loss=0.008751, over 3056148.10 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:43:03,536 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.495e+01 9.165e+01 9.828e+01 1.218e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 08:43:03,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2760473.3333333335, ans=0.125 2023-11-24 08:43:07,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2760473.3333333335, ans=0.2 2023-11-24 08:43:10,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2760473.3333333335, ans=15.0 2023-11-24 08:43:28,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2760606.6666666665, ans=0.07 2023-11-24 08:43:39,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414100 2023-11-24 08:43:54,377 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5300, loss[loss=0.06691, simple_loss=0.09231, pruned_loss=0.01353, audio_tagging_loss=0.007223, over 15121.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09081, pruned_loss=0.01298, audio_tagging_loss=0.008748, over 3057038.30 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:43:56,065 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.30 vs. limit=15.0 2023-11-24 08:44:19,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2760873.3333333335, ans=0.0 2023-11-24 08:44:40,589 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414150 2023-11-24 08:44:44,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2761006.6666666665, ans=0.125 2023-11-24 08:44:46,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2761006.6666666665, ans=0.0 2023-11-24 08:44:47,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2761006.6666666665, ans=0.0 2023-11-24 08:44:50,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2761006.6666666665, ans=0.125 2023-11-24 08:44:54,462 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.48 vs. limit=12.0 2023-11-24 08:44:56,022 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5350, loss[loss=0.06243, simple_loss=0.07828, pruned_loss=0.01442, audio_tagging_loss=0.00887, over 15708.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09089, pruned_loss=0.01304, audio_tagging_loss=0.008783, over 3048462.06 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:45:03,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2761073.3333333335, ans=0.0 2023-11-24 08:45:07,830 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.034e+01 8.463e+01 9.143e+01 9.837e+01 1.383e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 08:45:17,923 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2761140.0, ans=0.125 2023-11-24 08:45:23,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2761206.6666666665, ans=0.125 2023-11-24 08:45:34,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.83 vs. limit=15.0 2023-11-24 08:45:41,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.21 vs. limit=15.0 2023-11-24 08:45:41,489 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414200 2023-11-24 08:45:58,642 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5400, loss[loss=0.08243, simple_loss=0.1098, pruned_loss=0.01841, audio_tagging_loss=0.009142, over 14590.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09178, pruned_loss=0.01332, audio_tagging_loss=0.008924, over 3054037.26 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:46:40,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2761606.6666666665, ans=0.2 2023-11-24 08:46:44,158 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414250 2023-11-24 08:46:52,014 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2761673.3333333335, ans=0.125 2023-11-24 08:47:00,098 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5450, loss[loss=0.06265, simple_loss=0.08419, pruned_loss=0.01374, audio_tagging_loss=0.006813, over 15579.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09258, pruned_loss=0.01356, audio_tagging_loss=0.008839, over 3048388.31 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:47:07,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2761740.0, ans=0.0 2023-11-24 08:47:11,909 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.777e+01 9.198e+01 9.795e+01 1.303e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-24 08:47:13,726 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.91 vs. limit=6.0 2023-11-24 08:47:40,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2761940.0, ans=0.07 2023-11-24 08:47:42,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.07 vs. limit=15.0 2023-11-24 08:47:46,386 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414300 2023-11-24 08:47:50,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.00 vs. limit=12.0 2023-11-24 08:47:52,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2762006.6666666665, ans=0.125 2023-11-24 08:48:01,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2762073.3333333335, ans=0.125 2023-11-24 08:48:01,996 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5500, loss[loss=0.06754, simple_loss=0.09744, pruned_loss=0.01263, audio_tagging_loss=0.006189, over 14320.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09189, pruned_loss=0.01335, audio_tagging_loss=0.008893, over 3043101.68 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:48:08,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2762073.3333333335, ans=0.125 2023-11-24 08:48:31,054 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=22.5 2023-11-24 08:48:48,703 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414350 2023-11-24 08:48:49,272 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.52 vs. limit=15.0 2023-11-24 08:48:49,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.30 vs. limit=15.0 2023-11-24 08:48:50,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2762273.3333333335, ans=0.0 2023-11-24 08:49:05,187 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0 2023-11-24 08:49:05,645 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5550, loss[loss=0.07986, simple_loss=0.1047, pruned_loss=0.01706, audio_tagging_loss=0.01047, over 16472.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09171, pruned_loss=0.01333, audio_tagging_loss=0.009077, over 3047716.23 frames. ], batch size: 64, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:49:17,948 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.668e+01 8.369e+01 9.115e+01 9.992e+01 1.269e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 08:49:19,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2762473.3333333335, ans=0.125 2023-11-24 08:49:39,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.41 vs. limit=12.0 2023-11-24 08:49:49,754 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.03 vs. limit=12.0 2023-11-24 08:49:51,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414400 2023-11-24 08:50:08,642 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5600, loss[loss=0.08347, simple_loss=0.1202, pruned_loss=0.0168, audio_tagging_loss=0.006562, over 15722.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09097, pruned_loss=0.01302, audio_tagging_loss=0.009142, over 3052018.31 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:50:17,688 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.31 vs. limit=22.5 2023-11-24 08:50:20,528 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 08:50:46,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2762940.0, ans=0.125 2023-11-24 08:50:53,449 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 08:50:54,717 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414450 2023-11-24 08:50:56,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2762940.0, ans=0.2 2023-11-24 08:50:59,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2763006.6666666665, ans=0.125 2023-11-24 08:51:09,083 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 08:51:10,078 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5650, loss[loss=0.05409, simple_loss=0.07507, pruned_loss=0.008452, audio_tagging_loss=0.008105, over 16137.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09152, pruned_loss=0.01324, audio_tagging_loss=0.009214, over 3060499.61 frames. ], batch size: 62, lr: 1.94e-03, grad_scale: 32.0 2023-11-24 08:51:18,473 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 08:51:18,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2763073.3333333335, ans=0.1 2023-11-24 08:51:22,328 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.229e+01 8.298e+01 8.960e+01 9.539e+01 1.200e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-24 08:51:34,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=2763206.6666666665, ans=8.0 2023-11-24 08:51:39,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2763206.6666666665, ans=0.015 2023-11-24 08:51:42,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2763206.6666666665, ans=0.1 2023-11-24 08:51:56,292 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414500 2023-11-24 08:51:56,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2763273.3333333335, ans=0.125 2023-11-24 08:52:04,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2763340.0, ans=0.1 2023-11-24 08:52:07,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2763340.0, ans=0.1 2023-11-24 08:52:12,848 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5700, loss[loss=0.07074, simple_loss=0.09919, pruned_loss=0.01307, audio_tagging_loss=0.008078, over 16185.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09228, pruned_loss=0.01338, audio_tagging_loss=0.009097, over 3064641.95 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:52:45,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2763540.0, ans=0.125 2023-11-24 08:52:58,804 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414550 2023-11-24 08:53:03,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.43 vs. limit=6.0 2023-11-24 08:53:15,299 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5750, loss[loss=0.06693, simple_loss=0.09145, pruned_loss=0.01254, audio_tagging_loss=0.008658, over 14842.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09257, pruned_loss=0.01354, audio_tagging_loss=0.008927, over 3059133.49 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:53:16,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2763740.0, ans=0.125 2023-11-24 08:53:21,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2763740.0, ans=0.125 2023-11-24 08:53:28,334 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.460e+01 8.483e+01 9.031e+01 9.613e+01 1.136e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 08:54:01,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414600 2023-11-24 08:54:09,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2764006.6666666665, ans=0.0 2023-11-24 08:54:16,660 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 08:54:17,426 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5800, loss[loss=0.05889, simple_loss=0.08103, pruned_loss=0.01132, audio_tagging_loss=0.007055, over 16337.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09294, pruned_loss=0.01355, audio_tagging_loss=0.008856, over 3059454.51 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 08:54:22,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2764073.3333333335, ans=0.0 2023-11-24 08:55:04,009 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414650 2023-11-24 08:55:14,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn2.whiten.whitening_limit, batch_count=2764340.0, ans=22.5 2023-11-24 08:55:20,527 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5850, loss[loss=0.06933, simple_loss=0.08422, pruned_loss=0.01625, audio_tagging_loss=0.01096, over 14278.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09213, pruned_loss=0.01339, audio_tagging_loss=0.00888, over 3048837.58 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 08:55:32,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2764473.3333333335, ans=0.04949747468305833 2023-11-24 08:55:35,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2764473.3333333335, ans=0.0 2023-11-24 08:55:36,181 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.408e+01 8.653e+01 9.084e+01 9.959e+01 1.265e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 08:55:38,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2764473.3333333335, ans=0.125 2023-11-24 08:55:58,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2764606.6666666665, ans=0.125 2023-11-24 08:56:01,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2764606.6666666665, ans=0.125 2023-11-24 08:56:01,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2764606.6666666665, ans=0.0 2023-11-24 08:56:03,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2764606.6666666665, ans=0.5 2023-11-24 08:56:07,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414700 2023-11-24 08:56:07,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2764606.6666666665, ans=0.0 2023-11-24 08:56:20,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2764673.3333333335, ans=0.0 2023-11-24 08:56:21,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2764673.3333333335, ans=0.09899494936611666 2023-11-24 08:56:23,854 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5900, loss[loss=0.06914, simple_loss=0.08763, pruned_loss=0.0165, audio_tagging_loss=0.008831, over 15104.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09199, pruned_loss=0.01338, audio_tagging_loss=0.008909, over 3045617.44 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 08:56:25,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2764740.0, ans=0.125 2023-11-24 08:56:35,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2764806.6666666665, ans=0.0 2023-11-24 08:56:59,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2764873.3333333335, ans=0.0 2023-11-24 08:57:01,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2764940.0, ans=0.125 2023-11-24 08:57:10,268 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414750 2023-11-24 08:57:26,295 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 5950, loss[loss=0.09376, simple_loss=0.1292, pruned_loss=0.02063, audio_tagging_loss=0.008544, over 16307.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09194, pruned_loss=0.0134, audio_tagging_loss=0.008919, over 3055485.34 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 08:57:37,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2765140.0, ans=0.125 2023-11-24 08:57:40,400 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.559e+01 8.356e+01 8.865e+01 9.769e+01 1.184e+02, threshold=1.773e+02, percent-clipped=0.0 2023-11-24 08:57:48,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2765140.0, ans=0.5 2023-11-24 08:57:56,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2765206.6666666665, ans=0.125 2023-11-24 08:58:02,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2765273.3333333335, ans=0.125 2023-11-24 08:58:12,290 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414800 2023-11-24 08:58:15,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2765340.0, ans=0.125 2023-11-24 08:58:16,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2765340.0, ans=0.0 2023-11-24 08:58:27,962 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6000, loss[loss=0.05346, simple_loss=0.0615, pruned_loss=0.01068, audio_tagging_loss=0.01203, over 14988.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09213, pruned_loss=0.0135, audio_tagging_loss=0.008955, over 3053494.46 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 08:58:27,963 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 08:58:56,275 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.2884, 4.9884, 4.6969, 5.1440], device='cuda:3') 2023-11-24 08:59:09,814 INFO [train_asr.py:1253] (3/4) Epoch 35, validation: loss=0.05756, simple_loss=0.05084, pruned_loss=0.005093, audio_tagging_loss=0.02705, over 4681554.00 frames. 2023-11-24 08:59:09,815 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 08:59:10,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2765406.6666666665, ans=0.125 2023-11-24 08:59:19,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2765406.6666666665, ans=0.0 2023-11-24 08:59:33,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2765540.0, ans=0.125 2023-11-24 08:59:43,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2765540.0, ans=0.125 2023-11-24 08:59:54,900 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 08:59:56,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414850 2023-11-24 09:00:00,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.22 vs. limit=6.0 2023-11-24 09:00:11,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2765740.0, ans=0.125 2023-11-24 09:00:11,861 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6050, loss[loss=0.08467, simple_loss=0.1245, pruned_loss=0.01492, audio_tagging_loss=0.007492, over 16263.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.0909, pruned_loss=0.0132, audio_tagging_loss=0.008862, over 3048405.62 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 09:00:13,700 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.23 vs. limit=12.0 2023-11-24 09:00:18,078 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:00:26,526 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.506e+01 8.401e+01 9.139e+01 9.786e+01 1.325e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 09:00:45,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2765873.3333333335, ans=0.1 2023-11-24 09:00:45,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2765873.3333333335, ans=0.07 2023-11-24 09:00:52,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2765940.0, ans=0.0 2023-11-24 09:00:58,030 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414900 2023-11-24 09:00:59,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2765940.0, ans=0.125 2023-11-24 09:01:08,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2766006.6666666665, ans=0.0 2023-11-24 09:01:13,966 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6100, loss[loss=0.05269, simple_loss=0.0614, pruned_loss=0.01089, audio_tagging_loss=0.01109, over 15297.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09096, pruned_loss=0.01324, audio_tagging_loss=0.008839, over 3051098.16 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 09:01:27,100 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:02:00,265 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 414950 2023-11-24 09:02:06,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2766340.0, ans=0.125 2023-11-24 09:02:16,787 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6150, loss[loss=0.07831, simple_loss=0.1131, pruned_loss=0.01546, audio_tagging_loss=0.006295, over 14866.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09133, pruned_loss=0.01328, audio_tagging_loss=0.00879, over 3045020.67 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 09:02:24,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2766406.6666666665, ans=0.125 2023-11-24 09:02:31,214 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.378e+01 9.060e+01 9.653e+01 1.339e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 09:03:03,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415000 2023-11-24 09:03:04,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2766606.6666666665, ans=0.125 2023-11-24 09:03:15,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.93 vs. limit=15.0 2023-11-24 09:03:18,780 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6200, loss[loss=0.06405, simple_loss=0.07939, pruned_loss=0.01393, audio_tagging_loss=0.01043, over 14145.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09174, pruned_loss=0.01331, audio_tagging_loss=0.008854, over 3051714.10 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 09:03:46,853 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0 2023-11-24 09:04:04,864 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415050 2023-11-24 09:04:21,357 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6250, loss[loss=0.07375, simple_loss=0.1107, pruned_loss=0.01193, audio_tagging_loss=0.006469, over 15315.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09167, pruned_loss=0.01325, audio_tagging_loss=0.008954, over 3052927.10 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 09:04:33,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2767140.0, ans=0.125 2023-11-24 09:04:38,115 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.234e+01 8.847e+01 9.550e+01 1.216e+02, threshold=1.769e+02, percent-clipped=0.0 2023-11-24 09:04:44,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2767140.0, ans=0.125 2023-11-24 09:04:45,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2767206.6666666665, ans=0.5 2023-11-24 09:04:55,717 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.70 vs. limit=10.0 2023-11-24 09:05:05,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2767273.3333333335, ans=0.0 2023-11-24 09:05:08,088 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415100 2023-11-24 09:05:23,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2767406.6666666665, ans=0.125 2023-11-24 09:05:24,283 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6300, loss[loss=0.04922, simple_loss=0.06579, pruned_loss=0.007853, audio_tagging_loss=0.008468, over 15311.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09219, pruned_loss=0.01329, audio_tagging_loss=0.008976, over 3057717.98 frames. ], batch size: 62, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 09:05:37,862 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.16 vs. limit=22.5 2023-11-24 09:06:10,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415150 2023-11-24 09:06:10,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2767606.6666666665, ans=0.0 2023-11-24 09:06:20,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2767673.3333333335, ans=0.1 2023-11-24 09:06:26,033 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6350, loss[loss=0.05949, simple_loss=0.0821, pruned_loss=0.008912, audio_tagging_loss=0.009526, over 15889.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09192, pruned_loss=0.01318, audio_tagging_loss=0.009131, over 3049305.28 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 8.0 2023-11-24 09:06:26,348 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2767740.0, ans=0.125 2023-11-24 09:06:28,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2767740.0, ans=0.125 2023-11-24 09:06:33,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2767740.0, ans=0.125 2023-11-24 09:06:37,099 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0 2023-11-24 09:06:41,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.597e+01 9.129e+01 9.711e+01 1.215e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 09:06:41,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2767806.6666666665, ans=0.125 2023-11-24 09:06:44,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2767806.6666666665, ans=0.04949747468305833 2023-11-24 09:06:45,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2767806.6666666665, ans=0.1 2023-11-24 09:06:49,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.72 vs. limit=5.0 2023-11-24 09:06:49,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2767873.3333333335, ans=0.0 2023-11-24 09:06:56,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2767873.3333333335, ans=0.125 2023-11-24 09:07:03,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2767940.0, ans=0.125 2023-11-24 09:07:11,394 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415200 2023-11-24 09:07:20,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.13 vs. limit=12.0 2023-11-24 09:07:26,519 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0 2023-11-24 09:07:27,187 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6400, loss[loss=0.08845, simple_loss=0.1245, pruned_loss=0.01893, audio_tagging_loss=0.007257, over 15392.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.0931, pruned_loss=0.01354, audio_tagging_loss=0.009154, over 3050983.94 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 09:07:28,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2768073.3333333335, ans=0.0 2023-11-24 09:07:32,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2768073.3333333335, ans=0.125 2023-11-24 09:08:07,412 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:08:12,986 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415250 2023-11-24 09:08:16,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2768340.0, ans=0.125 2023-11-24 09:08:30,522 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6450, loss[loss=0.05749, simple_loss=0.07037, pruned_loss=0.01173, audio_tagging_loss=0.01057, over 14932.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.0922, pruned_loss=0.01332, audio_tagging_loss=0.009235, over 3050000.49 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0 2023-11-24 09:08:31,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2768406.6666666665, ans=0.125 2023-11-24 09:08:32,560 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.37 vs. limit=15.0 2023-11-24 09:08:45,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2768473.3333333335, ans=0.2 2023-11-24 09:08:46,586 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.512e+01 9.076e+01 9.568e+01 1.780e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-24 09:09:17,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415300 2023-11-24 09:09:25,228 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.86 vs. limit=22.5 2023-11-24 09:09:32,756 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6500, loss[loss=0.07362, simple_loss=0.106, pruned_loss=0.01326, audio_tagging_loss=0.007338, over 14429.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09132, pruned_loss=0.01323, audio_tagging_loss=0.009198, over 3047574.52 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:09:40,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2768740.0, ans=0.1 2023-11-24 09:09:54,056 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.06 vs. limit=22.5 2023-11-24 09:10:01,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2768873.3333333335, ans=0.2 2023-11-24 09:10:08,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2768940.0, ans=0.2 2023-11-24 09:10:14,520 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2768940.0, ans=0.1 2023-11-24 09:10:14,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2768940.0, ans=0.0 2023-11-24 09:10:18,025 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415350 2023-11-24 09:10:30,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2769006.6666666665, ans=0.0 2023-11-24 09:10:33,291 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6550, loss[loss=0.04821, simple_loss=0.05797, pruned_loss=0.008844, audio_tagging_loss=0.01038, over 15895.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09004, pruned_loss=0.01301, audio_tagging_loss=0.009116, over 3050460.46 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:10:47,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2769140.0, ans=0.125 2023-11-24 09:10:50,884 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.510e+01 8.986e+01 9.947e+01 1.296e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-24 09:10:56,385 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.11 vs. limit=15.0 2023-11-24 09:11:01,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2769206.6666666665, ans=0.125 2023-11-24 09:11:18,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2769273.3333333335, ans=0.05 2023-11-24 09:11:19,792 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415400 2023-11-24 09:11:29,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2769340.0, ans=0.0 2023-11-24 09:11:37,603 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6600, loss[loss=0.05524, simple_loss=0.06867, pruned_loss=0.01042, audio_tagging_loss=0.01048, over 14778.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.08962, pruned_loss=0.01289, audio_tagging_loss=0.009024, over 3046452.15 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:11:48,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2769473.3333333335, ans=0.125 2023-11-24 09:11:49,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2769473.3333333335, ans=0.0 2023-11-24 09:12:07,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0 2023-11-24 09:12:23,250 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415450 2023-11-24 09:12:34,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.43 vs. limit=22.5 2023-11-24 09:12:39,125 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6650, loss[loss=0.06071, simple_loss=0.07836, pruned_loss=0.01172, audio_tagging_loss=0.009816, over 15205.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.08931, pruned_loss=0.01291, audio_tagging_loss=0.008984, over 3051871.33 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:12:53,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2769806.6666666665, ans=0.125 2023-11-24 09:12:54,684 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.324e+01 8.429e+01 8.957e+01 9.773e+01 1.353e+02, threshold=1.791e+02, percent-clipped=0.0 2023-11-24 09:13:22,800 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=15.0 2023-11-24 09:13:25,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415500 2023-11-24 09:13:41,135 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6700, loss[loss=0.08087, simple_loss=0.1093, pruned_loss=0.01916, audio_tagging_loss=0.00706, over 14997.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08989, pruned_loss=0.01308, audio_tagging_loss=0.008892, over 3047243.36 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:13:43,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2770073.3333333335, ans=0.0 2023-11-24 09:13:43,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.14 vs. limit=22.5 2023-11-24 09:13:47,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2770073.3333333335, ans=0.125 2023-11-24 09:13:59,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2770140.0, ans=0.1 2023-11-24 09:14:27,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415550 2023-11-24 09:14:39,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2770340.0, ans=0.125 2023-11-24 09:14:40,556 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.89 vs. limit=22.5 2023-11-24 09:14:43,945 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6750, loss[loss=0.06199, simple_loss=0.08727, pruned_loss=0.009643, audio_tagging_loss=0.008711, over 15849.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09108, pruned_loss=0.01332, audio_tagging_loss=0.008884, over 3040162.95 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:15:00,421 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.302e+01 8.550e+01 9.289e+01 9.814e+01 1.264e+02, threshold=1.858e+02, percent-clipped=0.0 2023-11-24 09:15:07,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2770540.0, ans=0.0 2023-11-24 09:15:26,694 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.99 vs. limit=22.5 2023-11-24 09:15:28,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2770606.6666666665, ans=0.125 2023-11-24 09:15:30,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415600 2023-11-24 09:15:45,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2770673.3333333335, ans=0.125 2023-11-24 09:15:47,265 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6800, loss[loss=0.06812, simple_loss=0.08666, pruned_loss=0.01637, audio_tagging_loss=0.008427, over 15078.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09054, pruned_loss=0.01317, audio_tagging_loss=0.008891, over 3038204.80 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:15:53,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2770740.0, ans=0.125 2023-11-24 09:16:33,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415650 2023-11-24 09:16:48,700 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6850, loss[loss=0.05477, simple_loss=0.07617, pruned_loss=0.009844, audio_tagging_loss=0.00684, over 14443.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.08993, pruned_loss=0.01305, audio_tagging_loss=0.008913, over 3039846.41 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:17:03,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2771140.0, ans=0.125 2023-11-24 09:17:06,375 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.373e+01 8.924e+01 9.619e+01 1.192e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-24 09:17:14,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2771206.6666666665, ans=0.125 2023-11-24 09:17:14,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2771206.6666666665, ans=0.125 2023-11-24 09:17:16,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2771206.6666666665, ans=0.0 2023-11-24 09:17:34,738 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415700 2023-11-24 09:17:39,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2771340.0, ans=0.125 2023-11-24 09:17:40,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2771340.0, ans=0.1 2023-11-24 09:17:50,600 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6900, loss[loss=0.08313, simple_loss=0.1185, pruned_loss=0.01648, audio_tagging_loss=0.007414, over 15533.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.09008, pruned_loss=0.0129, audio_tagging_loss=0.008898, over 3039857.18 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:18:10,723 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.08 vs. limit=10.0 2023-11-24 09:18:11,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2771473.3333333335, ans=0.2 2023-11-24 09:18:23,469 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.94 vs. limit=22.5 2023-11-24 09:18:26,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2771606.6666666665, ans=0.125 2023-11-24 09:18:36,682 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415750 2023-11-24 09:18:37,852 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 09:18:47,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2771673.3333333335, ans=0.125 2023-11-24 09:18:53,289 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 6950, loss[loss=0.06491, simple_loss=0.0916, pruned_loss=0.01206, audio_tagging_loss=0.007052, over 15639.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09046, pruned_loss=0.01296, audio_tagging_loss=0.008882, over 3043157.48 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:19:10,529 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.481e+01 9.077e+01 9.564e+01 1.596e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-24 09:19:18,641 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.61 vs. limit=15.0 2023-11-24 09:19:19,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2771873.3333333335, ans=0.1 2023-11-24 09:19:20,057 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2771873.3333333335, ans=0.1 2023-11-24 09:19:24,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2771873.3333333335, ans=0.125 2023-11-24 09:19:39,870 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415800 2023-11-24 09:19:41,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2771940.0, ans=0.125 2023-11-24 09:19:49,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2772006.6666666665, ans=0.035 2023-11-24 09:19:55,977 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7000, loss[loss=0.07578, simple_loss=0.1118, pruned_loss=0.0125, audio_tagging_loss=0.007372, over 15550.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09069, pruned_loss=0.01305, audio_tagging_loss=0.008969, over 3039980.70 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:19:58,439 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2772073.3333333335, ans=0.1 2023-11-24 09:20:08,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2772140.0, ans=0.125 2023-11-24 09:20:09,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2772140.0, ans=0.125 2023-11-24 09:20:12,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2772140.0, ans=0.125 2023-11-24 09:20:12,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2772140.0, ans=10.0 2023-11-24 09:20:32,086 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2772273.3333333335, ans=0.0 2023-11-24 09:20:37,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.43 vs. limit=12.0 2023-11-24 09:20:42,074 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415850 2023-11-24 09:20:57,819 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7050, loss[loss=0.06609, simple_loss=0.09541, pruned_loss=0.01125, audio_tagging_loss=0.007132, over 14188.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09001, pruned_loss=0.013, audio_tagging_loss=0.008907, over 3049017.25 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:21:09,182 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.62 vs. limit=10.0 2023-11-24 09:21:11,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2772473.3333333335, ans=0.2 2023-11-24 09:21:15,711 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.245e+01 8.341e+01 9.106e+01 9.596e+01 1.490e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 09:21:22,372 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.78 vs. limit=15.0 2023-11-24 09:21:31,298 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.72 vs. limit=15.0 2023-11-24 09:21:32,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2772540.0, ans=0.1 2023-11-24 09:21:44,348 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415900 2023-11-24 09:22:00,638 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7100, loss[loss=0.06467, simple_loss=0.08512, pruned_loss=0.01323, audio_tagging_loss=0.008879, over 16856.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.0902, pruned_loss=0.01298, audio_tagging_loss=0.008983, over 3054186.10 frames. ], batch size: 67, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:22:26,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2772873.3333333335, ans=0.2 2023-11-24 09:22:36,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2772940.0, ans=0.2 2023-11-24 09:22:46,707 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 415950 2023-11-24 09:23:01,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2773073.3333333335, ans=0.125 2023-11-24 09:23:02,693 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7150, loss[loss=0.06448, simple_loss=0.08548, pruned_loss=0.01096, audio_tagging_loss=0.01079, over 16265.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.08975, pruned_loss=0.01295, audio_tagging_loss=0.008996, over 3055344.64 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:23:02,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2773073.3333333335, ans=0.0 2023-11-24 09:23:09,983 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:23:19,804 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.644e+01 9.348e+01 1.022e+02 1.334e+02, threshold=1.870e+02, percent-clipped=0.0 2023-11-24 09:23:21,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2773140.0, ans=0.0 2023-11-24 09:23:29,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2773206.6666666665, ans=0.0 2023-11-24 09:23:30,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2773206.6666666665, ans=0.125 2023-11-24 09:23:33,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2773206.6666666665, ans=0.125 2023-11-24 09:23:35,910 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.46 vs. limit=10.0 2023-11-24 09:23:43,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2773273.3333333335, ans=0.0 2023-11-24 09:23:48,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416000 2023-11-24 09:23:48,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2773273.3333333335, ans=0.0 2023-11-24 09:24:09,339 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7200, loss[loss=0.0773, simple_loss=0.09991, pruned_loss=0.01843, audio_tagging_loss=0.008917, over 15362.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.08901, pruned_loss=0.01291, audio_tagging_loss=0.009163, over 3052167.56 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:24:20,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2773406.6666666665, ans=0.125 2023-11-24 09:24:20,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2773406.6666666665, ans=0.025 2023-11-24 09:24:22,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2773473.3333333335, ans=0.125 2023-11-24 09:24:32,335 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2773473.3333333335, ans=0.05 2023-11-24 09:24:42,340 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.94 vs. limit=15.0 2023-11-24 09:24:55,799 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416050 2023-11-24 09:24:59,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2773673.3333333335, ans=0.125 2023-11-24 09:25:05,385 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.99 vs. limit=15.0 2023-11-24 09:25:12,606 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7250, loss[loss=0.06933, simple_loss=0.08829, pruned_loss=0.01494, audio_tagging_loss=0.01025, over 15006.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.08986, pruned_loss=0.01311, audio_tagging_loss=0.009151, over 3050824.12 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:25:27,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2773806.6666666665, ans=0.0 2023-11-24 09:25:29,221 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.544e+01 8.813e+01 9.334e+01 1.034e+02 1.372e+02, threshold=1.867e+02, percent-clipped=0.0 2023-11-24 09:25:40,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2773873.3333333335, ans=0.2 2023-11-24 09:25:40,992 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.31 vs. limit=6.0 2023-11-24 09:25:52,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2773940.0, ans=0.2 2023-11-24 09:25:58,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416100 2023-11-24 09:26:13,967 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7300, loss[loss=0.06164, simple_loss=0.08171, pruned_loss=0.01233, audio_tagging_loss=0.008448, over 14858.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09071, pruned_loss=0.01317, audio_tagging_loss=0.009066, over 3044391.33 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:26:22,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2774073.3333333335, ans=0.2 2023-11-24 09:26:24,488 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.38 vs. limit=10.0 2023-11-24 09:26:32,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=15.0 2023-11-24 09:26:35,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2774140.0, ans=0.125 2023-11-24 09:26:43,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2774206.6666666665, ans=0.0 2023-11-24 09:26:43,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.81 vs. limit=15.0 2023-11-24 09:26:49,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2774206.6666666665, ans=0.125 2023-11-24 09:26:54,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2774273.3333333335, ans=0.0 2023-11-24 09:26:55,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2774273.3333333335, ans=0.125 2023-11-24 09:27:00,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416150 2023-11-24 09:27:05,699 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.71 vs. limit=15.0 2023-11-24 09:27:06,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2774340.0, ans=0.125 2023-11-24 09:27:16,247 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7350, loss[loss=0.05848, simple_loss=0.08585, pruned_loss=0.007636, audio_tagging_loss=0.00792, over 15452.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09088, pruned_loss=0.01319, audio_tagging_loss=0.008914, over 3048316.55 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:27:23,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.06 vs. limit=15.0 2023-11-24 09:27:23,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2774406.6666666665, ans=0.125 2023-11-24 09:27:25,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2774406.6666666665, ans=0.0 2023-11-24 09:27:34,371 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.604e+01 9.108e+01 1.008e+02 1.406e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-24 09:27:34,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2774473.3333333335, ans=0.0 2023-11-24 09:27:45,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2774540.0, ans=0.125 2023-11-24 09:27:51,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2774540.0, ans=0.0 2023-11-24 09:27:53,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2774606.6666666665, ans=0.025 2023-11-24 09:27:56,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2774606.6666666665, ans=0.0 2023-11-24 09:28:02,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416200 2023-11-24 09:28:19,624 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7400, loss[loss=0.04706, simple_loss=0.05663, pruned_loss=0.007522, audio_tagging_loss=0.01123, over 15528.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09113, pruned_loss=0.01312, audio_tagging_loss=0.008775, over 3050422.26 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:28:31,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2774806.6666666665, ans=0.125 2023-11-24 09:28:37,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2774806.6666666665, ans=0.125 2023-11-24 09:28:51,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2774873.3333333335, ans=0.125 2023-11-24 09:29:05,710 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416250 2023-11-24 09:29:21,074 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7450, loss[loss=0.05396, simple_loss=0.07316, pruned_loss=0.008183, audio_tagging_loss=0.0092, over 16057.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09188, pruned_loss=0.01326, audio_tagging_loss=0.008727, over 3048697.33 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:29:25,419 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.03 vs. limit=10.0 2023-11-24 09:29:25,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2775073.3333333335, ans=0.125 2023-11-24 09:29:38,157 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 8.229e+01 9.115e+01 9.575e+01 1.366e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 09:30:04,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.39 vs. limit=22.5 2023-11-24 09:30:07,445 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416300 2023-11-24 09:30:15,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2775340.0, ans=0.2 2023-11-24 09:30:22,787 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7500, loss[loss=0.05516, simple_loss=0.07247, pruned_loss=0.009886, audio_tagging_loss=0.009036, over 14941.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09192, pruned_loss=0.01338, audio_tagging_loss=0.008715, over 3051709.67 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:30:35,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.74 vs. limit=15.0 2023-11-24 09:30:48,721 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.67 vs. limit=15.0 2023-11-24 09:30:54,061 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:30:54,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2775540.0, ans=0.1 2023-11-24 09:31:08,213 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416350 2023-11-24 09:31:08,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2775606.6666666665, ans=0.1 2023-11-24 09:31:14,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2775673.3333333335, ans=0.125 2023-11-24 09:31:24,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2775740.0, ans=0.125 2023-11-24 09:31:24,648 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.78 vs. limit=15.0 2023-11-24 09:31:25,167 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7550, loss[loss=0.09332, simple_loss=0.1341, pruned_loss=0.02036, audio_tagging_loss=0.005909, over 16086.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09225, pruned_loss=0.01351, audio_tagging_loss=0.008619, over 3040808.70 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:31:41,460 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.230e+01 8.548e+01 9.078e+01 9.572e+01 1.867e+02, threshold=1.816e+02, percent-clipped=1.0 2023-11-24 09:31:57,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2775873.3333333335, ans=0.2 2023-11-24 09:32:11,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416400 2023-11-24 09:32:12,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2775940.0, ans=0.125 2023-11-24 09:32:12,739 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.10 vs. limit=15.0 2023-11-24 09:32:18,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2776006.6666666665, ans=0.125 2023-11-24 09:32:22,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2776006.6666666665, ans=0.0 2023-11-24 09:32:26,776 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7600, loss[loss=0.08164, simple_loss=0.1156, pruned_loss=0.01608, audio_tagging_loss=0.007752, over 15695.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09129, pruned_loss=0.01337, audio_tagging_loss=0.008761, over 3038346.18 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:32:43,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2776140.0, ans=0.125 2023-11-24 09:32:57,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2776206.6666666665, ans=0.07 2023-11-24 09:33:12,572 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416450 2023-11-24 09:33:12,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2776273.3333333335, ans=0.125 2023-11-24 09:33:15,623 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.03 vs. limit=15.0 2023-11-24 09:33:27,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=2776406.6666666665, ans=15.0 2023-11-24 09:33:27,756 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7650, loss[loss=0.06137, simple_loss=0.07091, pruned_loss=0.01397, audio_tagging_loss=0.01194, over 14607.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09037, pruned_loss=0.01304, audio_tagging_loss=0.008831, over 3035508.85 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:33:30,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2776406.6666666665, ans=0.125 2023-11-24 09:33:32,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2776406.6666666665, ans=0.125 2023-11-24 09:33:48,912 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.062e+01 8.355e+01 9.042e+01 9.809e+01 1.325e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 09:33:52,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2776540.0, ans=0.0 2023-11-24 09:34:01,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2776540.0, ans=0.125 2023-11-24 09:34:05,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2776606.6666666665, ans=0.1 2023-11-24 09:34:07,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.33 vs. limit=15.0 2023-11-24 09:34:13,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416500 2023-11-24 09:34:30,125 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7700, loss[loss=0.06863, simple_loss=0.09326, pruned_loss=0.01268, audio_tagging_loss=0.009321, over 16085.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09149, pruned_loss=0.01327, audio_tagging_loss=0.008866, over 3034723.96 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:34:33,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2776740.0, ans=0.025 2023-11-24 09:34:53,960 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2776873.3333333335, ans=0.0 2023-11-24 09:35:14,386 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416550 2023-11-24 09:35:14,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2776940.0, ans=0.0 2023-11-24 09:35:17,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2777006.6666666665, ans=0.1 2023-11-24 09:35:31,197 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7750, loss[loss=0.04892, simple_loss=0.06677, pruned_loss=0.007502, audio_tagging_loss=0.008032, over 16288.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09004, pruned_loss=0.01319, audio_tagging_loss=0.009032, over 3031962.57 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:35:50,349 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.304e+01 9.008e+01 9.844e+01 1.304e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-24 09:36:02,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2777206.6666666665, ans=0.025 2023-11-24 09:36:05,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2777206.6666666665, ans=0.0 2023-11-24 09:36:17,204 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416600 2023-11-24 09:36:31,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2777340.0, ans=0.125 2023-11-24 09:36:33,296 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7800, loss[loss=0.06495, simple_loss=0.08601, pruned_loss=0.01249, audio_tagging_loss=0.009462, over 14782.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09107, pruned_loss=0.0134, audio_tagging_loss=0.009041, over 3037168.55 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:36:35,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.43 vs. limit=10.0 2023-11-24 09:36:39,969 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=12.0 2023-11-24 09:36:40,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2777406.6666666665, ans=0.125 2023-11-24 09:36:42,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.80 vs. limit=15.0 2023-11-24 09:36:57,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.78 vs. limit=6.0 2023-11-24 09:37:05,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2777540.0, ans=0.0 2023-11-24 09:37:19,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416650 2023-11-24 09:37:25,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2777673.3333333335, ans=0.0 2023-11-24 09:37:33,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2777673.3333333335, ans=0.125 2023-11-24 09:37:35,053 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7850, loss[loss=0.06127, simple_loss=0.07939, pruned_loss=0.01165, audio_tagging_loss=0.009925, over 15901.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09095, pruned_loss=0.01343, audio_tagging_loss=0.009024, over 3041969.95 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:37:36,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2777740.0, ans=0.125 2023-11-24 09:37:53,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=2777806.6666666665, ans=0.1 2023-11-24 09:37:55,541 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.418e+01 9.005e+01 9.795e+01 1.227e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-24 09:38:04,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2777873.3333333335, ans=0.2 2023-11-24 09:38:09,265 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.04 vs. limit=6.0 2023-11-24 09:38:21,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416700 2023-11-24 09:38:37,571 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7900, loss[loss=0.0769, simple_loss=0.09745, pruned_loss=0.01454, audio_tagging_loss=0.01363, over 16462.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09059, pruned_loss=0.0133, audio_tagging_loss=0.009161, over 3045640.11 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:38:57,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2778140.0, ans=0.1 2023-11-24 09:39:00,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2778206.6666666665, ans=0.2 2023-11-24 09:39:01,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2778206.6666666665, ans=6.0 2023-11-24 09:39:03,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2778206.6666666665, ans=0.125 2023-11-24 09:39:09,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2778206.6666666665, ans=0.1 2023-11-24 09:39:14,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2778273.3333333335, ans=0.05 2023-11-24 09:39:23,510 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416750 2023-11-24 09:39:36,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2778340.0, ans=0.125 2023-11-24 09:39:38,834 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 7950, loss[loss=0.0773, simple_loss=0.1001, pruned_loss=0.01931, audio_tagging_loss=0.007924, over 14928.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.0904, pruned_loss=0.01314, audio_tagging_loss=0.009229, over 3043505.24 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:39:47,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2778406.6666666665, ans=0.0 2023-11-24 09:39:53,579 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 09:39:58,836 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 8.587e+01 9.115e+01 9.680e+01 1.233e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 09:40:02,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2778473.3333333335, ans=0.0 2023-11-24 09:40:22,159 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=15.0 2023-11-24 09:40:25,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416800 2023-11-24 09:40:30,813 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.14 vs. limit=12.0 2023-11-24 09:40:41,547 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8000, loss[loss=0.07597, simple_loss=0.09694, pruned_loss=0.01772, audio_tagging_loss=0.009784, over 16607.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09019, pruned_loss=0.01313, audio_tagging_loss=0.009208, over 3041684.79 frames. ], batch size: 63, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:41:27,989 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416850 2023-11-24 09:41:29,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2778940.0, ans=0.0 2023-11-24 09:41:34,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2779006.6666666665, ans=0.1 2023-11-24 09:41:38,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2779006.6666666665, ans=0.125 2023-11-24 09:41:41,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2779006.6666666665, ans=0.125 2023-11-24 09:41:44,423 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8050, loss[loss=0.05774, simple_loss=0.07518, pruned_loss=0.009648, audio_tagging_loss=0.01051, over 15503.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.08959, pruned_loss=0.01304, audio_tagging_loss=0.009194, over 3041120.70 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:41:52,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2779073.3333333335, ans=0.125 2023-11-24 09:42:05,150 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.535e+01 9.115e+01 9.754e+01 2.166e+02, threshold=1.823e+02, percent-clipped=1.0 2023-11-24 09:42:22,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2779273.3333333335, ans=0.125 2023-11-24 09:42:28,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2779273.3333333335, ans=0.0 2023-11-24 09:42:30,570 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416900 2023-11-24 09:42:30,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2779273.3333333335, ans=0.09899494936611666 2023-11-24 09:42:39,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2779340.0, ans=0.125 2023-11-24 09:42:46,447 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8100, loss[loss=0.07405, simple_loss=0.09229, pruned_loss=0.0154, audio_tagging_loss=0.0125, over 14253.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09081, pruned_loss=0.01318, audio_tagging_loss=0.009051, over 3046366.92 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:42:46,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2779406.6666666665, ans=0.1 2023-11-24 09:42:49,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2779406.6666666665, ans=0.125 2023-11-24 09:42:56,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2779406.6666666665, ans=0.0 2023-11-24 09:43:04,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2779473.3333333335, ans=0.2 2023-11-24 09:43:15,847 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.26 vs. limit=12.0 2023-11-24 09:43:18,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2779540.0, ans=0.125 2023-11-24 09:43:19,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2779540.0, ans=0.2 2023-11-24 09:43:29,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2779606.6666666665, ans=0.125 2023-11-24 09:43:32,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 416950 2023-11-24 09:43:41,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2779673.3333333335, ans=0.125 2023-11-24 09:43:47,839 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8150, loss[loss=0.06651, simple_loss=0.09169, pruned_loss=0.01289, audio_tagging_loss=0.007772, over 15352.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09076, pruned_loss=0.01325, audio_tagging_loss=0.008977, over 3042027.85 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:43:53,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.84 vs. limit=10.0 2023-11-24 09:44:09,535 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.453e+01 8.620e+01 9.310e+01 1.002e+02 1.330e+02, threshold=1.862e+02, percent-clipped=0.0 2023-11-24 09:44:22,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0 2023-11-24 09:44:34,063 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417000 2023-11-24 09:44:50,745 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8200, loss[loss=0.06603, simple_loss=0.09086, pruned_loss=0.01311, audio_tagging_loss=0.007497, over 15849.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09164, pruned_loss=0.01332, audio_tagging_loss=0.008938, over 3039427.65 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:44:51,961 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 09:44:56,047 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0 2023-11-24 09:45:16,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2780206.6666666665, ans=0.2 2023-11-24 09:45:17,723 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2780206.6666666665, ans=0.0 2023-11-24 09:45:19,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=12.0 2023-11-24 09:45:37,007 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417050 2023-11-24 09:45:38,764 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0 2023-11-24 09:45:52,458 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8250, loss[loss=0.06151, simple_loss=0.07258, pruned_loss=0.01161, audio_tagging_loss=0.01361, over 13292.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09232, pruned_loss=0.01335, audio_tagging_loss=0.008841, over 3035298.54 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:45:53,194 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.63 vs. limit=15.0 2023-11-24 09:46:10,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2780473.3333333335, ans=0.05 2023-11-24 09:46:13,245 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.98 vs. limit=10.0 2023-11-24 09:46:13,773 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.658e+01 9.148e+01 9.760e+01 1.278e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 09:46:38,661 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417100 2023-11-24 09:46:41,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2780673.3333333335, ans=0.0 2023-11-24 09:46:45,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2780673.3333333335, ans=0.04949747468305833 2023-11-24 09:46:54,581 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8300, loss[loss=0.06991, simple_loss=0.0945, pruned_loss=0.01422, audio_tagging_loss=0.008433, over 15459.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09137, pruned_loss=0.01332, audio_tagging_loss=0.008848, over 3037216.41 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:47:19,548 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.58 vs. limit=15.0 2023-11-24 09:47:21,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2780873.3333333335, ans=15.0 2023-11-24 09:47:35,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2780940.0, ans=0.2 2023-11-24 09:47:39,445 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.09 vs. limit=15.0 2023-11-24 09:47:40,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2780940.0, ans=0.1 2023-11-24 09:47:41,296 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417150 2023-11-24 09:47:48,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2781006.6666666665, ans=0.125 2023-11-24 09:47:58,465 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8350, loss[loss=0.0558, simple_loss=0.06858, pruned_loss=0.01219, audio_tagging_loss=0.009325, over 14581.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09061, pruned_loss=0.0132, audio_tagging_loss=0.008884, over 3041943.85 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:48:03,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2781073.3333333335, ans=0.125 2023-11-24 09:48:03,480 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:48:18,478 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.379e+01 9.010e+01 9.696e+01 1.228e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-24 09:48:22,061 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.94 vs. limit=22.5 2023-11-24 09:48:44,411 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417200 2023-11-24 09:49:00,075 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8400, loss[loss=0.06656, simple_loss=0.07841, pruned_loss=0.01438, audio_tagging_loss=0.01297, over 14401.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09102, pruned_loss=0.01336, audio_tagging_loss=0.00889, over 3046751.77 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:49:02,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2781406.6666666665, ans=0.2 2023-11-24 09:49:20,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2781473.3333333335, ans=0.125 2023-11-24 09:49:29,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2781540.0, ans=0.0 2023-11-24 09:49:40,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2781606.6666666665, ans=0.125 2023-11-24 09:49:41,641 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:49:42,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2781606.6666666665, ans=0.0 2023-11-24 09:49:46,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417250 2023-11-24 09:49:56,674 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.29 vs. limit=22.5 2023-11-24 09:50:02,434 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8450, loss[loss=0.08341, simple_loss=0.1125, pruned_loss=0.01984, audio_tagging_loss=0.00732, over 16239.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09015, pruned_loss=0.01309, audio_tagging_loss=0.008977, over 3042904.54 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:50:03,118 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.77 vs. limit=15.0 2023-11-24 09:50:07,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2781740.0, ans=0.125 2023-11-24 09:50:24,535 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.178e+01 8.593e+01 9.155e+01 9.677e+01 1.308e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-24 09:50:24,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2781806.6666666665, ans=0.2 2023-11-24 09:50:39,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2781940.0, ans=0.2 2023-11-24 09:50:49,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417300 2023-11-24 09:50:49,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2781940.0, ans=0.125 2023-11-24 09:50:53,525 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=12.0 2023-11-24 09:51:06,318 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8500, loss[loss=0.07714, simple_loss=0.1095, pruned_loss=0.01447, audio_tagging_loss=0.007938, over 15368.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09139, pruned_loss=0.01306, audio_tagging_loss=0.008947, over 3051005.98 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:51:06,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2782073.3333333335, ans=0.125 2023-11-24 09:51:13,886 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2782073.3333333335, ans=0.0 2023-11-24 09:51:24,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2782140.0, ans=0.2 2023-11-24 09:51:27,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2782140.0, ans=0.1 2023-11-24 09:51:52,015 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417350 2023-11-24 09:51:55,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2782340.0, ans=0.0 2023-11-24 09:52:07,816 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8550, loss[loss=0.06216, simple_loss=0.07963, pruned_loss=0.01214, audio_tagging_loss=0.0102, over 16258.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09063, pruned_loss=0.01289, audio_tagging_loss=0.009048, over 3050301.21 frames. ], batch size: 63, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:52:11,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2782406.6666666665, ans=0.125 2023-11-24 09:52:17,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2782406.6666666665, ans=0.125 2023-11-24 09:52:24,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2782473.3333333335, ans=0.125 2023-11-24 09:52:26,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2782473.3333333335, ans=0.125 2023-11-24 09:52:29,799 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.685e+01 9.326e+01 1.025e+02 1.164e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 09:52:31,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2782540.0, ans=0.2 2023-11-24 09:52:34,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2782540.0, ans=0.125 2023-11-24 09:52:54,245 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417400 2023-11-24 09:52:55,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0 2023-11-24 09:53:10,015 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8600, loss[loss=0.06994, simple_loss=0.08621, pruned_loss=0.01305, audio_tagging_loss=0.01378, over 15163.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09076, pruned_loss=0.01294, audio_tagging_loss=0.00914, over 3049481.16 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:53:14,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2782740.0, ans=0.2 2023-11-24 09:53:23,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2782806.6666666665, ans=0.0 2023-11-24 09:53:49,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.41 vs. limit=15.0 2023-11-24 09:53:56,306 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417450 2023-11-24 09:54:12,798 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8650, loss[loss=0.05873, simple_loss=0.07179, pruned_loss=0.01157, audio_tagging_loss=0.01126, over 15915.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09166, pruned_loss=0.01306, audio_tagging_loss=0.00911, over 3049001.43 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:54:19,443 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.57 vs. limit=10.0 2023-11-24 09:54:20,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2783073.3333333335, ans=0.125 2023-11-24 09:54:25,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2783140.0, ans=15.0 2023-11-24 09:54:29,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2783140.0, ans=0.125 2023-11-24 09:54:35,508 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.880e+01 8.560e+01 9.076e+01 1.003e+02 1.218e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-24 09:54:49,487 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.43 vs. limit=15.0 2023-11-24 09:54:53,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2783273.3333333335, ans=0.125 2023-11-24 09:54:59,537 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417500 2023-11-24 09:55:16,150 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8700, loss[loss=0.05565, simple_loss=0.07562, pruned_loss=0.008821, audio_tagging_loss=0.009017, over 15418.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09125, pruned_loss=0.01306, audio_tagging_loss=0.009247, over 3047248.47 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:55:16,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.01 vs. limit=15.0 2023-11-24 09:55:21,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2783406.6666666665, ans=0.125 2023-11-24 09:55:21,510 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5 2023-11-24 09:55:28,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2783473.3333333335, ans=0.0 2023-11-24 09:55:30,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0 2023-11-24 09:55:36,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2783473.3333333335, ans=0.0 2023-11-24 09:55:57,322 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0 2023-11-24 09:56:00,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2783606.6666666665, ans=0.0 2023-11-24 09:56:02,431 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417550 2023-11-24 09:56:07,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2783673.3333333335, ans=0.125 2023-11-24 09:56:07,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2783673.3333333335, ans=0.125 2023-11-24 09:56:07,748 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0 2023-11-24 09:56:15,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2783673.3333333335, ans=0.125 2023-11-24 09:56:16,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.45 vs. limit=22.5 2023-11-24 09:56:17,915 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8750, loss[loss=0.08314, simple_loss=0.1127, pruned_loss=0.0199, audio_tagging_loss=0.006871, over 16201.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09167, pruned_loss=0.01321, audio_tagging_loss=0.009309, over 3052400.25 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 09:56:28,726 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.78 vs. limit=6.0 2023-11-24 09:56:30,981 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.96 vs. limit=15.0 2023-11-24 09:56:40,385 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.442e+01 8.767e+01 9.281e+01 1.037e+02 1.287e+02, threshold=1.856e+02, percent-clipped=0.0 2023-11-24 09:56:51,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2783873.3333333335, ans=0.0 2023-11-24 09:56:55,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.83 vs. limit=15.0 2023-11-24 09:57:00,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2783940.0, ans=0.0 2023-11-24 09:57:03,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417600 2023-11-24 09:57:07,277 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:57:20,230 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8800, loss[loss=0.07098, simple_loss=0.1007, pruned_loss=0.01215, audio_tagging_loss=0.008504, over 15696.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09212, pruned_loss=0.01328, audio_tagging_loss=0.009265, over 3051839.59 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:57:20,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2784073.3333333335, ans=0.2 2023-11-24 09:57:21,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2784073.3333333335, ans=0.2 2023-11-24 09:57:32,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2784140.0, ans=0.125 2023-11-24 09:57:34,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2784140.0, ans=0.2 2023-11-24 09:57:38,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2784140.0, ans=0.125 2023-11-24 09:57:48,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2784206.6666666665, ans=0.125 2023-11-24 09:57:50,923 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 09:57:56,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2784273.3333333335, ans=0.0 2023-11-24 09:58:05,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417650 2023-11-24 09:58:17,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2784340.0, ans=0.125 2023-11-24 09:58:17,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2784340.0, ans=0.2 2023-11-24 09:58:17,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2784340.0, ans=0.1 2023-11-24 09:58:21,983 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8850, loss[loss=0.07254, simple_loss=0.1005, pruned_loss=0.01485, audio_tagging_loss=0.007447, over 14160.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09316, pruned_loss=0.01343, audio_tagging_loss=0.009229, over 3049118.60 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:58:23,959 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.11 vs. limit=15.0 2023-11-24 09:58:25,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2784406.6666666665, ans=0.0 2023-11-24 09:58:26,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2784406.6666666665, ans=0.125 2023-11-24 09:58:32,837 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 09:58:35,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2784473.3333333335, ans=0.015 2023-11-24 09:58:43,356 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.550e+01 9.128e+01 9.722e+01 1.464e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 09:58:57,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2784606.6666666665, ans=0.1 2023-11-24 09:59:06,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2784606.6666666665, ans=0.125 2023-11-24 09:59:07,675 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417700 2023-11-24 09:59:07,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2784606.6666666665, ans=0.125 2023-11-24 09:59:23,017 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8900, loss[loss=0.05265, simple_loss=0.06647, pruned_loss=0.009647, audio_tagging_loss=0.009769, over 15065.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09371, pruned_loss=0.01388, audio_tagging_loss=0.00903, over 3049094.68 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 09:59:24,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.48 vs. limit=22.5 2023-11-24 09:59:35,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2784806.6666666665, ans=0.2 2023-11-24 09:59:40,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.05 vs. limit=22.5 2023-11-24 09:59:46,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2784806.6666666665, ans=0.125 2023-11-24 09:59:53,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2784873.3333333335, ans=0.125 2023-11-24 09:59:58,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2784873.3333333335, ans=0.125 2023-11-24 10:00:02,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2784940.0, ans=0.125 2023-11-24 10:00:06,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2784940.0, ans=0.125 2023-11-24 10:00:08,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417750 2023-11-24 10:00:23,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2785073.3333333335, ans=0.125 2023-11-24 10:00:24,363 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 8950, loss[loss=0.04972, simple_loss=0.06355, pruned_loss=0.008074, audio_tagging_loss=0.009867, over 15807.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09319, pruned_loss=0.01353, audio_tagging_loss=0.008859, over 3051860.88 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:00:47,811 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.675e+01 8.510e+01 9.195e+01 1.012e+02 1.504e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 10:01:09,900 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417800 2023-11-24 10:01:15,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2785340.0, ans=0.1 2023-11-24 10:01:15,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2785340.0, ans=0.05 2023-11-24 10:01:16,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2785340.0, ans=0.1 2023-11-24 10:01:20,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2785340.0, ans=0.125 2023-11-24 10:01:24,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2785340.0, ans=0.035 2023-11-24 10:01:26,227 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9000, loss[loss=0.07425, simple_loss=0.1064, pruned_loss=0.01527, audio_tagging_loss=0.005758, over 13933.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09333, pruned_loss=0.01352, audio_tagging_loss=0.008832, over 3046408.52 frames. ], batch size: 51, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:01:26,228 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 10:01:47,755 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.4779, 3.7863, 3.8108, 3.5540], device='cuda:3') 2023-11-24 10:01:52,866 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.0838, 3.7628, 2.3512, 3.5248], device='cuda:3') 2023-11-24 10:01:56,076 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.6510, 3.0048, 2.7676, 2.7311, 3.3943, 3.3734, 3.2953, 3.5845], device='cuda:3') 2023-11-24 10:01:57,805 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.3468, 3.0016, 3.3445, 2.9477, 3.6999, 3.7455, 3.2914, 3.2253], device='cuda:3') 2023-11-24 10:02:05,370 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4846, 3.3176, 3.7220, 3.5548], device='cuda:3') 2023-11-24 10:02:10,229 INFO [train_asr.py:1253] (3/4) Epoch 35, validation: loss=0.05875, simple_loss=0.05079, pruned_loss=0.005122, audio_tagging_loss=0.02823, over 4681554.00 frames. 2023-11-24 10:02:10,230 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 10:02:35,519 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.48 vs. limit=6.0 2023-11-24 10:02:41,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2785540.0, ans=0.2 2023-11-24 10:02:45,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2785540.0, ans=0.0 2023-11-24 10:02:49,281 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2785606.6666666665, ans=0.125 2023-11-24 10:02:56,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417850 2023-11-24 10:03:00,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2785673.3333333335, ans=0.125 2023-11-24 10:03:12,421 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9050, loss[loss=0.05467, simple_loss=0.07177, pruned_loss=0.01061, audio_tagging_loss=0.008173, over 14846.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09274, pruned_loss=0.01338, audio_tagging_loss=0.008749, over 3047602.83 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:03:36,292 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.575e+01 9.161e+01 9.743e+01 1.218e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 10:03:42,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2785873.3333333335, ans=0.2 2023-11-24 10:03:43,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2785873.3333333335, ans=0.125 2023-11-24 10:03:58,353 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417900 2023-11-24 10:04:12,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2786006.6666666665, ans=0.0 2023-11-24 10:04:14,926 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9100, loss[loss=0.08, simple_loss=0.1079, pruned_loss=0.01922, audio_tagging_loss=0.006809, over 14701.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09258, pruned_loss=0.01335, audio_tagging_loss=0.008668, over 3048663.19 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:04:25,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2786140.0, ans=0.1 2023-11-24 10:04:30,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.00 vs. limit=15.0 2023-11-24 10:04:32,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2786140.0, ans=0.0 2023-11-24 10:04:53,282 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2786273.3333333335, ans=0.125 2023-11-24 10:05:00,728 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 417950 2023-11-24 10:05:02,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2786273.3333333335, ans=0.125 2023-11-24 10:05:15,917 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9150, loss[loss=0.08158, simple_loss=0.1158, pruned_loss=0.01517, audio_tagging_loss=0.008516, over 14786.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09222, pruned_loss=0.0132, audio_tagging_loss=0.008643, over 3048918.66 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:05:23,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2786406.6666666665, ans=0.0 2023-11-24 10:05:37,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2786473.3333333335, ans=0.2 2023-11-24 10:05:39,654 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.404e+01 9.029e+01 9.911e+01 1.244e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 10:05:45,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2786540.0, ans=0.1 2023-11-24 10:05:50,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2786540.0, ans=0.025 2023-11-24 10:05:50,090 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2786540.0, ans=0.0 2023-11-24 10:06:01,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418000 2023-11-24 10:06:12,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2786673.3333333335, ans=0.125 2023-11-24 10:06:17,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2786740.0, ans=0.125 2023-11-24 10:06:18,318 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9200, loss[loss=0.08711, simple_loss=0.1149, pruned_loss=0.01834, audio_tagging_loss=0.01133, over 14094.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09212, pruned_loss=0.01319, audio_tagging_loss=0.008671, over 3056982.85 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:06:26,307 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2786740.0, ans=0.04949747468305833 2023-11-24 10:07:04,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418050 2023-11-24 10:07:13,498 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 10:07:20,420 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9250, loss[loss=0.07241, simple_loss=0.1014, pruned_loss=0.01509, audio_tagging_loss=0.00661, over 15832.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09201, pruned_loss=0.01338, audio_tagging_loss=0.008773, over 3059819.21 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:07:25,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.69 vs. limit=22.5 2023-11-24 10:07:38,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=12.0 2023-11-24 10:07:39,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2787140.0, ans=0.125 2023-11-24 10:07:43,437 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.443e+01 8.965e+01 9.843e+01 1.266e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 10:07:54,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2787206.6666666665, ans=0.125 2023-11-24 10:08:06,262 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418100 2023-11-24 10:08:16,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2787340.0, ans=15.0 2023-11-24 10:08:22,361 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9300, loss[loss=0.08167, simple_loss=0.1145, pruned_loss=0.01781, audio_tagging_loss=0.006589, over 14884.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09175, pruned_loss=0.01329, audio_tagging_loss=0.008784, over 3061009.91 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:08:24,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2787406.6666666665, ans=0.125 2023-11-24 10:08:30,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0 2023-11-24 10:08:37,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2787473.3333333335, ans=0.0 2023-11-24 10:08:42,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.73 vs. limit=15.0 2023-11-24 10:09:03,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2787606.6666666665, ans=0.0 2023-11-24 10:09:07,272 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2787606.6666666665, ans=0.1 2023-11-24 10:09:08,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418150 2023-11-24 10:09:08,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2787606.6666666665, ans=0.125 2023-11-24 10:09:23,909 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9350, loss[loss=0.06463, simple_loss=0.08497, pruned_loss=0.01011, audio_tagging_loss=0.01204, over 15028.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09203, pruned_loss=0.01342, audio_tagging_loss=0.008811, over 3063884.93 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:09:26,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2787740.0, ans=0.125 2023-11-24 10:09:37,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2787806.6666666665, ans=0.125 2023-11-24 10:09:37,984 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0 2023-11-24 10:09:47,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2787873.3333333335, ans=0.125 2023-11-24 10:09:48,172 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.653e+01 8.452e+01 8.975e+01 9.526e+01 2.573e+02, threshold=1.795e+02, percent-clipped=1.0 2023-11-24 10:10:09,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418200 2023-11-24 10:10:15,927 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2788006.6666666665, ans=0.125 2023-11-24 10:10:26,244 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9400, loss[loss=0.0728, simple_loss=0.09875, pruned_loss=0.01573, audio_tagging_loss=0.00769, over 15256.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09235, pruned_loss=0.01363, audio_tagging_loss=0.008876, over 3063126.10 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:10:27,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2788073.3333333335, ans=0.125 2023-11-24 10:10:39,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2788140.0, ans=0.1 2023-11-24 10:10:57,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2788206.6666666665, ans=0.125 2023-11-24 10:11:08,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2788273.3333333335, ans=0.0 2023-11-24 10:11:12,715 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418250 2023-11-24 10:11:25,991 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:11:28,919 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9450, loss[loss=0.06589, simple_loss=0.09624, pruned_loss=0.009433, audio_tagging_loss=0.008336, over 15005.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09143, pruned_loss=0.01342, audio_tagging_loss=0.00901, over 3055826.04 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:11:54,121 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.573e+01 9.142e+01 1.013e+02 1.307e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 10:12:00,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2788540.0, ans=0.125 2023-11-24 10:12:09,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2788606.6666666665, ans=0.1 2023-11-24 10:12:14,912 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418300 2023-11-24 10:12:20,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2788673.3333333335, ans=0.0 2023-11-24 10:12:21,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2788673.3333333335, ans=0.125 2023-11-24 10:12:30,832 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9500, loss[loss=0.07587, simple_loss=0.1053, pruned_loss=0.01315, audio_tagging_loss=0.01006, over 15153.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09171, pruned_loss=0.01345, audio_tagging_loss=0.008969, over 3062308.20 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:13:01,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2788873.3333333335, ans=0.0 2023-11-24 10:13:08,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.34 vs. limit=15.0 2023-11-24 10:13:12,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2788940.0, ans=0.0 2023-11-24 10:13:17,155 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418350 2023-11-24 10:13:23,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2789006.6666666665, ans=0.0 2023-11-24 10:13:29,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2789006.6666666665, ans=0.1 2023-11-24 10:13:34,257 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9550, loss[loss=0.0847, simple_loss=0.1153, pruned_loss=0.01698, audio_tagging_loss=0.01008, over 17145.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.092, pruned_loss=0.01335, audio_tagging_loss=0.009028, over 3060934.86 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:13:42,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2789073.3333333335, ans=15.0 2023-11-24 10:13:58,339 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.515e+01 8.574e+01 9.293e+01 1.001e+02 1.205e+02, threshold=1.859e+02, percent-clipped=0.0 2023-11-24 10:14:20,452 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418400 2023-11-24 10:14:24,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2789340.0, ans=0.125 2023-11-24 10:14:36,069 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9600, loss[loss=0.07296, simple_loss=0.0981, pruned_loss=0.0138, audio_tagging_loss=0.01012, over 14254.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09169, pruned_loss=0.01319, audio_tagging_loss=0.009104, over 3055425.08 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:14:38,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2789406.6666666665, ans=0.125 2023-11-24 10:14:47,043 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.77 vs. limit=15.0 2023-11-24 10:14:50,266 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.26 vs. limit=15.0 2023-11-24 10:14:57,805 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.45 vs. limit=15.0 2023-11-24 10:15:10,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2789540.0, ans=0.0 2023-11-24 10:15:14,931 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.81 vs. limit=22.5 2023-11-24 10:15:22,487 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418450 2023-11-24 10:15:38,343 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9650, loss[loss=0.05836, simple_loss=0.07323, pruned_loss=0.01168, audio_tagging_loss=0.01006, over 16107.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09093, pruned_loss=0.01319, audio_tagging_loss=0.009048, over 3060368.80 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:16:04,130 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.403e+01 8.395e+01 8.995e+01 9.967e+01 1.274e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-24 10:16:06,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2789873.3333333335, ans=0.035 2023-11-24 10:16:25,166 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418500 2023-11-24 10:16:26,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2789940.0, ans=0.125 2023-11-24 10:16:29,540 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2790006.6666666665, ans=0.125 2023-11-24 10:16:34,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2790006.6666666665, ans=0.125 2023-11-24 10:16:42,583 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9700, loss[loss=0.05696, simple_loss=0.06742, pruned_loss=0.01208, audio_tagging_loss=0.01117, over 15413.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09188, pruned_loss=0.01336, audio_tagging_loss=0.008936, over 3052956.40 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:16:49,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2790073.3333333335, ans=0.125 2023-11-24 10:17:00,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2790140.0, ans=10.0 2023-11-24 10:17:03,918 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2790140.0, ans=0.0 2023-11-24 10:17:06,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2790206.6666666665, ans=0.125 2023-11-24 10:17:20,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2790273.3333333335, ans=0.125 2023-11-24 10:17:27,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418550 2023-11-24 10:17:34,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2790340.0, ans=0.04949747468305833 2023-11-24 10:17:43,646 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9750, loss[loss=0.07737, simple_loss=0.1059, pruned_loss=0.01625, audio_tagging_loss=0.008178, over 14860.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09155, pruned_loss=0.01317, audio_tagging_loss=0.008841, over 3047403.82 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:17:54,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2790473.3333333335, ans=0.04949747468305833 2023-11-24 10:18:08,323 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.283e+01 8.345e+01 8.930e+01 9.592e+01 1.228e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 10:18:08,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2790540.0, ans=0.1 2023-11-24 10:18:29,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418600 2023-11-24 10:18:33,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2790673.3333333335, ans=0.0 2023-11-24 10:18:35,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2790673.3333333335, ans=0.0 2023-11-24 10:18:45,251 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9800, loss[loss=0.06842, simple_loss=0.09202, pruned_loss=0.01346, audio_tagging_loss=0.008951, over 13889.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.0917, pruned_loss=0.01312, audio_tagging_loss=0.00875, over 3045681.07 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:18:45,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2790740.0, ans=0.125 2023-11-24 10:18:53,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2790740.0, ans=0.1 2023-11-24 10:19:17,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2790873.3333333335, ans=0.1 2023-11-24 10:19:31,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418650 2023-11-24 10:19:40,631 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:19:48,361 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9850, loss[loss=0.06927, simple_loss=0.08082, pruned_loss=0.01684, audio_tagging_loss=0.01202, over 14520.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09134, pruned_loss=0.0132, audio_tagging_loss=0.008795, over 3048080.13 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:19:51,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2791073.3333333335, ans=0.125 2023-11-24 10:20:12,660 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.621e+01 9.085e+01 9.764e+01 1.357e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 10:20:22,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=12.61 vs. limit=15.0 2023-11-24 10:20:23,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2791273.3333333335, ans=0.2 2023-11-24 10:20:34,233 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418700 2023-11-24 10:20:50,735 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9900, loss[loss=0.06566, simple_loss=0.08451, pruned_loss=0.01257, audio_tagging_loss=0.01083, over 14721.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.0917, pruned_loss=0.01315, audio_tagging_loss=0.008751, over 3039323.04 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:20:51,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2791406.6666666665, ans=0.0 2023-11-24 10:20:55,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2791406.6666666665, ans=0.2 2023-11-24 10:21:02,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2791473.3333333335, ans=0.2 2023-11-24 10:21:36,776 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418750 2023-11-24 10:21:38,506 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.08 vs. limit=15.0 2023-11-24 10:21:40,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2791673.3333333335, ans=0.125 2023-11-24 10:21:45,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2791673.3333333335, ans=0.125 2023-11-24 10:21:52,132 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 9950, loss[loss=0.06954, simple_loss=0.09144, pruned_loss=0.01634, audio_tagging_loss=0.00748, over 14642.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09137, pruned_loss=0.01314, audio_tagging_loss=0.008712, over 3038668.75 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:22:01,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2791740.0, ans=0.2 2023-11-24 10:22:02,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2791740.0, ans=0.125 2023-11-24 10:22:18,402 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.444e+01 9.082e+01 9.757e+01 1.187e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-24 10:22:37,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2791940.0, ans=0.125 2023-11-24 10:22:39,138 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418800 2023-11-24 10:22:39,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2791940.0, ans=0.125 2023-11-24 10:22:39,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2791940.0, ans=0.0 2023-11-24 10:22:40,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2791940.0, ans=0.0 2023-11-24 10:22:55,824 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10000, loss[loss=0.07788, simple_loss=0.1051, pruned_loss=0.01736, audio_tagging_loss=0.007964, over 15591.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09094, pruned_loss=0.01313, audio_tagging_loss=0.008766, over 3035580.02 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:22:57,847 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.03 vs. limit=15.0 2023-11-24 10:23:21,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2792206.6666666665, ans=15.0 2023-11-24 10:23:25,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2792206.6666666665, ans=0.1 2023-11-24 10:23:27,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2792206.6666666665, ans=0.2 2023-11-24 10:23:28,915 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.71 vs. limit=15.0 2023-11-24 10:23:42,087 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418850 2023-11-24 10:23:48,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2792340.0, ans=0.125 2023-11-24 10:23:59,892 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10050, loss[loss=0.07437, simple_loss=0.1056, pruned_loss=0.01491, audio_tagging_loss=0.00665, over 14594.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.091, pruned_loss=0.01308, audio_tagging_loss=0.008806, over 3036402.28 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:24:08,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2792406.6666666665, ans=0.0 2023-11-24 10:24:18,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2792473.3333333335, ans=0.0 2023-11-24 10:24:20,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2792473.3333333335, ans=0.2 2023-11-24 10:24:25,781 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.576e+01 9.106e+01 9.929e+01 1.520e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 10:24:29,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2792540.0, ans=0.1 2023-11-24 10:24:44,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2792606.6666666665, ans=0.125 2023-11-24 10:24:47,103 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418900 2023-11-24 10:24:56,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2792673.3333333335, ans=0.0 2023-11-24 10:25:02,630 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10100, loss[loss=0.07535, simple_loss=0.1102, pruned_loss=0.01323, audio_tagging_loss=0.007047, over 16448.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09171, pruned_loss=0.01312, audio_tagging_loss=0.008811, over 3050550.96 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:25:07,972 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.99 vs. limit=10.0 2023-11-24 10:25:15,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2792806.6666666665, ans=0.125 2023-11-24 10:25:46,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2792940.0, ans=0.2 2023-11-24 10:25:47,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2792940.0, ans=0.0 2023-11-24 10:25:48,983 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 418950 2023-11-24 10:25:51,300 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:25:52,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2793006.6666666665, ans=0.125 2023-11-24 10:25:52,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2793006.6666666665, ans=0.125 2023-11-24 10:25:53,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2793006.6666666665, ans=0.0 2023-11-24 10:25:54,190 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.78 vs. limit=15.0 2023-11-24 10:26:04,195 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2793073.3333333335, ans=0.2 2023-11-24 10:26:05,133 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10150, loss[loss=0.07391, simple_loss=0.1008, pruned_loss=0.01536, audio_tagging_loss=0.008147, over 15843.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09195, pruned_loss=0.01316, audio_tagging_loss=0.008867, over 3056094.49 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:26:24,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2793140.0, ans=0.125 2023-11-24 10:26:28,186 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.60 vs. limit=22.5 2023-11-24 10:26:31,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.136e+01 8.585e+01 9.069e+01 9.759e+01 1.192e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 10:26:33,580 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:26:39,015 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.01 vs. limit=15.0 2023-11-24 10:26:41,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2793273.3333333335, ans=0.125 2023-11-24 10:26:43,334 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2793273.3333333335, ans=0.0 2023-11-24 10:26:45,310 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.09 vs. limit=6.0 2023-11-24 10:26:50,937 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419000 2023-11-24 10:26:51,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2793273.3333333335, ans=0.1 2023-11-24 10:27:08,905 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10200, loss[loss=0.07887, simple_loss=0.1207, pruned_loss=0.01241, audio_tagging_loss=0.006114, over 16096.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09217, pruned_loss=0.0132, audio_tagging_loss=0.008888, over 3061816.42 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:27:30,994 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:27:39,956 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2793540.0, ans=0.125 2023-11-24 10:27:46,491 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2793606.6666666665, ans=0.0 2023-11-24 10:27:51,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2793606.6666666665, ans=0.0 2023-11-24 10:27:55,676 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419050 2023-11-24 10:28:03,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2793673.3333333335, ans=0.1 2023-11-24 10:28:10,841 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10250, loss[loss=0.06292, simple_loss=0.08224, pruned_loss=0.01272, audio_tagging_loss=0.009076, over 15843.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09153, pruned_loss=0.01311, audio_tagging_loss=0.008989, over 3064390.26 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:28:37,728 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.437e+01 9.083e+01 9.683e+01 1.798e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 10:28:43,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2793873.3333333335, ans=0.125 2023-11-24 10:28:44,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2793873.3333333335, ans=0.125 2023-11-24 10:28:45,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2793873.3333333335, ans=0.1 2023-11-24 10:28:57,519 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419100 2023-11-24 10:28:58,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2793940.0, ans=0.0 2023-11-24 10:29:13,539 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10300, loss[loss=0.05744, simple_loss=0.07183, pruned_loss=0.01071, audio_tagging_loss=0.01082, over 15189.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.091, pruned_loss=0.01306, audio_tagging_loss=0.009144, over 3060970.83 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:29:15,006 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2794073.3333333335, ans=0.125 2023-11-24 10:29:44,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2794206.6666666665, ans=0.125 2023-11-24 10:29:44,551 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.45 vs. limit=22.5 2023-11-24 10:30:00,351 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419150 2023-11-24 10:30:06,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2794340.0, ans=0.125 2023-11-24 10:30:13,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2794340.0, ans=0.0 2023-11-24 10:30:16,903 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10350, loss[loss=0.05999, simple_loss=0.08079, pruned_loss=0.01038, audio_tagging_loss=0.00922, over 14403.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09164, pruned_loss=0.01319, audio_tagging_loss=0.009163, over 3056668.70 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:30:17,534 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.67 vs. limit=15.0 2023-11-24 10:30:20,059 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0 2023-11-24 10:30:37,777 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 10:30:41,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2794540.0, ans=0.125 2023-11-24 10:30:42,292 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.615e+01 9.196e+01 1.008e+02 1.353e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 10:30:55,335 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.18 vs. limit=15.0 2023-11-24 10:31:03,024 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419200 2023-11-24 10:31:19,394 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10400, loss[loss=0.06464, simple_loss=0.09337, pruned_loss=0.009627, audio_tagging_loss=0.008331, over 14896.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09122, pruned_loss=0.01326, audio_tagging_loss=0.009229, over 3053710.85 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:32:05,615 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419250 2023-11-24 10:32:21,635 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10450, loss[loss=0.06658, simple_loss=0.0978, pruned_loss=0.01221, audio_tagging_loss=0.005472, over 15836.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09065, pruned_loss=0.01326, audio_tagging_loss=0.00917, over 3048991.70 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:32:38,404 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 10:32:49,180 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.462e+01 9.061e+01 9.658e+01 1.208e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 10:32:53,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2795206.6666666665, ans=0.125 2023-11-24 10:32:57,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2795273.3333333335, ans=0.125 2023-11-24 10:33:07,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419300 2023-11-24 10:33:24,650 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10500, loss[loss=0.05577, simple_loss=0.07798, pruned_loss=0.008881, audio_tagging_loss=0.007903, over 14782.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09056, pruned_loss=0.01328, audio_tagging_loss=0.009048, over 3044291.91 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:33:34,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2795406.6666666665, ans=0.125 2023-11-24 10:33:40,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2795473.3333333335, ans=22.5 2023-11-24 10:33:43,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2795473.3333333335, ans=0.0 2023-11-24 10:34:02,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2795606.6666666665, ans=0.2 2023-11-24 10:34:10,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2795606.6666666665, ans=0.125 2023-11-24 10:34:11,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419350 2023-11-24 10:34:19,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2795673.3333333335, ans=0.0 2023-11-24 10:34:20,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2795673.3333333335, ans=0.1 2023-11-24 10:34:20,673 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.63 vs. limit=22.5 2023-11-24 10:34:27,463 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10550, loss[loss=0.07232, simple_loss=0.09538, pruned_loss=0.01571, audio_tagging_loss=0.00892, over 15151.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09014, pruned_loss=0.01315, audio_tagging_loss=0.008919, over 3049834.61 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:34:31,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2795740.0, ans=0.125 2023-11-24 10:34:45,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2795806.6666666665, ans=0.0 2023-11-24 10:34:45,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2795806.6666666665, ans=0.125 2023-11-24 10:34:48,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2795806.6666666665, ans=0.125 2023-11-24 10:34:54,569 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.441e+01 8.781e+01 9.299e+01 1.006e+02 1.266e+02, threshold=1.860e+02, percent-clipped=0.0 2023-11-24 10:34:55,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.55 vs. limit=15.0 2023-11-24 10:34:56,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.25 vs. limit=12.0 2023-11-24 10:35:03,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2795940.0, ans=0.0 2023-11-24 10:35:13,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419400 2023-11-24 10:35:20,336 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.40 vs. limit=22.5 2023-11-24 10:35:28,605 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.05 vs. limit=15.0 2023-11-24 10:35:29,252 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10600, loss[loss=0.06143, simple_loss=0.07944, pruned_loss=0.01308, audio_tagging_loss=0.008632, over 13713.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09095, pruned_loss=0.01327, audio_tagging_loss=0.008862, over 3042633.66 frames. ], batch size: 52, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:35:36,461 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.48 vs. limit=15.0 2023-11-24 10:35:37,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2796073.3333333335, ans=0.2 2023-11-24 10:35:56,567 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.41 vs. limit=6.0 2023-11-24 10:35:59,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.65 vs. limit=15.0 2023-11-24 10:36:15,416 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419450 2023-11-24 10:36:32,458 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10650, loss[loss=0.06609, simple_loss=0.08664, pruned_loss=0.01434, audio_tagging_loss=0.008432, over 15687.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09081, pruned_loss=0.01317, audio_tagging_loss=0.008886, over 3050692.25 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:36:43,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2796473.3333333335, ans=0.0 2023-11-24 10:36:47,862 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.07 vs. limit=22.5 2023-11-24 10:36:49,288 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.03 vs. limit=15.0 2023-11-24 10:36:59,550 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.535e+01 9.205e+01 9.977e+01 1.279e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-24 10:37:02,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2796540.0, ans=0.2 2023-11-24 10:37:14,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2796606.6666666665, ans=0.125 2023-11-24 10:37:15,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2796606.6666666665, ans=0.125 2023-11-24 10:37:18,823 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419500 2023-11-24 10:37:34,638 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10700, loss[loss=0.06138, simple_loss=0.08602, pruned_loss=0.01015, audio_tagging_loss=0.008217, over 14873.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09134, pruned_loss=0.01321, audio_tagging_loss=0.008847, over 3054654.37 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:37:34,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2796740.0, ans=0.125 2023-11-24 10:38:14,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2796940.0, ans=0.125 2023-11-24 10:38:21,133 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419550 2023-11-24 10:38:21,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2796940.0, ans=0.125 2023-11-24 10:38:28,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2797006.6666666665, ans=0.125 2023-11-24 10:38:37,181 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10750, loss[loss=0.07563, simple_loss=0.1006, pruned_loss=0.01526, audio_tagging_loss=0.01007, over 15049.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09098, pruned_loss=0.01332, audio_tagging_loss=0.008843, over 3051572.37 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0 2023-11-24 10:38:38,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2797073.3333333335, ans=0.125 2023-11-24 10:38:41,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2797073.3333333335, ans=0.0 2023-11-24 10:38:50,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.14 vs. limit=15.0 2023-11-24 10:39:00,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2797140.0, ans=0.125 2023-11-24 10:39:05,290 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.423e+01 9.138e+01 9.778e+01 1.311e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 10:39:12,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.74 vs. limit=15.0 2023-11-24 10:39:23,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2797273.3333333335, ans=0.125 2023-11-24 10:39:24,015 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419600 2023-11-24 10:39:40,621 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10800, loss[loss=0.07973, simple_loss=0.1098, pruned_loss=0.01745, audio_tagging_loss=0.007387, over 15434.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09045, pruned_loss=0.01315, audio_tagging_loss=0.008892, over 3048725.58 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0 2023-11-24 10:39:59,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2797473.3333333335, ans=0.125 2023-11-24 10:40:19,678 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.10 vs. limit=12.0 2023-11-24 10:40:21,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2797606.6666666665, ans=0.125 2023-11-24 10:40:26,285 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419650 2023-11-24 10:40:42,946 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10850, loss[loss=0.04747, simple_loss=0.05903, pruned_loss=0.008745, audio_tagging_loss=0.009215, over 15044.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09135, pruned_loss=0.01325, audio_tagging_loss=0.008826, over 3051721.04 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:40:48,658 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.39 vs. limit=5.0 2023-11-24 10:40:50,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2797740.0, ans=0.2 2023-11-24 10:41:09,947 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.591e+01 9.237e+01 1.014e+02 1.325e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 10:41:21,028 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 10:41:29,166 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419700 2023-11-24 10:41:39,724 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:41:44,409 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10900, loss[loss=0.06178, simple_loss=0.08585, pruned_loss=0.01188, audio_tagging_loss=0.006976, over 13732.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09106, pruned_loss=0.01316, audio_tagging_loss=0.008843, over 3048424.82 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:41:55,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2798073.3333333335, ans=0.1 2023-11-24 10:42:22,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2798273.3333333335, ans=0.125 2023-11-24 10:42:30,850 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419750 2023-11-24 10:42:47,318 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 10950, loss[loss=0.04803, simple_loss=0.06412, pruned_loss=0.006964, audio_tagging_loss=0.009001, over 16367.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.0907, pruned_loss=0.0132, audio_tagging_loss=0.008946, over 3051259.97 frames. ], batch size: 64, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:43:14,656 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.263e+01 9.239e+01 9.893e+01 2.160e+02, threshold=1.848e+02, percent-clipped=1.0 2023-11-24 10:43:33,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419800 2023-11-24 10:43:35,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2798606.6666666665, ans=0.125 2023-11-24 10:43:50,664 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11000, loss[loss=0.06504, simple_loss=0.08431, pruned_loss=0.01375, audio_tagging_loss=0.009132, over 14651.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09065, pruned_loss=0.01318, audio_tagging_loss=0.00898, over 3046746.49 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:43:52,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2798740.0, ans=0.09899494936611666 2023-11-24 10:43:58,923 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:44:12,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2798806.6666666665, ans=0.125 2023-11-24 10:44:16,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2798873.3333333335, ans=0.0 2023-11-24 10:44:31,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=2798940.0, ans=15.0 2023-11-24 10:44:35,208 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.90 vs. limit=10.0 2023-11-24 10:44:37,149 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419850 2023-11-24 10:44:40,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2799006.6666666665, ans=0.125 2023-11-24 10:44:52,417 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11050, loss[loss=0.05266, simple_loss=0.06589, pruned_loss=0.01176, audio_tagging_loss=0.007951, over 14690.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09055, pruned_loss=0.01312, audio_tagging_loss=0.009013, over 3050524.93 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 16.0 2023-11-24 10:45:07,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2799140.0, ans=0.125 2023-11-24 10:45:12,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2799140.0, ans=0.125 2023-11-24 10:45:21,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2799206.6666666665, ans=0.125 2023-11-24 10:45:21,957 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.396e+01 8.466e+01 9.230e+01 9.960e+01 1.939e+02, threshold=1.846e+02, percent-clipped=1.0 2023-11-24 10:45:38,870 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419900 2023-11-24 10:45:44,276 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=12.0 2023-11-24 10:45:55,572 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11100, loss[loss=0.06994, simple_loss=0.08814, pruned_loss=0.01251, audio_tagging_loss=0.01336, over 15320.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09025, pruned_loss=0.01302, audio_tagging_loss=0.009141, over 3051012.83 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 16.0 2023-11-24 10:46:26,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2799540.0, ans=0.125 2023-11-24 10:46:33,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2799606.6666666665, ans=0.0 2023-11-24 10:46:40,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2799606.6666666665, ans=0.125 2023-11-24 10:46:41,214 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 419950 2023-11-24 10:46:41,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2799606.6666666665, ans=0.125 2023-11-24 10:46:45,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2799673.3333333335, ans=0.09899494936611666 2023-11-24 10:46:53,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2799673.3333333335, ans=0.125 2023-11-24 10:46:58,127 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11150, loss[loss=0.07265, simple_loss=0.1089, pruned_loss=0.0124, audio_tagging_loss=0.005826, over 15323.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.0898, pruned_loss=0.01297, audio_tagging_loss=0.009215, over 3046946.11 frames. ], batch size: 54, lr: 1.92e-03, grad_scale: 16.0 2023-11-24 10:46:58,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2799740.0, ans=0.09899494936611666 2023-11-24 10:47:00,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2799740.0, ans=0.125 2023-11-24 10:47:26,048 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.463e+01 9.073e+01 9.645e+01 1.170e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-24 10:47:34,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2799940.0, ans=10.0 2023-11-24 10:47:44,400 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420000 2023-11-24 10:47:46,373 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=15.33 vs. limit=15.0 2023-11-24 10:47:51,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2800006.6666666665, ans=0.125 2023-11-24 10:48:00,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.55 vs. limit=10.0 2023-11-24 10:48:03,621 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11200, loss[loss=0.06891, simple_loss=0.09235, pruned_loss=0.014, audio_tagging_loss=0.008731, over 15179.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.08954, pruned_loss=0.0128, audio_tagging_loss=0.009256, over 3049801.61 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:48:12,229 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2800073.3333333335, ans=0.125 2023-11-24 10:48:42,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2800273.3333333335, ans=0.125 2023-11-24 10:48:44,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2800273.3333333335, ans=0.0 2023-11-24 10:48:50,452 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420050 2023-11-24 10:48:58,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2800340.0, ans=0.1 2023-11-24 10:49:04,854 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0 2023-11-24 10:49:06,598 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11250, loss[loss=0.05818, simple_loss=0.08538, pruned_loss=0.008853, audio_tagging_loss=0.006637, over 16215.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.08914, pruned_loss=0.01269, audio_tagging_loss=0.009201, over 3052207.91 frames. ], batch size: 60, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:49:15,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2800406.6666666665, ans=0.125 2023-11-24 10:49:20,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2800473.3333333335, ans=0.1 2023-11-24 10:49:26,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2800473.3333333335, ans=0.125 2023-11-24 10:49:29,014 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.07 vs. limit=12.0 2023-11-24 10:49:35,711 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.479e+01 9.057e+01 9.752e+01 1.909e+02, threshold=1.811e+02, percent-clipped=1.0 2023-11-24 10:49:37,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2800540.0, ans=0.125 2023-11-24 10:49:52,808 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420100 2023-11-24 10:50:09,692 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11300, loss[loss=0.06553, simple_loss=0.09169, pruned_loss=0.01246, audio_tagging_loss=0.00722, over 15171.00 frames. ], tot_loss[loss=0.06643, simple_loss=0.08939, pruned_loss=0.01263, audio_tagging_loss=0.009107, over 3054826.36 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:50:11,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2800740.0, ans=0.125 2023-11-24 10:50:40,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2800873.3333333335, ans=0.07 2023-11-24 10:50:43,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2800873.3333333335, ans=0.1 2023-11-24 10:50:55,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420150 2023-11-24 10:51:09,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2801073.3333333335, ans=0.1 2023-11-24 10:51:10,850 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11350, loss[loss=0.09428, simple_loss=0.1396, pruned_loss=0.01977, audio_tagging_loss=0.00473, over 16099.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09011, pruned_loss=0.01289, audio_tagging_loss=0.009029, over 3057919.06 frames. ], batch size: 59, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:51:12,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2801073.3333333335, ans=0.1 2023-11-24 10:51:14,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2801073.3333333335, ans=0.05 2023-11-24 10:51:20,423 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2801073.3333333335, ans=0.0 2023-11-24 10:51:40,024 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.605e+01 8.545e+01 9.244e+01 9.787e+01 1.195e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-24 10:51:47,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2801273.3333333335, ans=10.0 2023-11-24 10:51:56,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420200 2023-11-24 10:52:08,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2801340.0, ans=0.125 2023-11-24 10:52:09,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2801340.0, ans=0.125 2023-11-24 10:52:13,281 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11400, loss[loss=0.05944, simple_loss=0.08489, pruned_loss=0.01118, audio_tagging_loss=0.005814, over 15232.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09069, pruned_loss=0.01309, audio_tagging_loss=0.008899, over 3046514.22 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:52:23,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2801406.6666666665, ans=0.125 2023-11-24 10:52:29,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2801473.3333333335, ans=0.125 2023-11-24 10:52:30,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2801473.3333333335, ans=0.125 2023-11-24 10:52:33,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2801473.3333333335, ans=0.1 2023-11-24 10:52:43,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2801540.0, ans=0.125 2023-11-24 10:52:52,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2801606.6666666665, ans=0.125 2023-11-24 10:53:00,380 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420250 2023-11-24 10:53:08,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2801673.3333333335, ans=0.1 2023-11-24 10:53:10,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2801673.3333333335, ans=0.0 2023-11-24 10:53:17,795 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11450, loss[loss=0.07557, simple_loss=0.1013, pruned_loss=0.01404, audio_tagging_loss=0.01088, over 16139.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09112, pruned_loss=0.01326, audio_tagging_loss=0.008938, over 3046242.95 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 16.0 2023-11-24 10:53:21,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2801740.0, ans=0.07 2023-11-24 10:53:46,378 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.482e+01 8.489e+01 9.216e+01 1.000e+02 1.338e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 10:53:48,944 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.0 2023-11-24 10:54:03,652 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420300 2023-11-24 10:54:06,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2802006.6666666665, ans=0.1 2023-11-24 10:54:19,459 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11500, loss[loss=0.07165, simple_loss=0.09412, pruned_loss=0.0146, audio_tagging_loss=0.009997, over 14026.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09124, pruned_loss=0.01343, audio_tagging_loss=0.008824, over 3047678.36 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 16.0 2023-11-24 10:54:24,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2802073.3333333335, ans=0.2 2023-11-24 10:55:05,372 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420350 2023-11-24 10:55:20,739 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11550, loss[loss=0.07977, simple_loss=0.1021, pruned_loss=0.02011, audio_tagging_loss=0.008593, over 15038.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09139, pruned_loss=0.0135, audio_tagging_loss=0.008832, over 3045991.09 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 16.0 2023-11-24 10:55:50,992 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.625e+01 9.353e+01 9.944e+01 1.426e+02, threshold=1.871e+02, percent-clipped=0.0 2023-11-24 10:55:52,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2802540.0, ans=0.125 2023-11-24 10:55:52,573 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2802540.0, ans=0.1 2023-11-24 10:55:56,967 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 10:56:06,449 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420400 2023-11-24 10:56:22,931 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11600, loss[loss=0.0675, simple_loss=0.09812, pruned_loss=0.01174, audio_tagging_loss=0.006697, over 15048.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09074, pruned_loss=0.01323, audio_tagging_loss=0.008889, over 3047724.95 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:56:29,109 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 10:56:29,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2802740.0, ans=0.2 2023-11-24 10:56:33,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2802740.0, ans=0.0 2023-11-24 10:56:39,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.54 vs. limit=15.0 2023-11-24 10:56:50,010 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.71 vs. limit=15.0 2023-11-24 10:56:54,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2802873.3333333335, ans=0.125 2023-11-24 10:56:58,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2802940.0, ans=0.1 2023-11-24 10:57:03,993 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.81 vs. limit=6.0 2023-11-24 10:57:04,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2802940.0, ans=0.1 2023-11-24 10:57:07,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2802940.0, ans=0.125 2023-11-24 10:57:08,661 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420450 2023-11-24 10:57:13,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2803006.6666666665, ans=0.0 2023-11-24 10:57:20,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2803006.6666666665, ans=0.125 2023-11-24 10:57:24,773 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11650, loss[loss=0.05522, simple_loss=0.07771, pruned_loss=0.009115, audio_tagging_loss=0.007251, over 16205.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09076, pruned_loss=0.01317, audio_tagging_loss=0.00893, over 3046333.60 frames. ], batch size: 62, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:57:47,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.10 vs. limit=15.0 2023-11-24 10:57:47,428 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.18 vs. limit=12.0 2023-11-24 10:57:52,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2803206.6666666665, ans=0.125 2023-11-24 10:57:54,135 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.520e+01 9.268e+01 1.018e+02 1.416e+02, threshold=1.854e+02, percent-clipped=0.0 2023-11-24 10:58:03,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2803273.3333333335, ans=0.125 2023-11-24 10:58:10,510 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420500 2023-11-24 10:58:17,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2803340.0, ans=0.125 2023-11-24 10:58:25,802 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11700, loss[loss=0.07326, simple_loss=0.108, pruned_loss=0.01264, audio_tagging_loss=0.006605, over 15389.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09077, pruned_loss=0.01314, audio_tagging_loss=0.008965, over 3044417.63 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:58:33,658 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 10:58:34,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.70 vs. limit=15.0 2023-11-24 10:58:41,468 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2803473.3333333335, ans=0.1 2023-11-24 10:59:01,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2803540.0, ans=0.125 2023-11-24 10:59:01,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2803540.0, ans=0.05 2023-11-24 10:59:09,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2803606.6666666665, ans=0.0 2023-11-24 10:59:11,891 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420550 2023-11-24 10:59:13,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2803606.6666666665, ans=0.125 2023-11-24 10:59:16,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2803673.3333333335, ans=0.2 2023-11-24 10:59:28,514 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11750, loss[loss=0.06986, simple_loss=0.09352, pruned_loss=0.01477, audio_tagging_loss=0.008329, over 15380.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09181, pruned_loss=0.01332, audio_tagging_loss=0.008991, over 3042962.77 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 10:59:30,305 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.92 vs. limit=15.0 2023-11-24 10:59:53,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2803873.3333333335, ans=10.0 2023-11-24 10:59:56,960 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.83 vs. limit=15.0 2023-11-24 10:59:57,408 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.255e+01 8.962e+01 9.569e+01 1.238e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-24 11:00:04,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2803940.0, ans=0.125 2023-11-24 11:00:04,552 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.30 vs. limit=22.5 2023-11-24 11:00:13,383 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420600 2023-11-24 11:00:29,781 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11800, loss[loss=0.05357, simple_loss=0.07111, pruned_loss=0.008594, audio_tagging_loss=0.009421, over 15333.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09121, pruned_loss=0.01316, audio_tagging_loss=0.009046, over 3042065.47 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 11:00:58,721 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:01:16,278 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420650 2023-11-24 11:01:17,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2804273.3333333335, ans=0.125 2023-11-24 11:01:32,365 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11850, loss[loss=0.08812, simple_loss=0.1231, pruned_loss=0.01697, audio_tagging_loss=0.009612, over 15563.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09182, pruned_loss=0.01337, audio_tagging_loss=0.009116, over 3043634.56 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 11:02:02,504 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.352e+01 9.070e+01 9.952e+01 1.461e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 11:02:05,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2804540.0, ans=0.2 2023-11-24 11:02:14,413 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.85 vs. limit=22.5 2023-11-24 11:02:18,529 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420700 2023-11-24 11:02:34,399 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11900, loss[loss=0.06028, simple_loss=0.0815, pruned_loss=0.008047, audio_tagging_loss=0.01149, over 14941.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09098, pruned_loss=0.01312, audio_tagging_loss=0.009222, over 3041621.58 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 11:02:37,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2804740.0, ans=0.125 2023-11-24 11:02:45,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2804740.0, ans=0.1 2023-11-24 11:03:02,400 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:03:17,547 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.84 vs. limit=15.0 2023-11-24 11:03:20,559 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420750 2023-11-24 11:03:21,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2804940.0, ans=0.125 2023-11-24 11:03:21,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2804940.0, ans=0.1 2023-11-24 11:03:37,210 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 11950, loss[loss=0.06246, simple_loss=0.08722, pruned_loss=0.01026, audio_tagging_loss=0.008587, over 14597.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09134, pruned_loss=0.01325, audio_tagging_loss=0.009239, over 3045089.94 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 11:03:41,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2805073.3333333335, ans=0.2 2023-11-24 11:04:03,820 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:04:03,867 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:04:07,049 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.367e+01 9.062e+01 9.589e+01 1.237e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 11:04:11,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2805206.6666666665, ans=0.1 2023-11-24 11:04:22,831 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420800 2023-11-24 11:04:37,833 INFO [train_asr.py:1221] (3/4) Epoch 35, batch 12000, loss[loss=0.07541, simple_loss=0.09688, pruned_loss=0.01635, audio_tagging_loss=0.01062, over 14094.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09106, pruned_loss=0.0133, audio_tagging_loss=0.009346, over 3039178.71 frames. ], batch size: 53, lr: 1.92e-03, grad_scale: 32.0 2023-11-24 11:04:37,834 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 11:05:08,662 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9606, 3.8279, 4.8567, 4.5382], device='cuda:3') 2023-11-24 11:05:19,858 INFO [train_asr.py:1253] (3/4) Epoch 35, validation: loss=0.0585, simple_loss=0.05078, pruned_loss=0.005085, audio_tagging_loss=0.02803, over 4681554.00 frames. 2023-11-24 11:05:19,858 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 11:05:22,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2805406.6666666665, ans=0.125 2023-11-24 11:05:31,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2805473.3333333335, ans=0.125 2023-11-24 11:05:34,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.27 vs. limit=15.0 2023-11-24 11:06:25,324 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 0, loss[loss=0.1019, simple_loss=0.1206, pruned_loss=0.02283, audio_tagging_loss=0.01876, over 16137.00 frames. ], tot_loss[loss=0.1019, simple_loss=0.1206, pruned_loss=0.02283, audio_tagging_loss=0.01876, over 16137.00 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:06:25,325 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 11:06:44,432 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.7355, 4.8324, 4.8694, 4.7868], device='cuda:3') 2023-11-24 11:07:04,354 INFO [train_asr.py:1253] (3/4) Epoch 36, validation: loss=0.05761, simple_loss=0.0508, pruned_loss=0.005078, audio_tagging_loss=0.02713, over 4681554.00 frames. 2023-11-24 11:07:04,355 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 11:07:08,691 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.48 vs. limit=22.5 2023-11-24 11:07:22,241 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420850 2023-11-24 11:07:59,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2805820.0, ans=0.0 2023-11-24 11:08:06,409 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 50, loss[loss=0.0772, simple_loss=0.07686, pruned_loss=0.01627, audio_tagging_loss=0.0225, over 14491.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09122, pruned_loss=0.01304, audio_tagging_loss=0.01738, over 695716.44 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:08:07,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2805886.6666666665, ans=0.125 2023-11-24 11:08:09,899 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.471e+01 9.077e+01 9.826e+01 1.067e+02 1.319e+02, threshold=1.965e+02, percent-clipped=0.0 2023-11-24 11:08:16,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2805886.6666666665, ans=0.09899494936611666 2023-11-24 11:08:19,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2805953.3333333335, ans=0.0 2023-11-24 11:08:24,312 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2805953.3333333335, ans=0.1 2023-11-24 11:08:25,427 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420900 2023-11-24 11:08:40,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2806020.0, ans=0.125 2023-11-24 11:08:44,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2806086.6666666665, ans=0.2 2023-11-24 11:08:46,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2806086.6666666665, ans=0.0 2023-11-24 11:09:08,041 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 100, loss[loss=0.0634, simple_loss=0.07203, pruned_loss=0.0124, audio_tagging_loss=0.01499, over 14510.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09178, pruned_loss=0.01334, audio_tagging_loss=0.01657, over 1219496.92 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:09:09,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2806220.0, ans=0.125 2023-11-24 11:09:12,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2806220.0, ans=0.04949747468305833 2023-11-24 11:09:25,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2806286.6666666665, ans=0.125 2023-11-24 11:09:27,617 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 420950 2023-11-24 11:09:27,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2806286.6666666665, ans=0.125 2023-11-24 11:09:27,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2806286.6666666665, ans=0.95 2023-11-24 11:09:38,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2806353.3333333335, ans=0.0 2023-11-24 11:09:52,953 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.70 vs. limit=22.5 2023-11-24 11:10:02,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2806486.6666666665, ans=0.1 2023-11-24 11:10:02,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.39 vs. limit=15.0 2023-11-24 11:10:05,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2806486.6666666665, ans=0.2 2023-11-24 11:10:06,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2806486.6666666665, ans=0.09899494936611666 2023-11-24 11:10:11,661 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 150, loss[loss=0.07591, simple_loss=0.1069, pruned_loss=0.01428, audio_tagging_loss=0.008171, over 14494.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.0928, pruned_loss=0.01359, audio_tagging_loss=0.01486, over 1633547.76 frames. ], batch size: 52, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:10:15,191 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 8.014e+01 9.024e+01 9.585e+01 1.046e+02 2.115e+02, threshold=1.917e+02, percent-clipped=1.0 2023-11-24 11:10:22,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2806620.0, ans=0.125 2023-11-24 11:10:29,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421000 2023-11-24 11:10:33,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2806620.0, ans=0.0 2023-11-24 11:11:04,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2806820.0, ans=0.125 2023-11-24 11:11:05,749 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.92 vs. limit=6.0 2023-11-24 11:11:13,394 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 200, loss[loss=0.06711, simple_loss=0.08771, pruned_loss=0.01482, audio_tagging_loss=0.008437, over 15419.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09207, pruned_loss=0.01349, audio_tagging_loss=0.01317, over 1951041.10 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:11:31,238 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421050 2023-11-24 11:11:59,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2807086.6666666665, ans=0.125 2023-11-24 11:12:14,920 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 250, loss[loss=0.07285, simple_loss=0.09695, pruned_loss=0.01589, audio_tagging_loss=0.00848, over 15148.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09184, pruned_loss=0.01328, audio_tagging_loss=0.01187, over 2196254.06 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:12:18,499 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.652e+01 9.389e+01 9.985e+01 1.276e+02, threshold=1.878e+02, percent-clipped=0.0 2023-11-24 11:12:19,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2807220.0, ans=0.125 2023-11-24 11:12:34,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421100 2023-11-24 11:12:41,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2807353.3333333335, ans=0.125 2023-11-24 11:12:47,757 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:13:18,278 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 300, loss[loss=0.0554, simple_loss=0.06819, pruned_loss=0.0112, audio_tagging_loss=0.01011, over 14791.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09216, pruned_loss=0.01322, audio_tagging_loss=0.01105, over 2387207.33 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:13:19,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2807553.3333333335, ans=0.1 2023-11-24 11:13:23,494 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.68 vs. limit=12.0 2023-11-24 11:13:32,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2807620.0, ans=0.125 2023-11-24 11:13:36,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421150 2023-11-24 11:13:41,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2807686.6666666665, ans=0.1 2023-11-24 11:13:42,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.21 vs. limit=10.0 2023-11-24 11:13:43,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2807686.6666666665, ans=0.0 2023-11-24 11:14:02,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2807753.3333333335, ans=0.125 2023-11-24 11:14:04,498 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.94 vs. limit=15.0 2023-11-24 11:14:11,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2807820.0, ans=0.1 2023-11-24 11:14:19,894 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 350, loss[loss=0.05173, simple_loss=0.06585, pruned_loss=0.00833, audio_tagging_loss=0.01048, over 14396.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09281, pruned_loss=0.01337, audio_tagging_loss=0.01036, over 2540473.21 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:14:23,374 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.480e+01 9.093e+01 9.744e+01 1.336e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 11:14:26,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.40 vs. limit=22.5 2023-11-24 11:14:32,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2807953.3333333335, ans=0.125 2023-11-24 11:14:38,010 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421200 2023-11-24 11:14:55,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2808020.0, ans=0.0 2023-11-24 11:15:07,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2808086.6666666665, ans=0.0 2023-11-24 11:15:21,385 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 400, loss[loss=0.08537, simple_loss=0.1297, pruned_loss=0.01592, audio_tagging_loss=0.004616, over 16478.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09348, pruned_loss=0.01353, audio_tagging_loss=0.0099, over 2651288.40 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:15:22,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2808220.0, ans=0.125 2023-11-24 11:15:24,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2808220.0, ans=0.2 2023-11-24 11:15:38,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2808286.6666666665, ans=0.125 2023-11-24 11:15:41,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421250 2023-11-24 11:15:43,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2808286.6666666665, ans=0.07 2023-11-24 11:15:49,183 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.68 vs. limit=15.0 2023-11-24 11:16:06,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2808420.0, ans=0.0 2023-11-24 11:16:23,733 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 450, loss[loss=0.0618, simple_loss=0.0873, pruned_loss=0.009925, audio_tagging_loss=0.008221, over 15642.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09244, pruned_loss=0.01341, audio_tagging_loss=0.009673, over 2738264.28 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:16:27,924 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.299e+01 9.052e+01 9.986e+01 1.410e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 11:16:33,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2808553.3333333335, ans=0.125 2023-11-24 11:16:38,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2808620.0, ans=0.1 2023-11-24 11:16:41,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2808620.0, ans=0.125 2023-11-24 11:16:42,692 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421300 2023-11-24 11:16:46,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2808620.0, ans=0.0 2023-11-24 11:16:51,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2808686.6666666665, ans=0.125 2023-11-24 11:17:04,466 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.44 vs. limit=15.0 2023-11-24 11:17:05,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2808753.3333333335, ans=0.125 2023-11-24 11:17:09,622 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.80 vs. limit=15.0 2023-11-24 11:17:16,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2808820.0, ans=0.125 2023-11-24 11:17:23,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2808820.0, ans=0.2 2023-11-24 11:17:26,683 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 500, loss[loss=0.0558, simple_loss=0.07588, pruned_loss=0.007882, audio_tagging_loss=0.009974, over 15844.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.0922, pruned_loss=0.01333, audio_tagging_loss=0.00947, over 2806513.62 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:17:34,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2808886.6666666665, ans=0.125 2023-11-24 11:17:44,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421350 2023-11-24 11:18:12,430 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.28 vs. limit=15.0 2023-11-24 11:18:27,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2809220.0, ans=0.2 2023-11-24 11:18:27,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2809220.0, ans=0.1 2023-11-24 11:18:28,542 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 550, loss[loss=0.07298, simple_loss=0.1088, pruned_loss=0.01025, audio_tagging_loss=0.00831, over 15177.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09215, pruned_loss=0.01308, audio_tagging_loss=0.009397, over 2859280.14 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:18:32,036 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.126e+01 8.386e+01 8.873e+01 9.796e+01 1.246e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-24 11:18:46,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2809286.6666666665, ans=10.0 2023-11-24 11:18:47,309 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421400 2023-11-24 11:19:09,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2809420.0, ans=0.0 2023-11-24 11:19:14,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2809420.0, ans=0.05 2023-11-24 11:19:27,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2809486.6666666665, ans=0.125 2023-11-24 11:19:31,279 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 600, loss[loss=0.06205, simple_loss=0.08386, pruned_loss=0.01151, audio_tagging_loss=0.008609, over 14385.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09268, pruned_loss=0.0132, audio_tagging_loss=0.009274, over 2898241.11 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:19:32,790 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:19:36,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2809553.3333333335, ans=0.125 2023-11-24 11:19:46,107 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=12.0 2023-11-24 11:19:50,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421450 2023-11-24 11:19:54,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0 2023-11-24 11:20:00,588 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.97 vs. limit=15.0 2023-11-24 11:20:09,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2809753.3333333335, ans=0.125 2023-11-24 11:20:15,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2809753.3333333335, ans=0.2 2023-11-24 11:20:33,881 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 650, loss[loss=0.06093, simple_loss=0.08176, pruned_loss=0.01139, audio_tagging_loss=0.008668, over 14729.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09239, pruned_loss=0.0132, audio_tagging_loss=0.009288, over 2927318.53 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:20:37,353 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.541e+01 9.260e+01 1.007e+02 1.291e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 11:20:38,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2809886.6666666665, ans=0.125 2023-11-24 11:20:51,724 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421500 2023-11-24 11:20:54,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2809953.3333333335, ans=0.1 2023-11-24 11:20:59,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=2810020.0, ans=0.02 2023-11-24 11:21:04,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2810020.0, ans=0.05 2023-11-24 11:21:22,892 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.83 vs. limit=15.0 2023-11-24 11:21:35,323 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 700, loss[loss=0.05445, simple_loss=0.07606, pruned_loss=0.01026, audio_tagging_loss=0.006154, over 15089.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09241, pruned_loss=0.01321, audio_tagging_loss=0.009109, over 2961739.92 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:21:54,206 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421550 2023-11-24 11:22:01,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2810353.3333333335, ans=15.0 2023-11-24 11:22:13,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2810420.0, ans=0.125 2023-11-24 11:22:32,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2810486.6666666665, ans=0.125 2023-11-24 11:22:37,461 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 750, loss[loss=0.07111, simple_loss=0.1053, pruned_loss=0.01178, audio_tagging_loss=0.006691, over 14806.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.0922, pruned_loss=0.01331, audio_tagging_loss=0.00914, over 2985464.63 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:22:41,365 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.522e+01 9.060e+01 9.732e+01 1.237e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 11:22:41,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2810553.3333333335, ans=0.2 2023-11-24 11:22:42,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2810553.3333333335, ans=0.125 2023-11-24 11:22:49,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2810620.0, ans=0.125 2023-11-24 11:22:51,095 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.69 vs. limit=15.0 2023-11-24 11:22:55,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2810620.0, ans=0.2 2023-11-24 11:22:56,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421600 2023-11-24 11:23:05,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2810686.6666666665, ans=0.1 2023-11-24 11:23:19,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2810753.3333333335, ans=0.125 2023-11-24 11:23:23,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2810753.3333333335, ans=0.025 2023-11-24 11:23:26,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2810820.0, ans=0.0 2023-11-24 11:23:29,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2810820.0, ans=0.125 2023-11-24 11:23:31,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2810820.0, ans=0.125 2023-11-24 11:23:40,411 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 800, loss[loss=0.05777, simple_loss=0.07992, pruned_loss=0.007523, audio_tagging_loss=0.01028, over 15098.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09291, pruned_loss=0.01347, audio_tagging_loss=0.009181, over 2995561.18 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:23:45,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2810886.6666666665, ans=0.125 2023-11-24 11:23:58,870 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421650 2023-11-24 11:23:59,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2810953.3333333335, ans=0.125 2023-11-24 11:24:18,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.84 vs. limit=15.0 2023-11-24 11:24:20,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2811086.6666666665, ans=0.125 2023-11-24 11:24:33,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2811153.3333333335, ans=0.125 2023-11-24 11:24:36,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.80 vs. limit=22.5 2023-11-24 11:24:41,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2811220.0, ans=0.1 2023-11-24 11:24:42,438 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 850, loss[loss=0.0729, simple_loss=0.1027, pruned_loss=0.013, audio_tagging_loss=0.008578, over 16428.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09277, pruned_loss=0.01341, audio_tagging_loss=0.009274, over 3008293.09 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:24:47,194 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.597e+01 8.758e+01 9.246e+01 1.015e+02 2.108e+02, threshold=1.849e+02, percent-clipped=1.0 2023-11-24 11:25:01,344 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421700 2023-11-24 11:25:22,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2811420.0, ans=0.1 2023-11-24 11:25:45,248 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 900, loss[loss=0.07079, simple_loss=0.09289, pruned_loss=0.01675, audio_tagging_loss=0.007592, over 15499.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09224, pruned_loss=0.01337, audio_tagging_loss=0.009314, over 3019367.32 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:25:52,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2811553.3333333335, ans=0.125 2023-11-24 11:25:57,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2811620.0, ans=0.2 2023-11-24 11:25:59,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2811620.0, ans=10.0 2023-11-24 11:26:02,415 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.37 vs. limit=15.0 2023-11-24 11:26:04,200 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421750 2023-11-24 11:26:11,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2811686.6666666665, ans=0.125 2023-11-24 11:26:15,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0 2023-11-24 11:26:20,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2811686.6666666665, ans=0.0 2023-11-24 11:26:25,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0 2023-11-24 11:26:47,794 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 950, loss[loss=0.06133, simple_loss=0.07595, pruned_loss=0.009225, audio_tagging_loss=0.01413, over 15819.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09178, pruned_loss=0.01328, audio_tagging_loss=0.009288, over 3020351.58 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:26:52,397 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.504e+01 9.122e+01 9.832e+01 1.245e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-24 11:27:00,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2811953.3333333335, ans=0.125 2023-11-24 11:27:06,051 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421800 2023-11-24 11:27:19,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2812020.0, ans=0.0 2023-11-24 11:27:41,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2812153.3333333335, ans=0.125 2023-11-24 11:27:42,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2812153.3333333335, ans=0.05 2023-11-24 11:27:48,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.56 vs. limit=15.0 2023-11-24 11:27:49,565 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1000, loss[loss=0.06171, simple_loss=0.08412, pruned_loss=0.01056, audio_tagging_loss=0.009095, over 14413.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09244, pruned_loss=0.0134, audio_tagging_loss=0.009003, over 3031660.16 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:27:51,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.09 vs. limit=15.0 2023-11-24 11:28:08,621 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421850 2023-11-24 11:28:15,018 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 11:28:21,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2812353.3333333335, ans=0.125 2023-11-24 11:28:25,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2812353.3333333335, ans=0.0 2023-11-24 11:28:48,281 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:28:51,785 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1050, loss[loss=0.08208, simple_loss=0.1087, pruned_loss=0.0189, audio_tagging_loss=0.00882, over 14325.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09142, pruned_loss=0.01315, audio_tagging_loss=0.009009, over 3032955.16 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:28:57,202 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.446e+01 9.134e+01 9.863e+01 1.540e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 11:29:09,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2812620.0, ans=0.1 2023-11-24 11:29:11,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421900 2023-11-24 11:29:55,478 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1100, loss[loss=0.07202, simple_loss=0.09742, pruned_loss=0.01616, audio_tagging_loss=0.007148, over 15215.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09089, pruned_loss=0.01308, audio_tagging_loss=0.008925, over 3030614.20 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:29:57,914 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 11:30:13,235 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 421950 2023-11-24 11:30:50,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2813153.3333333335, ans=0.125 2023-11-24 11:30:56,387 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1150, loss[loss=0.07753, simple_loss=0.106, pruned_loss=0.01833, audio_tagging_loss=0.006211, over 15452.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09085, pruned_loss=0.01311, audio_tagging_loss=0.008863, over 3034316.58 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:31:00,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2813220.0, ans=0.125 2023-11-24 11:31:00,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2813220.0, ans=0.0 2023-11-24 11:31:01,046 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.442e+01 9.003e+01 9.685e+01 1.147e+02, threshold=1.801e+02, percent-clipped=0.0 2023-11-24 11:31:09,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2813286.6666666665, ans=0.0 2023-11-24 11:31:15,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422000 2023-11-24 11:31:18,314 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2813286.6666666665, ans=0.125 2023-11-24 11:31:26,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.63 vs. limit=15.0 2023-11-24 11:31:47,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2813486.6666666665, ans=0.0 2023-11-24 11:31:58,438 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1200, loss[loss=0.06544, simple_loss=0.07357, pruned_loss=0.01346, audio_tagging_loss=0.01519, over 15690.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09134, pruned_loss=0.01334, audio_tagging_loss=0.008932, over 3034978.42 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:32:17,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422050 2023-11-24 11:32:32,030 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:32:40,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2813753.3333333335, ans=0.1 2023-11-24 11:33:00,659 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1250, loss[loss=0.05875, simple_loss=0.07896, pruned_loss=0.01224, audio_tagging_loss=0.007031, over 15671.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09098, pruned_loss=0.0133, audio_tagging_loss=0.008864, over 3040026.94 frames. ], batch size: 63, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:33:07,043 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.529e+01 8.565e+01 9.272e+01 1.023e+02 1.182e+02, threshold=1.854e+02, percent-clipped=0.0 2023-11-24 11:33:15,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2813953.3333333335, ans=0.125 2023-11-24 11:33:18,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422100 2023-11-24 11:33:22,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2813953.3333333335, ans=0.125 2023-11-24 11:33:31,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2814020.0, ans=0.0 2023-11-24 11:33:45,111 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2814086.6666666665, ans=0.2 2023-11-24 11:33:55,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.76 vs. limit=15.0 2023-11-24 11:33:56,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2814153.3333333335, ans=0.0 2023-11-24 11:34:01,841 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1300, loss[loss=0.06219, simple_loss=0.09495, pruned_loss=0.008319, audio_tagging_loss=0.006401, over 15089.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09036, pruned_loss=0.01325, audio_tagging_loss=0.008911, over 3038685.95 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:34:19,687 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422150 2023-11-24 11:34:27,128 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=15.0 2023-11-24 11:35:04,217 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1350, loss[loss=0.05379, simple_loss=0.07305, pruned_loss=0.00979, audio_tagging_loss=0.00747, over 15125.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.0904, pruned_loss=0.01318, audio_tagging_loss=0.008879, over 3040105.26 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:35:04,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2814553.3333333335, ans=0.1 2023-11-24 11:35:10,807 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.582e+01 9.240e+01 9.907e+01 1.176e+02, threshold=1.848e+02, percent-clipped=0.0 2023-11-24 11:35:14,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2814553.3333333335, ans=0.125 2023-11-24 11:35:24,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422200 2023-11-24 11:35:29,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2814686.6666666665, ans=0.1 2023-11-24 11:35:40,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2814686.6666666665, ans=0.0 2023-11-24 11:35:48,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2814753.3333333335, ans=0.025 2023-11-24 11:35:49,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.35 vs. limit=22.5 2023-11-24 11:35:49,792 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 11:36:01,510 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:36:08,389 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1400, loss[loss=0.05694, simple_loss=0.07134, pruned_loss=0.008926, audio_tagging_loss=0.01234, over 15254.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09002, pruned_loss=0.01304, audio_tagging_loss=0.008992, over 3041960.73 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:36:17,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2814886.6666666665, ans=0.0 2023-11-24 11:36:24,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2814953.3333333335, ans=0.125 2023-11-24 11:36:25,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2814953.3333333335, ans=0.07 2023-11-24 11:36:26,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422250 2023-11-24 11:36:30,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2814953.3333333335, ans=0.0 2023-11-24 11:37:09,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2815220.0, ans=0.1 2023-11-24 11:37:10,642 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1450, loss[loss=0.07443, simple_loss=0.09109, pruned_loss=0.01659, audio_tagging_loss=0.01229, over 16554.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09088, pruned_loss=0.0131, audio_tagging_loss=0.009004, over 3044383.28 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:37:16,394 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.536e+01 9.281e+01 1.016e+02 1.384e+02, threshold=1.856e+02, percent-clipped=0.0 2023-11-24 11:37:20,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2815220.0, ans=0.0 2023-11-24 11:37:22,674 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 11:37:23,009 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.52 vs. limit=6.0 2023-11-24 11:37:28,228 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422300 2023-11-24 11:37:58,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2815486.6666666665, ans=0.5 2023-11-24 11:38:09,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2815486.6666666665, ans=0.125 2023-11-24 11:38:11,604 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1500, loss[loss=0.05914, simple_loss=0.07707, pruned_loss=0.01185, audio_tagging_loss=0.008751, over 16925.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09085, pruned_loss=0.01304, audio_tagging_loss=0.009097, over 3050008.97 frames. ], batch size: 67, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:38:18,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2815553.3333333335, ans=0.0 2023-11-24 11:38:21,904 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2815553.3333333335, ans=0.1 2023-11-24 11:38:24,367 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2815620.0, ans=0.125 2023-11-24 11:38:27,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2815620.0, ans=0.0 2023-11-24 11:38:31,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422350 2023-11-24 11:38:34,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2815620.0, ans=0.2 2023-11-24 11:38:46,680 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.55 vs. limit=15.0 2023-11-24 11:38:55,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2815753.3333333335, ans=0.0 2023-11-24 11:38:57,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.44 vs. limit=22.5 2023-11-24 11:39:13,898 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1550, loss[loss=0.06961, simple_loss=0.09274, pruned_loss=0.01108, audio_tagging_loss=0.01215, over 15716.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.0905, pruned_loss=0.01307, audio_tagging_loss=0.009158, over 3051655.37 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:39:20,780 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.494e+01 9.231e+01 9.838e+01 1.696e+02, threshold=1.846e+02, percent-clipped=0.0 2023-11-24 11:39:26,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2815953.3333333335, ans=0.1 2023-11-24 11:39:32,707 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422400 2023-11-24 11:39:34,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2815953.3333333335, ans=0.125 2023-11-24 11:39:45,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2816020.0, ans=0.125 2023-11-24 11:39:57,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2816086.6666666665, ans=0.0 2023-11-24 11:39:59,098 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2816086.6666666665, ans=0.125 2023-11-24 11:40:02,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2816086.6666666665, ans=0.0 2023-11-24 11:40:17,314 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1600, loss[loss=0.05742, simple_loss=0.06869, pruned_loss=0.01218, audio_tagging_loss=0.01088, over 15225.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09014, pruned_loss=0.01283, audio_tagging_loss=0.009133, over 3061052.85 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:40:35,169 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422450 2023-11-24 11:40:40,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2816353.3333333335, ans=0.125 2023-11-24 11:40:50,064 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.58 vs. limit=15.0 2023-11-24 11:40:54,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.33 vs. limit=15.0 2023-11-24 11:41:11,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2816486.6666666665, ans=0.2 2023-11-24 11:41:18,985 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1650, loss[loss=0.04918, simple_loss=0.05946, pruned_loss=0.0102, audio_tagging_loss=0.009253, over 14251.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.0899, pruned_loss=0.01285, audio_tagging_loss=0.009145, over 3059749.89 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:41:23,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2816553.3333333335, ans=0.125 2023-11-24 11:41:25,964 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.620e+01 9.407e+01 1.041e+02 1.351e+02, threshold=1.881e+02, percent-clipped=0.0 2023-11-24 11:41:37,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422500 2023-11-24 11:42:12,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2816820.0, ans=0.125 2023-11-24 11:42:20,967 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1700, loss[loss=0.06405, simple_loss=0.07317, pruned_loss=0.01692, audio_tagging_loss=0.01054, over 14904.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09114, pruned_loss=0.01303, audio_tagging_loss=0.009024, over 3062782.62 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:42:23,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2816886.6666666665, ans=0.0 2023-11-24 11:42:38,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2816953.3333333335, ans=0.125 2023-11-24 11:42:40,530 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422550 2023-11-24 11:42:59,076 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.55 vs. limit=6.0 2023-11-24 11:42:59,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2817086.6666666665, ans=0.5 2023-11-24 11:43:22,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2817153.3333333335, ans=0.125 2023-11-24 11:43:24,391 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1750, loss[loss=0.08162, simple_loss=0.1156, pruned_loss=0.01654, audio_tagging_loss=0.007267, over 15235.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.0905, pruned_loss=0.01309, audio_tagging_loss=0.008942, over 3051362.65 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:43:28,370 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2817220.0, ans=0.2 2023-11-24 11:43:31,547 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.461e+01 8.743e+01 9.239e+01 9.812e+01 1.237e+02, threshold=1.848e+02, percent-clipped=0.0 2023-11-24 11:43:31,870 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2817220.0, ans=0.125 2023-11-24 11:43:35,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.31 vs. limit=15.0 2023-11-24 11:43:42,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422600 2023-11-24 11:43:51,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2817353.3333333335, ans=0.125 2023-11-24 11:44:04,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2817420.0, ans=0.125 2023-11-24 11:44:12,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2817420.0, ans=0.125 2023-11-24 11:44:20,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2817486.6666666665, ans=0.1 2023-11-24 11:44:26,825 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1800, loss[loss=0.07914, simple_loss=0.1119, pruned_loss=0.01271, audio_tagging_loss=0.01049, over 15743.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09203, pruned_loss=0.01335, audio_tagging_loss=0.008831, over 3050690.38 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:44:30,898 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.68 vs. limit=15.0 2023-11-24 11:44:45,808 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422650 2023-11-24 11:45:01,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2817686.6666666665, ans=0.0 2023-11-24 11:45:15,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2817820.0, ans=0.2 2023-11-24 11:45:26,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2817820.0, ans=0.0 2023-11-24 11:45:28,829 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1850, loss[loss=0.05081, simple_loss=0.07256, pruned_loss=0.006477, audio_tagging_loss=0.00805, over 15858.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09172, pruned_loss=0.01323, audio_tagging_loss=0.008739, over 3054182.20 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:45:36,362 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.377e+01 8.376e+01 8.960e+01 9.835e+01 1.450e+02, threshold=1.792e+02, percent-clipped=0.0 2023-11-24 11:45:38,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2817886.6666666665, ans=0.2 2023-11-24 11:45:48,725 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422700 2023-11-24 11:46:05,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2818086.6666666665, ans=0.0 2023-11-24 11:46:06,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2818086.6666666665, ans=0.0 2023-11-24 11:46:10,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2818086.6666666665, ans=0.04949747468305833 2023-11-24 11:46:20,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2818153.3333333335, ans=0.0 2023-11-24 11:46:31,730 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1900, loss[loss=0.06804, simple_loss=0.09247, pruned_loss=0.0143, audio_tagging_loss=0.007504, over 14914.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09102, pruned_loss=0.01318, audio_tagging_loss=0.008765, over 3052714.02 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 11:46:32,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2818220.0, ans=0.95 2023-11-24 11:46:37,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2818220.0, ans=0.125 2023-11-24 11:46:41,198 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.33 vs. limit=15.0 2023-11-24 11:46:50,149 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422750 2023-11-24 11:47:08,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2818420.0, ans=0.0 2023-11-24 11:47:33,693 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 1950, loss[loss=0.06015, simple_loss=0.0716, pruned_loss=0.01336, audio_tagging_loss=0.01099, over 13427.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09129, pruned_loss=0.01331, audio_tagging_loss=0.008783, over 3046085.14 frames. ], batch size: 52, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 11:47:37,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2818553.3333333335, ans=0.125 2023-11-24 11:47:38,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2818553.3333333335, ans=0.2 2023-11-24 11:47:41,768 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.719e+01 9.311e+01 9.932e+01 1.598e+02, threshold=1.862e+02, percent-clipped=0.0 2023-11-24 11:47:45,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2818620.0, ans=0.125 2023-11-24 11:47:51,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422800 2023-11-24 11:47:56,479 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.01 vs. limit=6.0 2023-11-24 11:48:01,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2818686.6666666665, ans=0.0 2023-11-24 11:48:01,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2818686.6666666665, ans=0.125 2023-11-24 11:48:03,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2818686.6666666665, ans=0.0 2023-11-24 11:48:10,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2818753.3333333335, ans=0.125 2023-11-24 11:48:19,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2818753.3333333335, ans=0.125 2023-11-24 11:48:25,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2818820.0, ans=0.125 2023-11-24 11:48:33,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2818886.6666666665, ans=0.035 2023-11-24 11:48:34,868 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2000, loss[loss=0.06988, simple_loss=0.09633, pruned_loss=0.01195, audio_tagging_loss=0.009761, over 14779.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09188, pruned_loss=0.01342, audio_tagging_loss=0.008846, over 3043794.34 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:48:46,177 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=22.5 2023-11-24 11:48:52,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2818953.3333333335, ans=0.2 2023-11-24 11:48:54,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422850 2023-11-24 11:49:13,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2819086.6666666665, ans=0.125 2023-11-24 11:49:15,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2819086.6666666665, ans=0.125 2023-11-24 11:49:22,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2819086.6666666665, ans=0.0 2023-11-24 11:49:38,121 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2050, loss[loss=0.077, simple_loss=0.1075, pruned_loss=0.01563, audio_tagging_loss=0.007619, over 15500.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09197, pruned_loss=0.01365, audio_tagging_loss=0.008808, over 3042464.73 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:49:38,756 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.73 vs. limit=15.0 2023-11-24 11:49:45,537 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2819220.0, ans=0.125 2023-11-24 11:49:46,320 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.671e+01 8.908e+01 9.390e+01 1.017e+02 1.320e+02, threshold=1.878e+02, percent-clipped=0.0 2023-11-24 11:49:50,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2819286.6666666665, ans=0.125 2023-11-24 11:49:56,544 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422900 2023-11-24 11:50:05,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2819353.3333333335, ans=0.2 2023-11-24 11:50:13,491 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.83 vs. limit=22.5 2023-11-24 11:50:17,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.57 vs. limit=15.0 2023-11-24 11:50:20,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2819420.0, ans=0.125 2023-11-24 11:50:26,122 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=15.0 2023-11-24 11:50:40,030 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2100, loss[loss=0.06418, simple_loss=0.08589, pruned_loss=0.01053, audio_tagging_loss=0.0107, over 15272.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09257, pruned_loss=0.01359, audio_tagging_loss=0.008787, over 3048849.95 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:50:40,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2819553.3333333335, ans=0.1 2023-11-24 11:50:44,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.91 vs. limit=15.0 2023-11-24 11:50:52,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.57 vs. limit=10.0 2023-11-24 11:50:56,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2819620.0, ans=0.0 2023-11-24 11:50:58,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 422950 2023-11-24 11:51:03,906 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2819686.6666666665, ans=0.125 2023-11-24 11:51:15,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2819686.6666666665, ans=0.0 2023-11-24 11:51:30,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2819820.0, ans=0.04949747468305833 2023-11-24 11:51:30,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2819820.0, ans=0.125 2023-11-24 11:51:35,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2819820.0, ans=0.125 2023-11-24 11:51:42,206 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2150, loss[loss=0.05588, simple_loss=0.06986, pruned_loss=0.0114, audio_tagging_loss=0.00955, over 14619.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09269, pruned_loss=0.0136, audio_tagging_loss=0.008824, over 3048365.61 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:51:46,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2819886.6666666665, ans=0.2 2023-11-24 11:51:51,144 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.463e+01 9.382e+01 1.013e+02 1.297e+02, threshold=1.876e+02, percent-clipped=0.0 2023-11-24 11:52:01,975 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423000 2023-11-24 11:52:11,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2820020.0, ans=0.1 2023-11-24 11:52:18,820 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 11:52:19,049 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2820086.6666666665, ans=0.0 2023-11-24 11:52:31,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2820153.3333333335, ans=0.125 2023-11-24 11:52:34,638 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=17.89 vs. limit=15.0 2023-11-24 11:52:45,857 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2200, loss[loss=0.08651, simple_loss=0.1208, pruned_loss=0.02, audio_tagging_loss=0.00611, over 15842.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09276, pruned_loss=0.01364, audio_tagging_loss=0.008908, over 3044538.13 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:53:03,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423050 2023-11-24 11:53:13,905 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2820353.3333333335, ans=0.125 2023-11-24 11:53:16,608 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0 2023-11-24 11:53:25,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2820420.0, ans=0.1 2023-11-24 11:53:26,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2820420.0, ans=0.1 2023-11-24 11:53:40,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2820486.6666666665, ans=0.125 2023-11-24 11:53:47,546 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2250, loss[loss=0.07882, simple_loss=0.1078, pruned_loss=0.01416, audio_tagging_loss=0.01076, over 15924.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09306, pruned_loss=0.01363, audio_tagging_loss=0.008828, over 3038768.30 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:53:50,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2820553.3333333335, ans=10.0 2023-11-24 11:53:55,787 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.635e+01 8.644e+01 9.254e+01 1.001e+02 1.291e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 11:54:06,252 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423100 2023-11-24 11:54:15,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2820686.6666666665, ans=0.0 2023-11-24 11:54:16,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2820686.6666666665, ans=0.125 2023-11-24 11:54:49,846 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2300, loss[loss=0.06123, simple_loss=0.08065, pruned_loss=0.01511, audio_tagging_loss=0.005791, over 13484.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09213, pruned_loss=0.01334, audio_tagging_loss=0.008919, over 3042998.03 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:55:00,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2820886.6666666665, ans=0.05 2023-11-24 11:55:09,880 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423150 2023-11-24 11:55:10,575 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.36 vs. limit=15.0 2023-11-24 11:55:13,981 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.93 vs. limit=15.0 2023-11-24 11:55:24,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2821020.0, ans=0.125 2023-11-24 11:55:37,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2821086.6666666665, ans=0.0 2023-11-24 11:55:45,408 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 11:55:47,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2821153.3333333335, ans=0.125 2023-11-24 11:55:52,931 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2350, loss[loss=0.0781, simple_loss=0.1124, pruned_loss=0.01482, audio_tagging_loss=0.007088, over 15874.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09212, pruned_loss=0.01342, audio_tagging_loss=0.009002, over 3043585.26 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 11:55:55,527 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2821220.0, ans=0.125 2023-11-24 11:55:57,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.44 vs. limit=15.0 2023-11-24 11:56:01,693 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.333e+01 8.924e+01 9.646e+01 1.536e+02, threshold=1.785e+02, percent-clipped=0.0 2023-11-24 11:56:04,824 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.11 vs. limit=6.0 2023-11-24 11:56:11,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423200 2023-11-24 11:56:12,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2821286.6666666665, ans=0.2 2023-11-24 11:56:20,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2821353.3333333335, ans=0.2 2023-11-24 11:56:28,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.98 vs. limit=15.0 2023-11-24 11:56:40,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2821420.0, ans=0.125 2023-11-24 11:56:45,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2821486.6666666665, ans=0.125 2023-11-24 11:56:53,934 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=15.0 2023-11-24 11:56:55,523 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2400, loss[loss=0.06583, simple_loss=0.07844, pruned_loss=0.015, audio_tagging_loss=0.01162, over 14297.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09172, pruned_loss=0.01336, audio_tagging_loss=0.009121, over 3040179.70 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:56:58,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2821553.3333333335, ans=0.1 2023-11-24 11:57:13,512 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423250 2023-11-24 11:57:26,417 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.80 vs. limit=15.0 2023-11-24 11:57:36,699 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.82 vs. limit=15.0 2023-11-24 11:57:57,328 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2450, loss[loss=0.07927, simple_loss=0.1093, pruned_loss=0.01385, audio_tagging_loss=0.01075, over 15070.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09129, pruned_loss=0.01329, audio_tagging_loss=0.009174, over 3040404.49 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:58:06,732 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.416e+01 8.359e+01 8.901e+01 9.611e+01 1.267e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-24 11:58:16,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423300 2023-11-24 11:58:42,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2822086.6666666665, ans=0.125 2023-11-24 11:59:00,587 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2500, loss[loss=0.07002, simple_loss=0.09314, pruned_loss=0.01163, audio_tagging_loss=0.01182, over 14795.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09181, pruned_loss=0.01337, audio_tagging_loss=0.00921, over 3038262.92 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 11:59:18,858 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423350 2023-11-24 11:59:28,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2822353.3333333335, ans=0.125 2023-11-24 11:59:55,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2822486.6666666665, ans=0.125 2023-11-24 12:00:02,395 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2550, loss[loss=0.05015, simple_loss=0.06262, pruned_loss=0.009582, audio_tagging_loss=0.00926, over 16100.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09155, pruned_loss=0.01328, audio_tagging_loss=0.009082, over 3039693.32 frames. ], batch size: 63, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:00:10,607 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.497e+01 9.131e+01 9.713e+01 1.199e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 12:00:12,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.92 vs. limit=15.0 2023-11-24 12:00:14,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2822620.0, ans=0.0 2023-11-24 12:00:20,169 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423400 2023-11-24 12:00:45,586 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:00:55,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.82 vs. limit=10.0 2023-11-24 12:01:04,236 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2600, loss[loss=0.03843, simple_loss=0.0468, pruned_loss=0.006498, audio_tagging_loss=0.008533, over 14841.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.08927, pruned_loss=0.0129, audio_tagging_loss=0.009025, over 3034003.70 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:01:06,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2822886.6666666665, ans=0.125 2023-11-24 12:01:23,854 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423450 2023-11-24 12:01:38,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2823020.0, ans=0.1 2023-11-24 12:01:47,964 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.53 vs. limit=12.0 2023-11-24 12:02:07,941 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2650, loss[loss=0.06701, simple_loss=0.08966, pruned_loss=0.01361, audio_tagging_loss=0.008572, over 16207.00 frames. ], tot_loss[loss=0.06618, simple_loss=0.08897, pruned_loss=0.01269, audio_tagging_loss=0.009003, over 3031653.16 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:02:18,452 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.376e+01 8.976e+01 9.750e+01 1.206e+02, threshold=1.795e+02, percent-clipped=0.0 2023-11-24 12:02:27,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423500 2023-11-24 12:02:36,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2823353.3333333335, ans=0.0 2023-11-24 12:02:52,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2823420.0, ans=0.1 2023-11-24 12:03:09,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2823553.3333333335, ans=0.05 2023-11-24 12:03:10,462 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2700, loss[loss=0.07104, simple_loss=0.09812, pruned_loss=0.01263, audio_tagging_loss=0.009349, over 14894.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09007, pruned_loss=0.01294, audio_tagging_loss=0.008889, over 3038957.96 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:03:21,619 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2823620.0, ans=0.1 2023-11-24 12:03:26,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2823620.0, ans=0.125 2023-11-24 12:03:28,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423550 2023-11-24 12:03:49,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2823753.3333333335, ans=0.125 2023-11-24 12:03:51,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2823753.3333333335, ans=0.125 2023-11-24 12:03:52,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2823753.3333333335, ans=0.125 2023-11-24 12:04:11,623 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2750, loss[loss=0.06715, simple_loss=0.09887, pruned_loss=0.009678, audio_tagging_loss=0.008041, over 15487.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08928, pruned_loss=0.01277, audio_tagging_loss=0.008897, over 3039574.56 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:04:20,997 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.328e+01 8.931e+01 9.724e+01 1.117e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 12:04:30,131 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423600 2023-11-24 12:04:39,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2824020.0, ans=0.0 2023-11-24 12:05:03,811 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 12:05:14,004 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2800, loss[loss=0.07102, simple_loss=0.09688, pruned_loss=0.01134, audio_tagging_loss=0.01124, over 15302.00 frames. ], tot_loss[loss=0.06621, simple_loss=0.08907, pruned_loss=0.01269, audio_tagging_loss=0.008989, over 3044422.93 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:05:14,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2824220.0, ans=0.125 2023-11-24 12:05:16,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2824220.0, ans=0.0 2023-11-24 12:05:33,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423650 2023-11-24 12:05:44,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2824353.3333333335, ans=0.05 2023-11-24 12:05:53,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2824420.0, ans=0.05 2023-11-24 12:06:09,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2824486.6666666665, ans=0.125 2023-11-24 12:06:17,169 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2850, loss[loss=0.06217, simple_loss=0.09302, pruned_loss=0.009777, audio_tagging_loss=0.00588, over 15074.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08966, pruned_loss=0.0128, audio_tagging_loss=0.008809, over 3049307.87 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:06:17,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2824553.3333333335, ans=0.125 2023-11-24 12:06:26,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.601e+01 9.263e+01 1.001e+02 1.378e+02, threshold=1.853e+02, percent-clipped=0.0 2023-11-24 12:06:32,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2824620.0, ans=0.125 2023-11-24 12:06:35,031 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423700 2023-11-24 12:06:43,984 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.98 vs. limit=15.0 2023-11-24 12:06:51,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2824686.6666666665, ans=0.0 2023-11-24 12:07:01,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2824753.3333333335, ans=0.1 2023-11-24 12:07:02,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2824753.3333333335, ans=0.04949747468305833 2023-11-24 12:07:14,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2824820.0, ans=0.2 2023-11-24 12:07:18,751 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2900, loss[loss=0.07331, simple_loss=0.1074, pruned_loss=0.01248, audio_tagging_loss=0.007118, over 15004.00 frames. ], tot_loss[loss=0.06628, simple_loss=0.08931, pruned_loss=0.01275, audio_tagging_loss=0.008881, over 3044642.30 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:07:29,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2824953.3333333335, ans=0.125 2023-11-24 12:07:37,127 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423750 2023-11-24 12:07:53,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2825020.0, ans=0.1 2023-11-24 12:07:58,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2825086.6666666665, ans=0.125 2023-11-24 12:07:59,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2825086.6666666665, ans=0.05 2023-11-24 12:08:21,112 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 2950, loss[loss=0.06222, simple_loss=0.08313, pruned_loss=0.01258, audio_tagging_loss=0.008075, over 14676.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09012, pruned_loss=0.01289, audio_tagging_loss=0.008971, over 3044334.14 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:08:21,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2825220.0, ans=0.05 2023-11-24 12:08:31,479 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.518e+01 9.259e+01 1.017e+02 1.246e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 12:08:37,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2825286.6666666665, ans=0.0 2023-11-24 12:08:41,109 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423800 2023-11-24 12:08:53,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2825353.3333333335, ans=0.0 2023-11-24 12:09:06,730 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.82 vs. limit=12.0 2023-11-24 12:09:23,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2825553.3333333335, ans=0.125 2023-11-24 12:09:24,619 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3000, loss[loss=0.09527, simple_loss=0.1247, pruned_loss=0.02455, audio_tagging_loss=0.008352, over 15144.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09032, pruned_loss=0.01292, audio_tagging_loss=0.009043, over 3040181.10 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:09:24,620 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 12:10:06,929 INFO [train_asr.py:1253] (3/4) Epoch 36, validation: loss=0.05726, simple_loss=0.05083, pruned_loss=0.005098, audio_tagging_loss=0.02675, over 4681554.00 frames. 2023-11-24 12:10:06,930 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 12:10:18,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2825620.0, ans=0.025 2023-11-24 12:10:26,564 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423850 2023-11-24 12:10:38,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2825686.6666666665, ans=0.125 2023-11-24 12:10:56,070 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0 2023-11-24 12:11:07,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2825820.0, ans=0.125 2023-11-24 12:11:07,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2825820.0, ans=0.0 2023-11-24 12:11:09,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff3.min_abs, batch_count=2825886.6666666665, ans=0.2 2023-11-24 12:11:10,542 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3050, loss[loss=0.06764, simple_loss=0.08721, pruned_loss=0.01338, audio_tagging_loss=0.01066, over 15402.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.0913, pruned_loss=0.01319, audio_tagging_loss=0.009006, over 3037308.15 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:11:20,549 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.127e+01 8.750e+01 9.306e+01 1.016e+02 1.344e+02, threshold=1.861e+02, percent-clipped=0.0 2023-11-24 12:11:21,277 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.42 vs. limit=12.0 2023-11-24 12:11:29,552 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423900 2023-11-24 12:11:41,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2826020.0, ans=0.0 2023-11-24 12:11:44,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2826020.0, ans=0.1 2023-11-24 12:11:46,841 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 12:11:47,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2826086.6666666665, ans=0.1 2023-11-24 12:12:13,624 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3100, loss[loss=0.07235, simple_loss=0.0994, pruned_loss=0.01354, audio_tagging_loss=0.009107, over 16517.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.091, pruned_loss=0.01306, audio_tagging_loss=0.009052, over 3036102.33 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:12:22,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2826220.0, ans=0.125 2023-11-24 12:12:32,272 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 423950 2023-11-24 12:12:49,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2826353.3333333335, ans=0.125 2023-11-24 12:12:54,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2826420.0, ans=0.125 2023-11-24 12:13:16,564 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3150, loss[loss=0.07931, simple_loss=0.1082, pruned_loss=0.0154, audio_tagging_loss=0.009811, over 16172.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09068, pruned_loss=0.0128, audio_tagging_loss=0.009008, over 3041069.66 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:13:18,397 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.04 vs. limit=12.0 2023-11-24 12:13:24,208 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.71 vs. limit=15.0 2023-11-24 12:13:25,948 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.600e+01 9.080e+01 9.861e+01 1.236e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-24 12:13:34,179 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.39 vs. limit=22.5 2023-11-24 12:13:34,987 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424000 2023-11-24 12:13:57,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.55 vs. limit=10.0 2023-11-24 12:14:20,364 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:14:21,326 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3200, loss[loss=0.0678, simple_loss=0.09778, pruned_loss=0.01192, audio_tagging_loss=0.006993, over 14666.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09059, pruned_loss=0.01286, audio_tagging_loss=0.009114, over 3043068.56 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:14:21,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2826886.6666666665, ans=0.1 2023-11-24 12:14:31,334 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:14:41,500 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424050 2023-11-24 12:14:55,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2827020.0, ans=0.1 2023-11-24 12:15:11,694 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2827153.3333333335, ans=0.125 2023-11-24 12:15:11,939 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.32 vs. limit=15.0 2023-11-24 12:15:12,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2827153.3333333335, ans=0.0 2023-11-24 12:15:16,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2827153.3333333335, ans=0.0 2023-11-24 12:15:20,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2827153.3333333335, ans=0.125 2023-11-24 12:15:25,383 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3250, loss[loss=0.06567, simple_loss=0.08624, pruned_loss=0.01207, audio_tagging_loss=0.01049, over 15482.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09025, pruned_loss=0.01284, audio_tagging_loss=0.009209, over 3043363.30 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:15:29,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2827220.0, ans=0.125 2023-11-24 12:15:34,957 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.544e+01 9.313e+01 9.948e+01 1.268e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 12:15:37,570 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2827286.6666666665, ans=0.125 2023-11-24 12:15:40,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2827286.6666666665, ans=0.2 2023-11-24 12:15:41,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.29 vs. limit=22.5 2023-11-24 12:15:43,944 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424100 2023-11-24 12:15:44,429 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.78 vs. limit=15.0 2023-11-24 12:16:05,552 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2827420.0, ans=0.125 2023-11-24 12:16:11,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2827420.0, ans=0.04949747468305833 2023-11-24 12:16:24,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2827486.6666666665, ans=0.1 2023-11-24 12:16:27,557 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3300, loss[loss=0.07251, simple_loss=0.1041, pruned_loss=0.01336, audio_tagging_loss=0.007087, over 16053.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09107, pruned_loss=0.01295, audio_tagging_loss=0.009236, over 3051334.71 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:16:43,664 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2827620.0, ans=0.2 2023-11-24 12:16:46,367 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424150 2023-11-24 12:16:55,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2827686.6666666665, ans=0.07 2023-11-24 12:17:07,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2827753.3333333335, ans=0.125 2023-11-24 12:17:11,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2827753.3333333335, ans=0.125 2023-11-24 12:17:13,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2827753.3333333335, ans=0.0 2023-11-24 12:17:29,998 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3350, loss[loss=0.07591, simple_loss=0.09742, pruned_loss=0.01867, audio_tagging_loss=0.008526, over 14505.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09201, pruned_loss=0.01317, audio_tagging_loss=0.009075, over 3056099.91 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:17:35,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2827886.6666666665, ans=0.125 2023-11-24 12:17:40,622 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.571e+01 9.014e+01 9.710e+01 1.291e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-24 12:17:44,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2827953.3333333335, ans=0.0 2023-11-24 12:17:47,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2827953.3333333335, ans=0.125 2023-11-24 12:17:49,581 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424200 2023-11-24 12:17:55,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2828020.0, ans=0.95 2023-11-24 12:17:57,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2828020.0, ans=0.0 2023-11-24 12:18:04,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2828020.0, ans=0.125 2023-11-24 12:18:12,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2828086.6666666665, ans=0.0 2023-11-24 12:18:24,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2828153.3333333335, ans=0.1 2023-11-24 12:18:28,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2828153.3333333335, ans=0.125 2023-11-24 12:18:30,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.60 vs. limit=22.5 2023-11-24 12:18:31,477 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.36 vs. limit=15.0 2023-11-24 12:18:33,260 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3400, loss[loss=0.08882, simple_loss=0.1276, pruned_loss=0.01852, audio_tagging_loss=0.006503, over 15773.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09156, pruned_loss=0.01318, audio_tagging_loss=0.009057, over 3054681.09 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0 2023-11-24 12:18:40,432 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys.whitening_limit, batch_count=2828220.0, ans=6.0 2023-11-24 12:18:51,915 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424250 2023-11-24 12:18:54,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2828286.6666666665, ans=0.2 2023-11-24 12:19:03,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2828353.3333333335, ans=0.0 2023-11-24 12:19:17,772 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:19:21,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2828420.0, ans=0.1 2023-11-24 12:19:31,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2828486.6666666665, ans=0.0 2023-11-24 12:19:31,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2828486.6666666665, ans=0.2 2023-11-24 12:19:35,827 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3450, loss[loss=0.06782, simple_loss=0.08593, pruned_loss=0.01786, audio_tagging_loss=0.006997, over 14737.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09159, pruned_loss=0.01327, audio_tagging_loss=0.008967, over 3049527.41 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:19:47,009 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.469e+01 9.145e+01 9.844e+01 1.227e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 12:19:54,191 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424300 2023-11-24 12:19:56,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2828620.0, ans=0.0 2023-11-24 12:20:28,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2828820.0, ans=0.0 2023-11-24 12:20:30,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2828820.0, ans=0.0 2023-11-24 12:20:38,440 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3500, loss[loss=0.08909, simple_loss=0.1221, pruned_loss=0.01916, audio_tagging_loss=0.008856, over 16340.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09247, pruned_loss=0.01334, audio_tagging_loss=0.008846, over 3055712.92 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:20:58,026 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424350 2023-11-24 12:20:59,314 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:21:11,073 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 12:21:21,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.93 vs. limit=15.0 2023-11-24 12:21:30,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2829153.3333333335, ans=0.125 2023-11-24 12:21:37,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2829153.3333333335, ans=0.0 2023-11-24 12:21:41,492 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3550, loss[loss=0.04636, simple_loss=0.06421, pruned_loss=0.006606, audio_tagging_loss=0.007651, over 15596.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09204, pruned_loss=0.0132, audio_tagging_loss=0.008814, over 3051778.18 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:21:48,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2829220.0, ans=0.0 2023-11-24 12:21:52,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2829220.0, ans=0.1 2023-11-24 12:21:52,765 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 8.449e+01 8.981e+01 9.607e+01 1.173e+02, threshold=1.796e+02, percent-clipped=0.0 2023-11-24 12:21:55,644 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2829286.6666666665, ans=0.125 2023-11-24 12:22:00,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424400 2023-11-24 12:22:04,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2829286.6666666665, ans=10.0 2023-11-24 12:22:08,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.08 vs. limit=6.0 2023-11-24 12:22:12,539 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:22:44,315 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3600, loss[loss=0.06686, simple_loss=0.09763, pruned_loss=0.01129, audio_tagging_loss=0.006746, over 14798.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09212, pruned_loss=0.01317, audio_tagging_loss=0.008867, over 3052953.79 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:22:47,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.61 vs. limit=22.5 2023-11-24 12:23:03,161 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424450 2023-11-24 12:23:14,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2829686.6666666665, ans=0.2 2023-11-24 12:23:20,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2829686.6666666665, ans=0.0 2023-11-24 12:23:27,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2829753.3333333335, ans=0.0 2023-11-24 12:23:33,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2829820.0, ans=0.0 2023-11-24 12:23:41,228 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.36 vs. limit=15.0 2023-11-24 12:23:42,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2829820.0, ans=0.1 2023-11-24 12:23:42,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.41 vs. limit=22.5 2023-11-24 12:23:46,403 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3650, loss[loss=0.07404, simple_loss=0.1128, pruned_loss=0.01141, audio_tagging_loss=0.006236, over 15704.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09156, pruned_loss=0.01309, audio_tagging_loss=0.008758, over 3052790.88 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:23:52,326 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.10 vs. limit=15.0 2023-11-24 12:23:59,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.262e+01 8.304e+01 8.721e+01 9.501e+01 1.428e+02, threshold=1.744e+02, percent-clipped=0.0 2023-11-24 12:24:06,053 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424500 2023-11-24 12:24:08,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2829953.3333333335, ans=0.125 2023-11-24 12:24:15,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2830020.0, ans=0.2 2023-11-24 12:24:28,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2830086.6666666665, ans=0.1 2023-11-24 12:24:34,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2830086.6666666665, ans=0.125 2023-11-24 12:24:40,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2830153.3333333335, ans=0.5 2023-11-24 12:24:43,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2830153.3333333335, ans=0.1 2023-11-24 12:24:49,731 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3700, loss[loss=0.05739, simple_loss=0.06997, pruned_loss=0.01287, audio_tagging_loss=0.009541, over 15090.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09151, pruned_loss=0.01318, audio_tagging_loss=0.008776, over 3052192.54 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:24:51,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2830220.0, ans=0.1 2023-11-24 12:25:02,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2830286.6666666665, ans=0.0 2023-11-24 12:25:03,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2830286.6666666665, ans=0.2 2023-11-24 12:25:05,995 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2830286.6666666665, ans=0.125 2023-11-24 12:25:06,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2830286.6666666665, ans=0.1 2023-11-24 12:25:08,382 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424550 2023-11-24 12:25:12,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2830286.6666666665, ans=0.0 2023-11-24 12:25:22,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2830353.3333333335, ans=0.125 2023-11-24 12:25:22,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2830353.3333333335, ans=0.1 2023-11-24 12:25:50,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2830553.3333333335, ans=0.125 2023-11-24 12:25:51,800 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3750, loss[loss=0.07285, simple_loss=0.08943, pruned_loss=0.0156, audio_tagging_loss=0.01254, over 15539.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09115, pruned_loss=0.01312, audio_tagging_loss=0.008836, over 3049698.45 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:25:53,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2830553.3333333335, ans=0.0 2023-11-24 12:25:54,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2830553.3333333335, ans=0.125 2023-11-24 12:26:03,766 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.375e+01 8.897e+01 9.293e+01 9.941e+01 1.332e+02, threshold=1.859e+02, percent-clipped=0.0 2023-11-24 12:26:09,865 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424600 2023-11-24 12:26:12,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2830620.0, ans=0.0 2023-11-24 12:26:33,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2830753.3333333335, ans=0.125 2023-11-24 12:26:35,387 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 12:26:52,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2830886.6666666665, ans=0.2 2023-11-24 12:26:53,215 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3800, loss[loss=0.08858, simple_loss=0.1134, pruned_loss=0.02259, audio_tagging_loss=0.009302, over 15194.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.0908, pruned_loss=0.01312, audio_tagging_loss=0.008888, over 3049901.04 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:26:58,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2830886.6666666665, ans=0.125 2023-11-24 12:27:03,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2830886.6666666665, ans=0.125 2023-11-24 12:27:12,752 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424650 2023-11-24 12:27:19,823 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.90 vs. limit=15.0 2023-11-24 12:27:25,961 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.04 vs. limit=15.0 2023-11-24 12:27:27,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2831020.0, ans=0.125 2023-11-24 12:27:36,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2831086.6666666665, ans=0.125 2023-11-24 12:27:44,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2831153.3333333335, ans=0.125 2023-11-24 12:27:46,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2831153.3333333335, ans=0.125 2023-11-24 12:27:56,863 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3850, loss[loss=0.07028, simple_loss=0.09532, pruned_loss=0.01132, audio_tagging_loss=0.0113, over 14867.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09036, pruned_loss=0.01295, audio_tagging_loss=0.009061, over 3053904.04 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:28:00,542 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2831220.0, ans=0.1 2023-11-24 12:28:03,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2831220.0, ans=0.025 2023-11-24 12:28:09,280 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.465e+01 9.054e+01 9.529e+01 1.182e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-24 12:28:15,427 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424700 2023-11-24 12:28:16,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2831286.6666666665, ans=0.0 2023-11-24 12:28:32,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2831420.0, ans=0.125 2023-11-24 12:28:36,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2831420.0, ans=0.125 2023-11-24 12:28:41,142 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:28:42,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2831420.0, ans=0.125 2023-11-24 12:28:45,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2831486.6666666665, ans=0.1 2023-11-24 12:28:55,354 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2831486.6666666665, ans=0.125 2023-11-24 12:28:58,698 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3900, loss[loss=0.06557, simple_loss=0.09092, pruned_loss=0.01155, audio_tagging_loss=0.008558, over 14256.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09085, pruned_loss=0.0129, audio_tagging_loss=0.008985, over 3049879.96 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:28:59,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2831553.3333333335, ans=0.125 2023-11-24 12:29:16,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424750 2023-11-24 12:29:18,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2831620.0, ans=0.125 2023-11-24 12:29:18,916 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2831620.0, ans=0.0 2023-11-24 12:29:41,663 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.02 vs. limit=22.5 2023-11-24 12:29:43,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2831753.3333333335, ans=0.1 2023-11-24 12:29:45,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=13.83 vs. limit=15.0 2023-11-24 12:30:00,525 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 3950, loss[loss=0.0766, simple_loss=0.1009, pruned_loss=0.01679, audio_tagging_loss=0.009372, over 15408.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.0917, pruned_loss=0.01317, audio_tagging_loss=0.009096, over 3047055.68 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:30:06,210 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.59 vs. limit=22.5 2023-11-24 12:30:13,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2831953.3333333335, ans=0.1 2023-11-24 12:30:14,457 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.446e+01 8.514e+01 9.052e+01 9.990e+01 1.654e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 12:30:16,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2831953.3333333335, ans=0.0 2023-11-24 12:30:17,381 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:30:19,974 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424800 2023-11-24 12:30:41,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2832086.6666666665, ans=0.125 2023-11-24 12:30:44,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2832086.6666666665, ans=0.125 2023-11-24 12:31:03,779 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4000, loss[loss=0.05831, simple_loss=0.07989, pruned_loss=0.009189, audio_tagging_loss=0.009174, over 15508.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09207, pruned_loss=0.01316, audio_tagging_loss=0.009141, over 3044179.44 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:31:19,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2832286.6666666665, ans=0.125 2023-11-24 12:31:23,633 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424850 2023-11-24 12:31:30,261 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.16 vs. limit=15.0 2023-11-24 12:31:34,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2832353.3333333335, ans=0.2 2023-11-24 12:31:39,654 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.13 vs. limit=15.0 2023-11-24 12:31:51,792 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2832420.0, ans=0.2 2023-11-24 12:31:55,934 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2832486.6666666665, ans=0.125 2023-11-24 12:31:56,234 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0 2023-11-24 12:31:58,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2832486.6666666665, ans=0.125 2023-11-24 12:32:07,619 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4050, loss[loss=0.06583, simple_loss=0.09232, pruned_loss=0.01076, audio_tagging_loss=0.008905, over 14797.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09183, pruned_loss=0.01298, audio_tagging_loss=0.009166, over 3043509.04 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:32:08,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2832553.3333333335, ans=0.0 2023-11-24 12:32:11,201 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 12:32:16,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2832553.3333333335, ans=0.125 2023-11-24 12:32:21,749 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.652e+01 8.871e+01 9.409e+01 1.005e+02 1.323e+02, threshold=1.882e+02, percent-clipped=0.0 2023-11-24 12:32:25,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424900 2023-11-24 12:32:27,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2832620.0, ans=0.125 2023-11-24 12:32:39,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2832686.6666666665, ans=0.1 2023-11-24 12:32:40,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2832686.6666666665, ans=0.07 2023-11-24 12:32:43,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2832753.3333333335, ans=0.125 2023-11-24 12:32:52,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2832753.3333333335, ans=0.125 2023-11-24 12:33:09,319 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4100, loss[loss=0.06831, simple_loss=0.09598, pruned_loss=0.01372, audio_tagging_loss=0.006602, over 16706.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.0918, pruned_loss=0.01297, audio_tagging_loss=0.00921, over 3044219.54 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:33:21,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2832953.3333333335, ans=0.125 2023-11-24 12:33:28,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 424950 2023-11-24 12:33:34,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2833020.0, ans=0.0 2023-11-24 12:33:35,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2833020.0, ans=0.125 2023-11-24 12:33:41,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2833020.0, ans=0.1 2023-11-24 12:33:44,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2833020.0, ans=0.125 2023-11-24 12:33:52,866 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:34:03,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2833153.3333333335, ans=0.125 2023-11-24 12:34:05,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2833153.3333333335, ans=0.1 2023-11-24 12:34:12,400 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4150, loss[loss=0.0538, simple_loss=0.07736, pruned_loss=0.009212, audio_tagging_loss=0.005909, over 13796.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09225, pruned_loss=0.01298, audio_tagging_loss=0.009072, over 3045092.50 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:34:20,441 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2833220.0, ans=0.1 2023-11-24 12:34:27,228 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.532e+01 9.065e+01 9.755e+01 1.245e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-24 12:34:31,693 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425000 2023-11-24 12:34:32,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2833286.6666666665, ans=0.0 2023-11-24 12:34:47,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2833353.3333333335, ans=0.125 2023-11-24 12:34:57,031 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 12:35:14,961 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4200, loss[loss=0.07254, simple_loss=0.1041, pruned_loss=0.01415, audio_tagging_loss=0.006341, over 16069.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09175, pruned_loss=0.01313, audio_tagging_loss=0.008932, over 3048889.09 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:35:26,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2833553.3333333335, ans=0.2 2023-11-24 12:35:30,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2833620.0, ans=0.0 2023-11-24 12:35:34,445 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425050 2023-11-24 12:36:18,763 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4250, loss[loss=0.07457, simple_loss=0.09921, pruned_loss=0.01775, audio_tagging_loss=0.007214, over 15409.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09139, pruned_loss=0.01309, audio_tagging_loss=0.008932, over 3054138.42 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:36:22,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2833886.6666666665, ans=0.125 2023-11-24 12:36:32,904 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.335e+01 8.749e+01 9.313e+01 9.960e+01 2.008e+02, threshold=1.863e+02, percent-clipped=1.0 2023-11-24 12:36:35,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.43 vs. limit=15.0 2023-11-24 12:36:37,223 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425100 2023-11-24 12:36:44,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.92 vs. limit=15.0 2023-11-24 12:36:50,351 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.97 vs. limit=15.0 2023-11-24 12:36:55,156 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:36:58,034 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.38 vs. limit=15.0 2023-11-24 12:37:00,967 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2834086.6666666665, ans=0.125 2023-11-24 12:37:14,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2834153.3333333335, ans=0.0 2023-11-24 12:37:20,344 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.63 vs. limit=10.0 2023-11-24 12:37:20,822 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4300, loss[loss=0.06379, simple_loss=0.08906, pruned_loss=0.01004, audio_tagging_loss=0.009223, over 14790.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.0924, pruned_loss=0.0132, audio_tagging_loss=0.008808, over 3052292.75 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:37:40,032 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425150 2023-11-24 12:37:45,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2834353.3333333335, ans=0.0 2023-11-24 12:37:51,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2834353.3333333335, ans=0.2 2023-11-24 12:38:16,811 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2834486.6666666665, ans=0.125 2023-11-24 12:38:24,216 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4350, loss[loss=0.07469, simple_loss=0.1065, pruned_loss=0.01368, audio_tagging_loss=0.007749, over 15699.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09242, pruned_loss=0.01325, audio_tagging_loss=0.008775, over 3054651.08 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0 2023-11-24 12:38:38,490 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.90 vs. limit=22.5 2023-11-24 12:38:39,498 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.655e+01 9.322e+01 1.008e+02 1.169e+02, threshold=1.864e+02, percent-clipped=0.0 2023-11-24 12:38:43,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425200 2023-11-24 12:38:50,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2834686.6666666665, ans=0.0 2023-11-24 12:39:16,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2834820.0, ans=0.0 2023-11-24 12:39:27,399 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4400, loss[loss=0.06975, simple_loss=0.09779, pruned_loss=0.01233, audio_tagging_loss=0.008536, over 14690.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09212, pruned_loss=0.01331, audio_tagging_loss=0.008782, over 3048028.44 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0 2023-11-24 12:39:27,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2834886.6666666665, ans=0.0 2023-11-24 12:39:37,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2834886.6666666665, ans=10.0 2023-11-24 12:39:38,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2834953.3333333335, ans=0.125 2023-11-24 12:39:45,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425250 2023-11-24 12:39:46,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0 2023-11-24 12:39:51,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.74 vs. limit=15.0 2023-11-24 12:40:29,274 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4450, loss[loss=0.06107, simple_loss=0.08762, pruned_loss=0.009238, audio_tagging_loss=0.008018, over 14676.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09263, pruned_loss=0.01328, audio_tagging_loss=0.008779, over 3053215.02 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:40:44,625 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.474e+01 8.586e+01 9.357e+01 9.969e+01 1.625e+02, threshold=1.871e+02, percent-clipped=0.0 2023-11-24 12:40:48,587 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425300 2023-11-24 12:40:52,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2835286.6666666665, ans=0.0 2023-11-24 12:41:09,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2835420.0, ans=0.0 2023-11-24 12:41:16,157 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:41:20,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.32 vs. limit=15.0 2023-11-24 12:41:30,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2835486.6666666665, ans=0.125 2023-11-24 12:41:32,471 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4500, loss[loss=0.07207, simple_loss=0.1016, pruned_loss=0.01284, audio_tagging_loss=0.008452, over 16516.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09223, pruned_loss=0.01312, audio_tagging_loss=0.008743, over 3055902.99 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:41:40,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2835553.3333333335, ans=0.125 2023-11-24 12:41:50,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2835620.0, ans=0.07 2023-11-24 12:41:51,042 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425350 2023-11-24 12:42:12,990 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.27 vs. limit=12.0 2023-11-24 12:42:35,599 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4550, loss[loss=0.06333, simple_loss=0.09229, pruned_loss=0.009045, audio_tagging_loss=0.008144, over 15471.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09115, pruned_loss=0.01288, audio_tagging_loss=0.008794, over 3052713.35 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:42:38,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2835886.6666666665, ans=0.125 2023-11-24 12:42:44,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2835886.6666666665, ans=0.0 2023-11-24 12:42:48,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2835953.3333333335, ans=0.125 2023-11-24 12:42:50,418 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.548e+01 8.325e+01 9.085e+01 9.707e+01 1.236e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 12:42:54,121 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425400 2023-11-24 12:43:06,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2836020.0, ans=0.09899494936611666 2023-11-24 12:43:10,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2836020.0, ans=0.125 2023-11-24 12:43:12,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2836086.6666666665, ans=15.0 2023-11-24 12:43:15,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2836086.6666666665, ans=0.125 2023-11-24 12:43:23,428 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 12:43:38,253 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4600, loss[loss=0.07643, simple_loss=0.1039, pruned_loss=0.01513, audio_tagging_loss=0.009356, over 15313.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09125, pruned_loss=0.01299, audio_tagging_loss=0.00889, over 3045811.36 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:43:43,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2836220.0, ans=0.0 2023-11-24 12:43:52,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2836286.6666666665, ans=0.125 2023-11-24 12:43:57,393 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425450 2023-11-24 12:44:03,035 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.26 vs. limit=15.0 2023-11-24 12:44:04,037 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:44:36,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2836486.6666666665, ans=0.0 2023-11-24 12:44:41,248 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4650, loss[loss=0.06888, simple_loss=0.09033, pruned_loss=0.01325, audio_tagging_loss=0.01047, over 15312.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09133, pruned_loss=0.01313, audio_tagging_loss=0.008891, over 3040724.29 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:44:55,936 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.381e+01 8.503e+01 9.255e+01 1.001e+02 1.285e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 12:44:59,687 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425500 2023-11-24 12:45:17,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2836753.3333333335, ans=0.0 2023-11-24 12:45:18,518 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2836753.3333333335, ans=0.05 2023-11-24 12:45:23,674 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2836753.3333333335, ans=0.1 2023-11-24 12:45:34,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2836820.0, ans=0.0 2023-11-24 12:45:38,271 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2836820.0, ans=0.125 2023-11-24 12:45:43,857 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4700, loss[loss=0.07459, simple_loss=0.09125, pruned_loss=0.01995, audio_tagging_loss=0.009018, over 14883.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09049, pruned_loss=0.01312, audio_tagging_loss=0.009007, over 3045486.65 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:45:49,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2836886.6666666665, ans=0.125 2023-11-24 12:45:55,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2836953.3333333335, ans=0.125 2023-11-24 12:45:56,545 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0 2023-11-24 12:46:02,032 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425550 2023-11-24 12:46:09,200 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2837020.0, ans=0.125 2023-11-24 12:46:20,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2837086.6666666665, ans=0.1 2023-11-24 12:46:24,389 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.27 vs. limit=12.0 2023-11-24 12:46:34,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2837153.3333333335, ans=0.125 2023-11-24 12:46:35,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2837153.3333333335, ans=0.125 2023-11-24 12:46:42,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2837153.3333333335, ans=0.125 2023-11-24 12:46:45,790 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4750, loss[loss=0.08063, simple_loss=0.1127, pruned_loss=0.01622, audio_tagging_loss=0.00804, over 15574.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09072, pruned_loss=0.0131, audio_tagging_loss=0.009062, over 3047164.66 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:46:48,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2837220.0, ans=0.0 2023-11-24 12:46:52,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2837220.0, ans=0.07 2023-11-24 12:47:01,267 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.647e+01 9.446e+01 1.032e+02 1.298e+02, threshold=1.889e+02, percent-clipped=0.0 2023-11-24 12:47:05,672 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425600 2023-11-24 12:47:39,252 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:47:49,891 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4800, loss[loss=0.05948, simple_loss=0.0771, pruned_loss=0.01106, audio_tagging_loss=0.009875, over 15761.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09151, pruned_loss=0.01319, audio_tagging_loss=0.009047, over 3045350.14 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:48:02,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2837620.0, ans=0.0 2023-11-24 12:48:02,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2837620.0, ans=0.125 2023-11-24 12:48:07,724 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.23 vs. limit=15.0 2023-11-24 12:48:09,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425650 2023-11-24 12:48:19,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2837686.6666666665, ans=0.125 2023-11-24 12:48:29,035 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2837753.3333333335, ans=0.0 2023-11-24 12:48:31,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2837753.3333333335, ans=0.0 2023-11-24 12:48:54,401 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4850, loss[loss=0.07029, simple_loss=0.08847, pruned_loss=0.0162, audio_tagging_loss=0.009856, over 15398.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09118, pruned_loss=0.01323, audio_tagging_loss=0.009136, over 3049059.49 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:48:59,851 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.07 vs. limit=22.5 2023-11-24 12:49:08,623 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.265e+01 8.570e+01 9.281e+01 9.821e+01 1.175e+02, threshold=1.856e+02, percent-clipped=0.0 2023-11-24 12:49:12,370 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425700 2023-11-24 12:49:22,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2838020.0, ans=0.125 2023-11-24 12:49:24,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2838020.0, ans=0.125 2023-11-24 12:49:46,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2838153.3333333335, ans=0.125 2023-11-24 12:49:47,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.99 vs. limit=22.5 2023-11-24 12:49:54,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2838153.3333333335, ans=0.125 2023-11-24 12:49:56,096 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4900, loss[loss=0.05139, simple_loss=0.06804, pruned_loss=0.01077, audio_tagging_loss=0.006601, over 13903.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09117, pruned_loss=0.01323, audio_tagging_loss=0.009047, over 3042776.65 frames. ], batch size: 52, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:49:56,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2838220.0, ans=0.0 2023-11-24 12:50:15,251 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425750 2023-11-24 12:50:21,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2838353.3333333335, ans=0.1 2023-11-24 12:50:21,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2838353.3333333335, ans=0.125 2023-11-24 12:50:34,386 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2838420.0, ans=0.0 2023-11-24 12:50:41,486 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2838420.0, ans=0.125 2023-11-24 12:50:58,318 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 4950, loss[loss=0.06706, simple_loss=0.08494, pruned_loss=0.0132, audio_tagging_loss=0.01139, over 15511.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09143, pruned_loss=0.01333, audio_tagging_loss=0.008949, over 3044385.39 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:50:58,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2838553.3333333335, ans=0.0 2023-11-24 12:51:14,271 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 8.587e+01 9.264e+01 9.833e+01 1.255e+02, threshold=1.853e+02, percent-clipped=0.0 2023-11-24 12:51:17,907 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425800 2023-11-24 12:51:30,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2838686.6666666665, ans=0.2 2023-11-24 12:51:30,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2838686.6666666665, ans=0.125 2023-11-24 12:51:41,565 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.65 vs. limit=10.0 2023-11-24 12:51:42,749 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.76 vs. limit=15.0 2023-11-24 12:52:02,012 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5000, loss[loss=0.06787, simple_loss=0.09593, pruned_loss=0.01303, audio_tagging_loss=0.006878, over 14426.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09031, pruned_loss=0.01311, audio_tagging_loss=0.008857, over 3046165.69 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:52:04,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2838886.6666666665, ans=0.0 2023-11-24 12:52:12,419 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.46 vs. limit=15.0 2023-11-24 12:52:19,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425850 2023-11-24 12:52:35,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2839020.0, ans=0.0 2023-11-24 12:52:45,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2839086.6666666665, ans=0.05 2023-11-24 12:53:03,495 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5050, loss[loss=0.0708, simple_loss=0.08992, pruned_loss=0.01424, audio_tagging_loss=0.01159, over 15786.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09099, pruned_loss=0.01322, audio_tagging_loss=0.008787, over 3047197.35 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:53:11,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2839220.0, ans=0.125 2023-11-24 12:53:17,868 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.090e+01 8.266e+01 8.922e+01 9.681e+01 1.367e+02, threshold=1.784e+02, percent-clipped=0.0 2023-11-24 12:53:22,189 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425900 2023-11-24 12:53:22,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2839286.6666666665, ans=0.125 2023-11-24 12:53:48,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2839420.0, ans=0.125 2023-11-24 12:53:51,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2839420.0, ans=0.1 2023-11-24 12:53:51,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2839420.0, ans=0.1 2023-11-24 12:54:03,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.67 vs. limit=15.0 2023-11-24 12:54:06,715 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5100, loss[loss=0.0609, simple_loss=0.08283, pruned_loss=0.009979, audio_tagging_loss=0.00951, over 15044.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09089, pruned_loss=0.01317, audio_tagging_loss=0.008829, over 3053272.43 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:54:07,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2839553.3333333335, ans=0.125 2023-11-24 12:54:26,927 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 425950 2023-11-24 12:55:11,186 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5150, loss[loss=0.0755, simple_loss=0.1097, pruned_loss=0.01259, audio_tagging_loss=0.008081, over 15078.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.091, pruned_loss=0.01323, audio_tagging_loss=0.008786, over 3047769.69 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:55:21,856 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:55:26,163 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.369e+01 9.000e+01 9.837e+01 1.217e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-24 12:55:29,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426000 2023-11-24 12:55:44,511 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=15.0 2023-11-24 12:55:50,607 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2840086.6666666665, ans=0.1 2023-11-24 12:56:13,963 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.25 vs. limit=15.0 2023-11-24 12:56:14,387 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5200, loss[loss=0.06143, simple_loss=0.0851, pruned_loss=0.01027, audio_tagging_loss=0.008614, over 15699.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09038, pruned_loss=0.01316, audio_tagging_loss=0.008841, over 3044251.22 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 12:56:24,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2840220.0, ans=0.125 2023-11-24 12:56:26,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2840286.6666666665, ans=0.1 2023-11-24 12:56:27,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2840286.6666666665, ans=0.0 2023-11-24 12:56:32,716 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426050 2023-11-24 12:56:41,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2840353.3333333335, ans=0.125 2023-11-24 12:56:52,795 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2840420.0, ans=0.125 2023-11-24 12:57:12,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2840486.6666666665, ans=0.1 2023-11-24 12:57:12,199 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 12:57:14,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.66 vs. limit=22.5 2023-11-24 12:57:15,465 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5250, loss[loss=0.08133, simple_loss=0.1072, pruned_loss=0.01993, audio_tagging_loss=0.007785, over 14229.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09122, pruned_loss=0.01349, audio_tagging_loss=0.008878, over 3052187.79 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:57:21,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2840553.3333333335, ans=0.125 2023-11-24 12:57:26,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.71 vs. limit=15.0 2023-11-24 12:57:32,146 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.30 vs. limit=15.0 2023-11-24 12:57:32,386 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.236e+01 8.528e+01 9.139e+01 1.004e+02 1.210e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 12:57:35,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426100 2023-11-24 12:57:42,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2840686.6666666665, ans=0.0 2023-11-24 12:58:19,035 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5300, loss[loss=0.09336, simple_loss=0.1226, pruned_loss=0.02148, audio_tagging_loss=0.01061, over 16198.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09087, pruned_loss=0.0133, audio_tagging_loss=0.008925, over 3047971.79 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:58:23,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2840886.6666666665, ans=0.0 2023-11-24 12:58:25,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2840886.6666666665, ans=0.125 2023-11-24 12:58:38,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426150 2023-11-24 12:58:55,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2841086.6666666665, ans=0.125 2023-11-24 12:59:02,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2841086.6666666665, ans=0.0 2023-11-24 12:59:18,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2841153.3333333335, ans=0.0 2023-11-24 12:59:20,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2841153.3333333335, ans=0.125 2023-11-24 12:59:21,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=2841220.0, ans=22.5 2023-11-24 12:59:22,040 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5350, loss[loss=0.09266, simple_loss=0.1277, pruned_loss=0.02186, audio_tagging_loss=0.006967, over 14873.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09076, pruned_loss=0.01322, audio_tagging_loss=0.008883, over 3044965.45 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 12:59:27,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2841220.0, ans=0.2 2023-11-24 12:59:33,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2841286.6666666665, ans=0.2 2023-11-24 12:59:37,775 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.311e+01 8.528e+01 9.137e+01 9.887e+01 1.205e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 12:59:40,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426200 2023-11-24 13:00:02,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2841420.0, ans=0.95 2023-11-24 13:00:24,518 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5400, loss[loss=0.04809, simple_loss=0.0579, pruned_loss=0.007175, audio_tagging_loss=0.01197, over 15347.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09115, pruned_loss=0.01327, audio_tagging_loss=0.008922, over 3047581.09 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:00:43,705 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426250 2023-11-24 13:00:55,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.24 vs. limit=15.0 2023-11-24 13:00:59,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2841686.6666666665, ans=0.125 2023-11-24 13:01:05,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2841753.3333333335, ans=0.125 2023-11-24 13:01:16,021 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2841820.0, ans=0.125 2023-11-24 13:01:27,626 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5450, loss[loss=0.08681, simple_loss=0.1108, pruned_loss=0.02286, audio_tagging_loss=0.008545, over 15014.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.0908, pruned_loss=0.0132, audio_tagging_loss=0.009025, over 3051407.73 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:01:27,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2841886.6666666665, ans=0.1 2023-11-24 13:01:39,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2841953.3333333335, ans=0.2 2023-11-24 13:01:43,550 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.576e+01 9.179e+01 9.817e+01 1.405e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-24 13:01:46,724 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426300 2023-11-24 13:01:56,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2842020.0, ans=0.125 2023-11-24 13:01:56,799 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.73 vs. limit=15.0 2023-11-24 13:02:01,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2842020.0, ans=0.0 2023-11-24 13:02:13,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2842086.6666666665, ans=0.2 2023-11-24 13:02:21,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2842153.3333333335, ans=0.2 2023-11-24 13:02:26,014 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.32 vs. limit=15.0 2023-11-24 13:02:30,298 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5500, loss[loss=0.06335, simple_loss=0.08664, pruned_loss=0.0112, audio_tagging_loss=0.008826, over 15437.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09162, pruned_loss=0.0133, audio_tagging_loss=0.008916, over 3047768.05 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:02:48,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426350 2023-11-24 13:02:49,027 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:03:03,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2842353.3333333335, ans=0.125 2023-11-24 13:03:18,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2842420.0, ans=0.125 2023-11-24 13:03:32,885 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5550, loss[loss=0.06202, simple_loss=0.08691, pruned_loss=0.009928, audio_tagging_loss=0.008637, over 16640.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09248, pruned_loss=0.0135, audio_tagging_loss=0.008967, over 3056060.91 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 8.0 2023-11-24 13:03:50,770 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.724e+01 9.337e+01 1.024e+02 1.345e+02, threshold=1.867e+02, percent-clipped=0.0 2023-11-24 13:03:52,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426400 2023-11-24 13:03:59,482 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.55 vs. limit=15.0 2023-11-24 13:04:30,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2842820.0, ans=0.125 2023-11-24 13:04:36,532 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5600, loss[loss=0.04742, simple_loss=0.06587, pruned_loss=0.006713, audio_tagging_loss=0.007769, over 14806.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09251, pruned_loss=0.01332, audio_tagging_loss=0.008987, over 3060021.23 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:04:39,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.37 vs. limit=15.0 2023-11-24 13:04:50,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2842953.3333333335, ans=0.1 2023-11-24 13:04:55,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426450 2023-11-24 13:05:06,045 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.50 vs. limit=22.5 2023-11-24 13:05:15,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2843086.6666666665, ans=0.125 2023-11-24 13:05:17,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2843086.6666666665, ans=0.2 2023-11-24 13:05:17,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2843086.6666666665, ans=0.125 2023-11-24 13:05:20,390 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 13:05:23,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2843086.6666666665, ans=0.2 2023-11-24 13:05:25,940 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2843153.3333333335, ans=0.125 2023-11-24 13:05:27,600 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.97 vs. limit=15.0 2023-11-24 13:05:39,339 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5650, loss[loss=0.06902, simple_loss=0.09664, pruned_loss=0.01172, audio_tagging_loss=0.008978, over 15366.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09248, pruned_loss=0.01329, audio_tagging_loss=0.009013, over 3060486.46 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:05:49,186 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2843220.0, ans=0.0 2023-11-24 13:05:53,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2843286.6666666665, ans=0.2 2023-11-24 13:05:56,550 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.507e+01 8.997e+01 9.661e+01 1.393e+02, threshold=1.799e+02, percent-clipped=0.0 2023-11-24 13:05:57,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426500 2023-11-24 13:06:01,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2843286.6666666665, ans=0.1 2023-11-24 13:06:11,017 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2843353.3333333335, ans=0.0 2023-11-24 13:06:22,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.80 vs. limit=15.0 2023-11-24 13:06:28,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2843486.6666666665, ans=0.0 2023-11-24 13:06:41,760 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5700, loss[loss=0.09511, simple_loss=0.1396, pruned_loss=0.0191, audio_tagging_loss=0.006216, over 15175.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09282, pruned_loss=0.01333, audio_tagging_loss=0.008984, over 3056053.15 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:06:52,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2843553.3333333335, ans=0.125 2023-11-24 13:06:53,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2843620.0, ans=0.0 2023-11-24 13:07:01,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426550 2023-11-24 13:07:16,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2843686.6666666665, ans=0.125 2023-11-24 13:07:18,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2843753.3333333335, ans=0.1 2023-11-24 13:07:26,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2843753.3333333335, ans=0.125 2023-11-24 13:07:34,889 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2843820.0, ans=0.0 2023-11-24 13:07:44,425 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5750, loss[loss=0.07814, simple_loss=0.1153, pruned_loss=0.01411, audio_tagging_loss=0.006384, over 16378.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09222, pruned_loss=0.01317, audio_tagging_loss=0.008945, over 3059178.54 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:07:48,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2843886.6666666665, ans=0.125 2023-11-24 13:07:51,390 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.93 vs. limit=15.0 2023-11-24 13:08:02,008 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.384e+01 8.873e+01 9.530e+01 1.243e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-24 13:08:03,343 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426600 2023-11-24 13:08:06,163 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2843953.3333333335, ans=0.1 2023-11-24 13:08:34,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2844153.3333333335, ans=0.125 2023-11-24 13:08:48,084 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5800, loss[loss=0.06226, simple_loss=0.08161, pruned_loss=0.01399, audio_tagging_loss=0.007471, over 14400.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09087, pruned_loss=0.01297, audio_tagging_loss=0.008846, over 3053800.10 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:09:02,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2844286.6666666665, ans=0.1 2023-11-24 13:09:03,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2844286.6666666665, ans=0.2 2023-11-24 13:09:05,933 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426650 2023-11-24 13:09:08,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2844286.6666666665, ans=0.95 2023-11-24 13:09:10,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2844286.6666666665, ans=0.0 2023-11-24 13:09:30,634 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.47 vs. limit=22.5 2023-11-24 13:09:31,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2844420.0, ans=0.125 2023-11-24 13:09:34,216 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2844420.0, ans=0.2 2023-11-24 13:09:42,952 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.72 vs. limit=15.0 2023-11-24 13:09:49,429 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5850, loss[loss=0.07738, simple_loss=0.1111, pruned_loss=0.01581, audio_tagging_loss=0.006018, over 16689.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.0915, pruned_loss=0.01306, audio_tagging_loss=0.008798, over 3051386.37 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:09:49,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2844553.3333333335, ans=0.125 2023-11-24 13:09:50,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2844553.3333333335, ans=0.2 2023-11-24 13:09:53,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2844553.3333333335, ans=0.125 2023-11-24 13:09:53,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2844553.3333333335, ans=0.125 2023-11-24 13:09:53,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.74 vs. limit=15.0 2023-11-24 13:10:07,024 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.013e+01 8.489e+01 9.123e+01 9.809e+01 1.343e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-24 13:10:08,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426700 2023-11-24 13:10:13,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2844686.6666666665, ans=0.0 2023-11-24 13:10:18,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2844686.6666666665, ans=0.2 2023-11-24 13:10:28,515 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:10:30,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2844753.3333333335, ans=0.125 2023-11-24 13:10:52,120 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5900, loss[loss=0.04193, simple_loss=0.04034, pruned_loss=0.005914, audio_tagging_loss=0.01585, over 13952.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09204, pruned_loss=0.0132, audio_tagging_loss=0.008812, over 3051994.25 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:10:57,693 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2844886.6666666665, ans=0.125 2023-11-24 13:11:02,929 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0 2023-11-24 13:11:05,869 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:11:06,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2844953.3333333335, ans=0.04949747468305833 2023-11-24 13:11:10,656 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.51 vs. limit=15.0 2023-11-24 13:11:11,060 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426750 2023-11-24 13:11:44,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2845153.3333333335, ans=0.125 2023-11-24 13:11:53,913 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 5950, loss[loss=0.0556, simple_loss=0.07299, pruned_loss=0.01114, audio_tagging_loss=0.007965, over 14561.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09181, pruned_loss=0.01325, audio_tagging_loss=0.008808, over 3050928.75 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:11:59,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2845220.0, ans=0.0 2023-11-24 13:12:03,462 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.81 vs. limit=22.5 2023-11-24 13:12:05,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2845286.6666666665, ans=0.125 2023-11-24 13:12:11,020 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.386e+01 9.064e+01 9.655e+01 1.412e+02, threshold=1.813e+02, percent-clipped=0.0 2023-11-24 13:12:12,322 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426800 2023-11-24 13:12:30,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2845420.0, ans=0.125 2023-11-24 13:12:53,240 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.95 vs. limit=10.0 2023-11-24 13:12:56,166 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6000, loss[loss=0.05938, simple_loss=0.08457, pruned_loss=0.01104, audio_tagging_loss=0.006055, over 14635.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09243, pruned_loss=0.01336, audio_tagging_loss=0.008713, over 3051134.55 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:12:56,167 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 13:13:36,412 INFO [train_asr.py:1253] (3/4) Epoch 36, validation: loss=0.05813, simple_loss=0.0509, pruned_loss=0.005269, audio_tagging_loss=0.02741, over 4681554.00 frames. 2023-11-24 13:13:36,413 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 13:13:37,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2845553.3333333335, ans=0.0 2023-11-24 13:13:41,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.97 vs. limit=15.0 2023-11-24 13:13:44,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2845553.3333333335, ans=0.0 2023-11-24 13:13:51,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2845620.0, ans=0.0 2023-11-24 13:13:53,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2845620.0, ans=0.125 2023-11-24 13:13:54,694 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426850 2023-11-24 13:14:20,384 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 13:14:22,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2845753.3333333335, ans=0.2 2023-11-24 13:14:26,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2845820.0, ans=0.125 2023-11-24 13:14:31,536 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2845820.0, ans=0.1 2023-11-24 13:14:38,879 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6050, loss[loss=0.07448, simple_loss=0.09703, pruned_loss=0.01594, audio_tagging_loss=0.01002, over 14666.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09084, pruned_loss=0.01301, audio_tagging_loss=0.008817, over 3046808.73 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:14:55,410 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.206e+01 8.477e+01 9.229e+01 1.011e+02 1.265e+02, threshold=1.846e+02, percent-clipped=0.0 2023-11-24 13:14:56,727 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426900 2023-11-24 13:14:59,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2845953.3333333335, ans=0.0 2023-11-24 13:15:02,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2846020.0, ans=0.125 2023-11-24 13:15:04,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.45 vs. limit=15.0 2023-11-24 13:15:05,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2846020.0, ans=0.0 2023-11-24 13:15:12,568 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.20 vs. limit=15.0 2023-11-24 13:15:22,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2846086.6666666665, ans=0.0 2023-11-24 13:15:26,244 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=12.0 2023-11-24 13:15:39,758 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6100, loss[loss=0.05375, simple_loss=0.06374, pruned_loss=0.01088, audio_tagging_loss=0.011, over 15393.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08989, pruned_loss=0.01286, audio_tagging_loss=0.00878, over 3037044.89 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:15:51,610 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.66 vs. limit=8.0 2023-11-24 13:15:59,095 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 426950 2023-11-24 13:15:59,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2846286.6666666665, ans=0.05 2023-11-24 13:16:15,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2846353.3333333335, ans=0.1 2023-11-24 13:16:15,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2846353.3333333335, ans=0.125 2023-11-24 13:16:18,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.91 vs. limit=12.0 2023-11-24 13:16:39,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2846486.6666666665, ans=0.2 2023-11-24 13:16:42,234 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6150, loss[loss=0.06476, simple_loss=0.09264, pruned_loss=0.01058, audio_tagging_loss=0.007864, over 15823.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.08994, pruned_loss=0.01289, audio_tagging_loss=0.008805, over 3035696.23 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:17:00,449 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.380e+01 9.080e+01 9.869e+01 1.269e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-24 13:17:01,761 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427000 2023-11-24 13:17:19,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2846753.3333333335, ans=0.025 2023-11-24 13:17:31,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2846820.0, ans=0.1 2023-11-24 13:17:39,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2846820.0, ans=0.125 2023-11-24 13:17:46,196 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6200, loss[loss=0.0606, simple_loss=0.07657, pruned_loss=0.01288, audio_tagging_loss=0.00944, over 15943.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.08972, pruned_loss=0.01282, audio_tagging_loss=0.008868, over 3034955.08 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:18:03,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2846953.3333333335, ans=0.125 2023-11-24 13:18:04,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427050 2023-11-24 13:18:15,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2847020.0, ans=0.125 2023-11-24 13:18:15,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2847020.0, ans=0.025 2023-11-24 13:18:33,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2847086.6666666665, ans=0.125 2023-11-24 13:18:48,616 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6250, loss[loss=0.06129, simple_loss=0.0794, pruned_loss=0.01323, audio_tagging_loss=0.008361, over 15826.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08964, pruned_loss=0.01277, audio_tagging_loss=0.009084, over 3038521.21 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:18:52,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2847220.0, ans=0.125 2023-11-24 13:19:07,101 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.748e+01 9.375e+01 1.005e+02 1.259e+02, threshold=1.875e+02, percent-clipped=0.0 2023-11-24 13:19:07,249 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427100 2023-11-24 13:19:23,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2847353.3333333335, ans=0.125 2023-11-24 13:19:30,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2847420.0, ans=0.2 2023-11-24 13:19:51,312 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6300, loss[loss=0.04999, simple_loss=0.05956, pruned_loss=0.008671, audio_tagging_loss=0.01154, over 16658.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09066, pruned_loss=0.01287, audio_tagging_loss=0.009184, over 3041399.81 frames. ], batch size: 64, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:20:05,063 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:20:11,581 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427150 2023-11-24 13:20:11,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2847620.0, ans=0.2 2023-11-24 13:20:12,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2847620.0, ans=0.125 2023-11-24 13:20:13,078 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2847620.0, ans=0.125 2023-11-24 13:20:21,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2847686.6666666665, ans=0.0 2023-11-24 13:20:22,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2847686.6666666665, ans=0.1 2023-11-24 13:20:25,273 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.70 vs. limit=10.0 2023-11-24 13:20:38,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2847753.3333333335, ans=0.125 2023-11-24 13:20:55,128 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6350, loss[loss=0.06249, simple_loss=0.09203, pruned_loss=0.007844, audio_tagging_loss=0.00863, over 17386.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09001, pruned_loss=0.01277, audio_tagging_loss=0.009226, over 3048291.39 frames. ], batch size: 65, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:21:04,767 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.16 vs. limit=15.0 2023-11-24 13:21:05,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2847886.6666666665, ans=0.125 2023-11-24 13:21:13,484 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.568e+01 9.156e+01 9.751e+01 1.252e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-24 13:21:13,701 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427200 2023-11-24 13:21:13,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2847953.3333333335, ans=0.125 2023-11-24 13:21:38,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2848086.6666666665, ans=0.1 2023-11-24 13:21:54,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2848153.3333333335, ans=0.125 2023-11-24 13:21:57,646 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6400, loss[loss=0.06389, simple_loss=0.08934, pruned_loss=0.008089, audio_tagging_loss=0.01113, over 15492.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09003, pruned_loss=0.01288, audio_tagging_loss=0.009272, over 3047082.79 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:22:03,963 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2848220.0, ans=0.125 2023-11-24 13:22:07,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2848220.0, ans=0.0 2023-11-24 13:22:15,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427250 2023-11-24 13:22:59,738 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6450, loss[loss=0.06388, simple_loss=0.08676, pruned_loss=0.01152, audio_tagging_loss=0.008972, over 14804.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.08998, pruned_loss=0.01292, audio_tagging_loss=0.009316, over 3046112.04 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:23:00,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2848553.3333333335, ans=0.1 2023-11-24 13:23:02,954 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0 2023-11-24 13:23:18,989 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.428e+01 8.325e+01 9.052e+01 9.909e+01 1.282e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 13:23:19,156 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427300 2023-11-24 13:23:31,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2848686.6666666665, ans=0.125 2023-11-24 13:23:43,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2848753.3333333335, ans=0.125 2023-11-24 13:24:03,305 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6500, loss[loss=0.07201, simple_loss=0.1003, pruned_loss=0.01178, audio_tagging_loss=0.01007, over 16856.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.08962, pruned_loss=0.01281, audio_tagging_loss=0.009349, over 3052963.40 frames. ], batch size: 63, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:24:22,745 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427350 2023-11-24 13:24:24,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2848953.3333333335, ans=0.125 2023-11-24 13:24:26,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2848953.3333333335, ans=0.1 2023-11-24 13:24:47,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2849086.6666666665, ans=0.0 2023-11-24 13:24:49,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2849086.6666666665, ans=0.125 2023-11-24 13:24:49,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.90 vs. limit=15.0 2023-11-24 13:24:52,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.14 vs. limit=15.0 2023-11-24 13:25:06,998 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6550, loss[loss=0.06017, simple_loss=0.08257, pruned_loss=0.01068, audio_tagging_loss=0.00821, over 15989.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09021, pruned_loss=0.01281, audio_tagging_loss=0.009134, over 3052517.66 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:25:24,747 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.54 vs. limit=12.0 2023-11-24 13:25:25,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427400 2023-11-24 13:25:26,380 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.536e+01 9.192e+01 9.836e+01 1.277e+02, threshold=1.838e+02, percent-clipped=0.0 2023-11-24 13:25:31,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2849353.3333333335, ans=10.0 2023-11-24 13:25:35,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2849353.3333333335, ans=0.1 2023-11-24 13:25:43,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2849420.0, ans=0.125 2023-11-24 13:26:02,899 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.49 vs. limit=15.0 2023-11-24 13:26:09,476 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6600, loss[loss=0.06066, simple_loss=0.07927, pruned_loss=0.01067, audio_tagging_loss=0.01036, over 15866.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.08961, pruned_loss=0.01272, audio_tagging_loss=0.009025, over 3052469.86 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:26:22,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=2849620.0, ans=0.2 2023-11-24 13:26:26,650 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2849620.0, ans=0.09899494936611666 2023-11-24 13:26:28,878 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427450 2023-11-24 13:26:36,911 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2849686.6666666665, ans=0.125 2023-11-24 13:26:43,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2849686.6666666665, ans=0.95 2023-11-24 13:26:48,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2849753.3333333335, ans=0.1 2023-11-24 13:26:57,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2849753.3333333335, ans=0.125 2023-11-24 13:27:13,526 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6650, loss[loss=0.05753, simple_loss=0.07625, pruned_loss=0.00852, audio_tagging_loss=0.01088, over 14187.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08937, pruned_loss=0.01283, audio_tagging_loss=0.008956, over 3047685.67 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:27:27,736 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.80 vs. limit=15.0 2023-11-24 13:27:31,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427500 2023-11-24 13:27:33,575 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.517e+01 9.117e+01 9.877e+01 1.434e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 13:28:02,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2850153.3333333335, ans=0.125 2023-11-24 13:28:05,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2850153.3333333335, ans=0.0 2023-11-24 13:28:16,073 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6700, loss[loss=0.07094, simple_loss=0.0875, pruned_loss=0.01408, audio_tagging_loss=0.01312, over 15400.00 frames. ], tot_loss[loss=0.06653, simple_loss=0.0898, pruned_loss=0.01278, audio_tagging_loss=0.008848, over 3048341.23 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:28:16,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2850220.0, ans=0.125 2023-11-24 13:28:34,744 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427550 2023-11-24 13:29:03,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2850420.0, ans=0.1 2023-11-24 13:29:19,042 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6750, loss[loss=0.08005, simple_loss=0.1146, pruned_loss=0.01656, audio_tagging_loss=0.006173, over 15980.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.08999, pruned_loss=0.01288, audio_tagging_loss=0.008819, over 3048891.99 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:29:28,349 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.34 vs. limit=22.5 2023-11-24 13:29:38,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427600 2023-11-24 13:29:39,144 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.434e+01 8.966e+01 9.987e+01 1.535e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 13:29:54,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2850686.6666666665, ans=0.0 2023-11-24 13:30:21,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2850886.6666666665, ans=0.0 2023-11-24 13:30:22,662 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6800, loss[loss=0.07203, simple_loss=0.09626, pruned_loss=0.0144, audio_tagging_loss=0.009504, over 15415.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.09005, pruned_loss=0.01286, audio_tagging_loss=0.008892, over 3055665.61 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:30:35,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2850953.3333333335, ans=0.1 2023-11-24 13:30:41,081 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427650 2023-11-24 13:30:49,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2851020.0, ans=0.125 2023-11-24 13:30:58,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2851086.6666666665, ans=0.07 2023-11-24 13:31:00,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2851086.6666666665, ans=0.2 2023-11-24 13:31:22,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2851153.3333333335, ans=0.1 2023-11-24 13:31:24,673 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6850, loss[loss=0.04967, simple_loss=0.07119, pruned_loss=0.006527, audio_tagging_loss=0.007545, over 16861.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08953, pruned_loss=0.01279, audio_tagging_loss=0.008882, over 3056343.74 frames. ], batch size: 64, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:31:27,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2851220.0, ans=0.04949747468305833 2023-11-24 13:31:35,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2851286.6666666665, ans=0.125 2023-11-24 13:31:43,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427700 2023-11-24 13:31:44,241 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.309e+01 9.090e+01 9.871e+01 1.187e+02, threshold=1.818e+02, percent-clipped=0.0 2023-11-24 13:31:50,977 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2851353.3333333335, ans=0.1 2023-11-24 13:31:55,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2851353.3333333335, ans=0.125 2023-11-24 13:31:58,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2851353.3333333335, ans=0.125 2023-11-24 13:32:03,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2851420.0, ans=0.2 2023-11-24 13:32:11,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2851420.0, ans=0.125 2023-11-24 13:32:12,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2851420.0, ans=0.125 2023-11-24 13:32:26,282 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6900, loss[loss=0.05208, simple_loss=0.06704, pruned_loss=0.007886, audio_tagging_loss=0.01068, over 15911.00 frames. ], tot_loss[loss=0.06607, simple_loss=0.0889, pruned_loss=0.01266, audio_tagging_loss=0.00896, over 3054722.16 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:32:43,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.58 vs. limit=15.0 2023-11-24 13:32:45,919 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427750 2023-11-24 13:32:47,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.76 vs. limit=15.0 2023-11-24 13:32:53,639 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.08 vs. limit=15.0 2023-11-24 13:33:03,684 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.72 vs. limit=10.0 2023-11-24 13:33:14,574 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 13:33:20,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2851820.0, ans=0.2 2023-11-24 13:33:28,827 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 6950, loss[loss=0.07229, simple_loss=0.09979, pruned_loss=0.01282, audio_tagging_loss=0.009569, over 15021.00 frames. ], tot_loss[loss=0.06625, simple_loss=0.08921, pruned_loss=0.01267, audio_tagging_loss=0.008977, over 3051714.30 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:33:44,333 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.12 vs. limit=6.0 2023-11-24 13:33:44,579 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.78 vs. limit=22.5 2023-11-24 13:33:45,178 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2851953.3333333335, ans=0.125 2023-11-24 13:33:47,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427800 2023-11-24 13:33:50,040 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.545e+01 9.109e+01 9.801e+01 1.234e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-24 13:34:07,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2852086.6666666665, ans=0.125 2023-11-24 13:34:20,096 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.04 vs. limit=15.0 2023-11-24 13:34:31,856 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7000, loss[loss=0.05911, simple_loss=0.08631, pruned_loss=0.008931, audio_tagging_loss=0.007026, over 16834.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09038, pruned_loss=0.01293, audio_tagging_loss=0.009015, over 3057666.16 frames. ], batch size: 63, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:34:49,881 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427850 2023-11-24 13:35:04,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2852353.3333333335, ans=0.0 2023-11-24 13:35:30,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2852486.6666666665, ans=0.2 2023-11-24 13:35:34,274 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7050, loss[loss=0.07825, simple_loss=0.09397, pruned_loss=0.01955, audio_tagging_loss=0.01171, over 14541.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09008, pruned_loss=0.01297, audio_tagging_loss=0.009169, over 3059928.17 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:35:53,297 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427900 2023-11-24 13:35:57,375 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.121e+01 8.577e+01 9.267e+01 9.852e+01 1.175e+02, threshold=1.853e+02, percent-clipped=0.0 2023-11-24 13:36:05,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2852686.6666666665, ans=0.125 2023-11-24 13:36:18,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2852753.3333333335, ans=0.05 2023-11-24 13:36:25,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2852820.0, ans=0.125 2023-11-24 13:36:37,826 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7100, loss[loss=0.07549, simple_loss=0.09639, pruned_loss=0.0196, audio_tagging_loss=0.007702, over 15302.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09008, pruned_loss=0.01302, audio_tagging_loss=0.009168, over 3061833.83 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 8.0 2023-11-24 13:36:39,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2852886.6666666665, ans=0.0 2023-11-24 13:36:39,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2852886.6666666665, ans=0.025 2023-11-24 13:36:40,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2852886.6666666665, ans=0.0 2023-11-24 13:36:56,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 427950 2023-11-24 13:37:01,048 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.09 vs. limit=15.0 2023-11-24 13:37:15,325 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.17 vs. limit=15.0 2023-11-24 13:37:24,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2853086.6666666665, ans=0.2 2023-11-24 13:37:27,586 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.60 vs. limit=15.0 2023-11-24 13:37:28,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2853153.3333333335, ans=0.1 2023-11-24 13:37:31,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2853153.3333333335, ans=0.1 2023-11-24 13:37:40,817 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7150, loss[loss=0.07678, simple_loss=0.1062, pruned_loss=0.01596, audio_tagging_loss=0.007716, over 15250.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.08979, pruned_loss=0.0129, audio_tagging_loss=0.009067, over 3055554.17 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 8.0 2023-11-24 13:37:51,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2853220.0, ans=0.2 2023-11-24 13:37:59,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428000 2023-11-24 13:37:59,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2853286.6666666665, ans=0.125 2023-11-24 13:37:59,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2853286.6666666665, ans=0.0 2023-11-24 13:38:05,904 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.644e+01 9.347e+01 1.029e+02 1.240e+02, threshold=1.869e+02, percent-clipped=0.0 2023-11-24 13:38:14,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2853353.3333333335, ans=0.125 2023-11-24 13:38:44,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2853486.6666666665, ans=0.125 2023-11-24 13:38:46,555 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7200, loss[loss=0.06038, simple_loss=0.07582, pruned_loss=0.01175, audio_tagging_loss=0.01072, over 17276.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09086, pruned_loss=0.01305, audio_tagging_loss=0.009142, over 3060291.88 frames. ], batch size: 65, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:38:55,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2853553.3333333335, ans=0.0 2023-11-24 13:38:55,410 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.70 vs. limit=15.0 2023-11-24 13:39:04,907 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428050 2023-11-24 13:39:16,172 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.31 vs. limit=15.0 2023-11-24 13:39:36,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2853820.0, ans=0.125 2023-11-24 13:39:45,463 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.19 vs. limit=12.0 2023-11-24 13:39:48,303 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7250, loss[loss=0.07642, simple_loss=0.106, pruned_loss=0.01489, audio_tagging_loss=0.008526, over 15271.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09036, pruned_loss=0.0127, audio_tagging_loss=0.009192, over 3047047.16 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:39:48,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2853886.6666666665, ans=0.2 2023-11-24 13:39:52,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2853886.6666666665, ans=0.125 2023-11-24 13:39:55,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2853886.6666666665, ans=0.125 2023-11-24 13:40:01,566 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.00 vs. limit=15.0 2023-11-24 13:40:03,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2853953.3333333335, ans=0.2 2023-11-24 13:40:08,061 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428100 2023-11-24 13:40:12,049 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.442e+01 8.374e+01 8.915e+01 9.867e+01 1.264e+02, threshold=1.783e+02, percent-clipped=0.0 2023-11-24 13:40:26,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2854086.6666666665, ans=0.0 2023-11-24 13:40:29,365 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0 2023-11-24 13:40:48,657 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2854153.3333333335, ans=10.0 2023-11-24 13:40:51,997 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7300, loss[loss=0.06221, simple_loss=0.08533, pruned_loss=0.01058, audio_tagging_loss=0.008962, over 14774.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09139, pruned_loss=0.01283, audio_tagging_loss=0.009057, over 3049222.22 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:41:03,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2854286.6666666665, ans=0.2 2023-11-24 13:41:09,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2854286.6666666665, ans=0.125 2023-11-24 13:41:10,684 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428150 2023-11-24 13:41:10,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2854286.6666666665, ans=0.125 2023-11-24 13:41:16,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2854353.3333333335, ans=0.2 2023-11-24 13:41:19,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2854353.3333333335, ans=0.5 2023-11-24 13:41:28,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2854420.0, ans=0.0 2023-11-24 13:41:38,385 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.22 vs. limit=12.0 2023-11-24 13:41:43,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2854486.6666666665, ans=0.125 2023-11-24 13:41:45,632 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2854486.6666666665, ans=0.0 2023-11-24 13:41:53,641 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7350, loss[loss=0.04839, simple_loss=0.05675, pruned_loss=0.01178, audio_tagging_loss=0.008227, over 14185.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09107, pruned_loss=0.01289, audio_tagging_loss=0.00896, over 3043098.89 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:42:12,270 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428200 2023-11-24 13:42:15,970 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.426e+01 8.548e+01 9.171e+01 1.029e+02 1.460e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-24 13:42:16,414 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:42:17,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2854686.6666666665, ans=0.2 2023-11-24 13:42:23,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2854686.6666666665, ans=0.125 2023-11-24 13:42:25,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2854686.6666666665, ans=0.5 2023-11-24 13:42:29,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2854686.6666666665, ans=0.125 2023-11-24 13:42:30,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2854753.3333333335, ans=0.125 2023-11-24 13:42:44,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2854820.0, ans=0.125 2023-11-24 13:42:50,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2854820.0, ans=0.1 2023-11-24 13:42:55,291 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7400, loss[loss=0.05984, simple_loss=0.07834, pruned_loss=0.01047, audio_tagging_loss=0.01019, over 15019.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09109, pruned_loss=0.01282, audio_tagging_loss=0.008831, over 3044544.26 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:42:56,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2854886.6666666665, ans=0.0 2023-11-24 13:42:59,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2854886.6666666665, ans=0.0 2023-11-24 13:43:01,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2854886.6666666665, ans=0.025 2023-11-24 13:43:11,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.87 vs. limit=15.0 2023-11-24 13:43:14,802 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428250 2023-11-24 13:43:17,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2854953.3333333335, ans=0.05 2023-11-24 13:43:31,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2855086.6666666665, ans=0.0 2023-11-24 13:43:58,289 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7450, loss[loss=0.07797, simple_loss=0.1086, pruned_loss=0.01752, audio_tagging_loss=0.006134, over 14885.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09108, pruned_loss=0.01274, audio_tagging_loss=0.008739, over 3040795.97 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:44:17,776 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428300 2023-11-24 13:44:21,194 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.449e+01 8.702e+01 9.284e+01 1.003e+02 1.240e+02, threshold=1.857e+02, percent-clipped=0.0 2023-11-24 13:45:01,223 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7500, loss[loss=0.07853, simple_loss=0.1134, pruned_loss=0.01601, audio_tagging_loss=0.005818, over 15905.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09137, pruned_loss=0.01281, audio_tagging_loss=0.00871, over 3045274.80 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:45:19,269 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428350 2023-11-24 13:45:19,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2855620.0, ans=0.1 2023-11-24 13:45:44,442 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.19 vs. limit=22.5 2023-11-24 13:46:02,706 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7550, loss[loss=0.07204, simple_loss=0.09799, pruned_loss=0.01428, audio_tagging_loss=0.008771, over 16097.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.09073, pruned_loss=0.01272, audio_tagging_loss=0.008707, over 3050006.94 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:46:13,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2855886.6666666665, ans=10.0 2023-11-24 13:46:21,634 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428400 2023-11-24 13:46:26,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.421e+01 8.555e+01 9.258e+01 9.894e+01 1.443e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 13:46:39,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2856086.6666666665, ans=0.125 2023-11-24 13:46:47,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2856086.6666666665, ans=0.125 2023-11-24 13:46:49,751 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.98 vs. limit=10.0 2023-11-24 13:47:05,643 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7600, loss[loss=0.05483, simple_loss=0.06685, pruned_loss=0.01114, audio_tagging_loss=0.01027, over 15383.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09064, pruned_loss=0.01281, audio_tagging_loss=0.008736, over 3053965.16 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:47:24,819 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428450 2023-11-24 13:47:54,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2856486.6666666665, ans=0.0 2023-11-24 13:47:57,555 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.31 vs. limit=15.0 2023-11-24 13:48:03,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2856486.6666666665, ans=0.07 2023-11-24 13:48:08,992 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7650, loss[loss=0.07416, simple_loss=0.09961, pruned_loss=0.01535, audio_tagging_loss=0.009007, over 14819.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.09037, pruned_loss=0.01274, audio_tagging_loss=0.008797, over 3055670.60 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:48:10,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2856553.3333333335, ans=0.1 2023-11-24 13:48:23,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2856620.0, ans=0.0 2023-11-24 13:48:27,321 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428500 2023-11-24 13:48:30,892 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.493e+01 9.050e+01 9.587e+01 1.815e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 13:48:44,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2856686.6666666665, ans=0.1 2023-11-24 13:48:57,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.65 vs. limit=15.0 2023-11-24 13:49:00,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2856820.0, ans=0.2 2023-11-24 13:49:12,508 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7700, loss[loss=0.0621, simple_loss=0.0796, pruned_loss=0.01138, audio_tagging_loss=0.01093, over 15950.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09073, pruned_loss=0.01289, audio_tagging_loss=0.008829, over 3055137.15 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:49:16,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2856886.6666666665, ans=0.125 2023-11-24 13:49:31,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428550 2023-11-24 13:49:34,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2856953.3333333335, ans=0.125 2023-11-24 13:49:35,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2856953.3333333335, ans=0.0 2023-11-24 13:49:50,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2857086.6666666665, ans=0.125 2023-11-24 13:49:52,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2857086.6666666665, ans=0.125 2023-11-24 13:50:09,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2857153.3333333335, ans=0.0 2023-11-24 13:50:15,596 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7750, loss[loss=0.07288, simple_loss=0.1051, pruned_loss=0.01422, audio_tagging_loss=0.006122, over 15428.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.09016, pruned_loss=0.01275, audio_tagging_loss=0.008928, over 3050552.05 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:50:34,744 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428600 2023-11-24 13:50:38,542 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.354e+01 9.227e+01 9.861e+01 1.458e+02, threshold=1.845e+02, percent-clipped=0.0 2023-11-24 13:50:44,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2857353.3333333335, ans=0.1 2023-11-24 13:50:47,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2857353.3333333335, ans=0.2 2023-11-24 13:51:05,403 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.28 vs. limit=22.5 2023-11-24 13:51:18,251 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7800, loss[loss=0.06323, simple_loss=0.08113, pruned_loss=0.01333, audio_tagging_loss=0.009339, over 15279.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.08992, pruned_loss=0.01271, audio_tagging_loss=0.008992, over 3049541.56 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:51:23,987 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:51:36,758 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428650 2023-11-24 13:51:39,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2857620.0, ans=0.125 2023-11-24 13:51:46,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2857686.6666666665, ans=0.0 2023-11-24 13:51:51,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.64 vs. limit=10.0 2023-11-24 13:51:58,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.26 vs. limit=15.0 2023-11-24 13:52:16,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.69 vs. limit=22.5 2023-11-24 13:52:20,577 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7850, loss[loss=0.06683, simple_loss=0.07653, pruned_loss=0.01609, audio_tagging_loss=0.01248, over 14879.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09051, pruned_loss=0.01293, audio_tagging_loss=0.008924, over 3053192.28 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:52:39,537 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428700 2023-11-24 13:52:40,948 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2857953.3333333335, ans=0.05 2023-11-24 13:52:43,076 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.656e+01 8.512e+01 9.149e+01 9.887e+01 1.408e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 13:52:44,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2858020.0, ans=0.2 2023-11-24 13:52:45,306 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.88 vs. limit=22.5 2023-11-24 13:53:03,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2858086.6666666665, ans=0.5 2023-11-24 13:53:19,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2858153.3333333335, ans=0.125 2023-11-24 13:53:23,075 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7900, loss[loss=0.06842, simple_loss=0.09549, pruned_loss=0.01237, audio_tagging_loss=0.008308, over 15660.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.0916, pruned_loss=0.01302, audio_tagging_loss=0.008919, over 3059951.35 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:53:27,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2858220.0, ans=0.0 2023-11-24 13:53:36,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2858286.6666666665, ans=0.05 2023-11-24 13:53:42,167 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428750 2023-11-24 13:53:48,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2858353.3333333335, ans=0.2 2023-11-24 13:53:58,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2858353.3333333335, ans=0.0 2023-11-24 13:54:26,268 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 7950, loss[loss=0.06387, simple_loss=0.08818, pruned_loss=0.008842, audio_tagging_loss=0.01094, over 15343.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09154, pruned_loss=0.01316, audio_tagging_loss=0.00894, over 3058380.56 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:54:37,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2858620.0, ans=0.0 2023-11-24 13:54:41,151 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 13:54:42,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2858620.0, ans=0.0 2023-11-24 13:54:42,654 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2858620.0, ans=0.125 2023-11-24 13:54:44,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428800 2023-11-24 13:54:49,826 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.877e+01 9.547e+01 1.019e+02 1.290e+02, threshold=1.909e+02, percent-clipped=0.0 2023-11-24 13:55:15,720 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.94 vs. limit=10.0 2023-11-24 13:55:23,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2858820.0, ans=0.1 2023-11-24 13:55:28,617 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8000, loss[loss=0.07506, simple_loss=0.1005, pruned_loss=0.01484, audio_tagging_loss=0.009938, over 15045.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09222, pruned_loss=0.0135, audio_tagging_loss=0.009112, over 3059395.42 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 13:55:39,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2858953.3333333335, ans=0.1 2023-11-24 13:55:46,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428850 2023-11-24 13:56:30,604 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8050, loss[loss=0.0636, simple_loss=0.08175, pruned_loss=0.01367, audio_tagging_loss=0.009055, over 14419.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09286, pruned_loss=0.01362, audio_tagging_loss=0.009059, over 3055669.51 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:56:36,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2859220.0, ans=0.2 2023-11-24 13:56:48,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2859286.6666666665, ans=0.0 2023-11-24 13:56:49,562 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428900 2023-11-24 13:56:49,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2859286.6666666665, ans=0.1 2023-11-24 13:56:49,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2859286.6666666665, ans=0.125 2023-11-24 13:56:54,816 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.54 vs. limit=22.5 2023-11-24 13:56:55,197 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.620e+01 9.250e+01 9.827e+01 1.123e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 13:57:10,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2859420.0, ans=0.0 2023-11-24 13:57:32,541 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8100, loss[loss=0.05435, simple_loss=0.06697, pruned_loss=0.008521, audio_tagging_loss=0.01234, over 14868.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09304, pruned_loss=0.01361, audio_tagging_loss=0.008955, over 3049010.04 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:57:51,010 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 428950 2023-11-24 13:58:07,660 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:58:11,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2859753.3333333335, ans=0.2 2023-11-24 13:58:22,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2859820.0, ans=0.5 2023-11-24 13:58:34,219 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8150, loss[loss=0.0956, simple_loss=0.1398, pruned_loss=0.02136, audio_tagging_loss=0.004354, over 15446.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09241, pruned_loss=0.01352, audio_tagging_loss=0.008791, over 3052978.44 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:58:53,438 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429000 2023-11-24 13:58:58,087 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 13:59:00,117 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.313e+01 8.661e+01 9.369e+01 1.006e+02 1.730e+02, threshold=1.874e+02, percent-clipped=0.0 2023-11-24 13:59:06,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2860020.0, ans=0.2 2023-11-24 13:59:06,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2860020.0, ans=0.125 2023-11-24 13:59:18,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2860086.6666666665, ans=0.0 2023-11-24 13:59:25,917 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2860153.3333333335, ans=0.125 2023-11-24 13:59:36,946 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8200, loss[loss=0.064, simple_loss=0.08974, pruned_loss=0.01315, audio_tagging_loss=0.005979, over 15104.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09151, pruned_loss=0.0132, audio_tagging_loss=0.008716, over 3050358.33 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 13:59:37,010 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 13:59:40,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.66 vs. limit=22.5 2023-11-24 13:59:51,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2860286.6666666665, ans=0.2 2023-11-24 13:59:55,747 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429050 2023-11-24 13:59:59,215 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.22 vs. limit=15.0 2023-11-24 14:00:27,706 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.88 vs. limit=15.0 2023-11-24 14:00:39,592 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8250, loss[loss=0.07972, simple_loss=0.1107, pruned_loss=0.01524, audio_tagging_loss=0.009123, over 15006.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09143, pruned_loss=0.01316, audio_tagging_loss=0.008658, over 3047581.30 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:00:39,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2860553.3333333335, ans=0.0 2023-11-24 14:00:49,150 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.50 vs. limit=10.0 2023-11-24 14:00:57,984 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429100 2023-11-24 14:01:02,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2860686.6666666665, ans=0.125 2023-11-24 14:01:03,754 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.495e+01 9.085e+01 9.878e+01 1.172e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 14:01:04,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2860686.6666666665, ans=0.1 2023-11-24 14:01:41,632 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8300, loss[loss=0.06895, simple_loss=0.101, pruned_loss=0.01074, audio_tagging_loss=0.007723, over 15011.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.0916, pruned_loss=0.01298, audio_tagging_loss=0.008642, over 3055100.07 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 8.0 2023-11-24 14:01:47,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2860886.6666666665, ans=0.125 2023-11-24 14:01:47,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2860886.6666666665, ans=0.09899494936611666 2023-11-24 14:02:00,235 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429150 2023-11-24 14:02:05,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2861020.0, ans=0.125 2023-11-24 14:02:16,133 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0 2023-11-24 14:02:30,553 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.72 vs. limit=15.0 2023-11-24 14:02:38,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2861153.3333333335, ans=0.125 2023-11-24 14:02:42,981 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8350, loss[loss=0.0824, simple_loss=0.116, pruned_loss=0.01855, audio_tagging_loss=0.005847, over 15762.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09309, pruned_loss=0.0133, audio_tagging_loss=0.008609, over 3055614.52 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 8.0 2023-11-24 14:02:53,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2861220.0, ans=0.125 2023-11-24 14:02:53,859 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:03:02,461 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429200 2023-11-24 14:03:10,591 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.776e+01 9.274e+01 1.012e+02 1.289e+02, threshold=1.855e+02, percent-clipped=0.0 2023-11-24 14:03:38,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2861486.6666666665, ans=0.0 2023-11-24 14:03:46,364 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8400, loss[loss=0.04128, simple_loss=0.05518, pruned_loss=0.004254, audio_tagging_loss=0.009434, over 15191.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09319, pruned_loss=0.0132, audio_tagging_loss=0.008567, over 3052398.84 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:04:03,548 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2861620.0, ans=0.125 2023-11-24 14:04:04,686 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429250 2023-11-24 14:04:15,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2861686.6666666665, ans=0.125 2023-11-24 14:04:18,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.84 vs. limit=22.5 2023-11-24 14:04:20,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2861686.6666666665, ans=0.2 2023-11-24 14:04:21,920 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2861753.3333333335, ans=0.0 2023-11-24 14:04:39,071 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2861820.0, ans=0.125 2023-11-24 14:04:40,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2861820.0, ans=0.015 2023-11-24 14:04:47,759 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8450, loss[loss=0.05751, simple_loss=0.07046, pruned_loss=0.01152, audio_tagging_loss=0.01077, over 14369.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09216, pruned_loss=0.01294, audio_tagging_loss=0.008611, over 3046082.55 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:05:04,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2861953.3333333335, ans=0.125 2023-11-24 14:05:05,646 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429300 2023-11-24 14:05:13,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.551e+01 9.201e+01 9.862e+01 3.144e+02, threshold=1.840e+02, percent-clipped=1.0 2023-11-24 14:05:46,974 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.01 vs. limit=15.0 2023-11-24 14:05:48,434 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8500, loss[loss=0.07971, simple_loss=0.1091, pruned_loss=0.01766, audio_tagging_loss=0.007493, over 14473.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09239, pruned_loss=0.01299, audio_tagging_loss=0.008628, over 3044933.81 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:06:06,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2862286.6666666665, ans=0.0 2023-11-24 14:06:08,148 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429350 2023-11-24 14:06:14,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2862353.3333333335, ans=0.0 2023-11-24 14:06:15,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2862353.3333333335, ans=0.125 2023-11-24 14:06:18,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2862353.3333333335, ans=0.2 2023-11-24 14:06:31,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2862420.0, ans=0.125 2023-11-24 14:06:31,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2862420.0, ans=0.125 2023-11-24 14:06:39,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2862486.6666666665, ans=0.125 2023-11-24 14:06:41,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2862486.6666666665, ans=0.0 2023-11-24 14:06:51,112 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8550, loss[loss=0.0712, simple_loss=0.08998, pruned_loss=0.01593, audio_tagging_loss=0.01028, over 15891.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09216, pruned_loss=0.01303, audio_tagging_loss=0.008672, over 3053187.57 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:07:10,183 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429400 2023-11-24 14:07:17,338 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.305e+01 8.644e+01 9.213e+01 9.720e+01 1.244e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 14:07:23,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2862686.6666666665, ans=0.1 2023-11-24 14:07:34,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2862753.3333333335, ans=0.1 2023-11-24 14:07:51,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2862820.0, ans=0.125 2023-11-24 14:07:54,011 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8600, loss[loss=0.06857, simple_loss=0.08738, pruned_loss=0.01526, audio_tagging_loss=0.009617, over 14721.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09203, pruned_loss=0.01298, audio_tagging_loss=0.00884, over 3050798.16 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:07:55,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2862886.6666666665, ans=0.0 2023-11-24 14:08:01,463 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2862886.6666666665, ans=0.125 2023-11-24 14:08:12,067 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429450 2023-11-24 14:08:30,409 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.06 vs. limit=22.5 2023-11-24 14:08:48,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2863153.3333333335, ans=0.1 2023-11-24 14:08:55,746 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8650, loss[loss=0.06166, simple_loss=0.07616, pruned_loss=0.01374, audio_tagging_loss=0.009838, over 16239.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09142, pruned_loss=0.01286, audio_tagging_loss=0.008925, over 3055460.64 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:09:14,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429500 2023-11-24 14:09:18,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2863286.6666666665, ans=0.0 2023-11-24 14:09:21,999 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.258e+01 8.469e+01 9.080e+01 9.893e+01 1.324e+02, threshold=1.816e+02, percent-clipped=0.0 2023-11-24 14:09:37,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2863420.0, ans=0.0 2023-11-24 14:09:49,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.12 vs. limit=15.0 2023-11-24 14:09:57,745 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8700, loss[loss=0.06696, simple_loss=0.08972, pruned_loss=0.01452, audio_tagging_loss=0.007588, over 15206.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09055, pruned_loss=0.0127, audio_tagging_loss=0.008994, over 3047754.95 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:10:12,773 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:10:15,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2863620.0, ans=0.07 2023-11-24 14:10:17,283 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429550 2023-11-24 14:11:00,203 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8750, loss[loss=0.07217, simple_loss=0.09573, pruned_loss=0.01298, audio_tagging_loss=0.01132, over 16029.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09104, pruned_loss=0.01297, audio_tagging_loss=0.009087, over 3042230.68 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0 2023-11-24 14:11:17,935 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429600 2023-11-24 14:11:20,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2863953.3333333335, ans=0.125 2023-11-24 14:11:25,143 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.456e+01 8.680e+01 9.303e+01 1.028e+02 1.677e+02, threshold=1.861e+02, percent-clipped=0.0 2023-11-24 14:11:53,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2864153.3333333335, ans=0.2 2023-11-24 14:12:02,028 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8800, loss[loss=0.0501, simple_loss=0.06108, pruned_loss=0.009532, audio_tagging_loss=0.01003, over 14080.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09153, pruned_loss=0.01307, audio_tagging_loss=0.009173, over 3045878.66 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 14:12:21,452 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429650 2023-11-24 14:12:30,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2864353.3333333335, ans=0.2 2023-11-24 14:12:58,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2864486.6666666665, ans=0.025 2023-11-24 14:13:04,076 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8850, loss[loss=0.08426, simple_loss=0.1147, pruned_loss=0.0175, audio_tagging_loss=0.009382, over 15086.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09124, pruned_loss=0.01297, audio_tagging_loss=0.009094, over 3046990.25 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 14:13:05,636 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:13:16,432 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:13:17,822 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:13:23,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429700 2023-11-24 14:13:30,786 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.004e+01 8.562e+01 9.103e+01 9.836e+01 1.222e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 14:13:35,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2864686.6666666665, ans=0.0 2023-11-24 14:13:35,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2864686.6666666665, ans=0.1 2023-11-24 14:14:00,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2864820.0, ans=0.125 2023-11-24 14:14:01,626 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.16 vs. limit=15.0 2023-11-24 14:14:04,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2864820.0, ans=0.0 2023-11-24 14:14:07,371 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8900, loss[loss=0.06986, simple_loss=0.1007, pruned_loss=0.01266, audio_tagging_loss=0.006831, over 16127.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09166, pruned_loss=0.01294, audio_tagging_loss=0.008973, over 3047298.18 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 14:14:22,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2864953.3333333335, ans=0.125 2023-11-24 14:14:25,779 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429750 2023-11-24 14:14:25,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2864953.3333333335, ans=0.1 2023-11-24 14:14:25,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2864953.3333333335, ans=0.1 2023-11-24 14:14:36,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2865020.0, ans=0.1 2023-11-24 14:14:42,157 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:14:45,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.73 vs. limit=15.0 2023-11-24 14:14:49,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2865086.6666666665, ans=0.125 2023-11-24 14:14:50,164 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.55 vs. limit=15.0 2023-11-24 14:14:57,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2865153.3333333335, ans=0.0 2023-11-24 14:15:03,896 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2865153.3333333335, ans=0.05 2023-11-24 14:15:09,583 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 8950, loss[loss=0.05822, simple_loss=0.08727, pruned_loss=0.007794, audio_tagging_loss=0.006789, over 15193.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09164, pruned_loss=0.01297, audio_tagging_loss=0.008858, over 3049023.92 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0 2023-11-24 14:15:27,806 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429800 2023-11-24 14:15:32,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2865286.6666666665, ans=6.0 2023-11-24 14:15:35,387 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.550e+01 9.254e+01 9.924e+01 1.359e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 14:15:38,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2865353.3333333335, ans=0.125 2023-11-24 14:15:46,075 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.40 vs. limit=22.5 2023-11-24 14:16:11,699 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9000, loss[loss=0.05335, simple_loss=0.07423, pruned_loss=0.008339, audio_tagging_loss=0.007893, over 15924.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.0908, pruned_loss=0.01288, audio_tagging_loss=0.008827, over 3043315.67 frames. ], batch size: 62, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:16:11,700 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 14:16:41,582 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.3851, 3.0063, 3.3443, 2.9387, 3.7770, 3.7753, 3.2322, 3.2348], device='cuda:3') 2023-11-24 14:16:50,252 INFO [train_asr.py:1253] (3/4) Epoch 36, validation: loss=0.05864, simple_loss=0.05081, pruned_loss=0.005226, audio_tagging_loss=0.02801, over 4681554.00 frames. 2023-11-24 14:16:50,253 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 14:17:01,722 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2865620.0, ans=0.0 2023-11-24 14:17:08,453 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429850 2023-11-24 14:17:08,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2865620.0, ans=0.09899494936611666 2023-11-24 14:17:47,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2865820.0, ans=0.0 2023-11-24 14:17:52,341 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9050, loss[loss=0.06384, simple_loss=0.09386, pruned_loss=0.008287, audio_tagging_loss=0.00862, over 15118.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09218, pruned_loss=0.01313, audio_tagging_loss=0.008759, over 3054828.29 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:18:01,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2865886.6666666665, ans=0.125 2023-11-24 14:18:09,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2865953.3333333335, ans=0.2 2023-11-24 14:18:10,809 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429900 2023-11-24 14:18:11,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2865953.3333333335, ans=0.0 2023-11-24 14:18:18,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2866020.0, ans=0.2 2023-11-24 14:18:19,340 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.179e+01 8.711e+01 9.316e+01 9.957e+01 1.805e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 14:18:31,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2866086.6666666665, ans=0.125 2023-11-24 14:18:39,820 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:18:40,841 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2866153.3333333335, ans=0.125 2023-11-24 14:18:53,962 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9100, loss[loss=0.07943, simple_loss=0.1118, pruned_loss=0.01636, audio_tagging_loss=0.00719, over 14287.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09157, pruned_loss=0.01301, audio_tagging_loss=0.00872, over 3055446.08 frames. ], batch size: 52, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:19:09,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2866286.6666666665, ans=0.1 2023-11-24 14:19:09,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2866286.6666666665, ans=0.0 2023-11-24 14:19:13,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 429950 2023-11-24 14:19:13,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2866286.6666666665, ans=0.1 2023-11-24 14:19:23,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2866353.3333333335, ans=0.2 2023-11-24 14:19:45,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2866486.6666666665, ans=0.0 2023-11-24 14:19:47,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2866486.6666666665, ans=0.035 2023-11-24 14:19:56,568 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9150, loss[loss=0.06912, simple_loss=0.09246, pruned_loss=0.01189, audio_tagging_loss=0.011, over 14295.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09174, pruned_loss=0.01301, audio_tagging_loss=0.008656, over 3050765.91 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:20:15,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430000 2023-11-24 14:20:19,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.88 vs. limit=15.0 2023-11-24 14:20:20,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2866686.6666666665, ans=0.125 2023-11-24 14:20:23,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=2866686.6666666665, ans=0.2 2023-11-24 14:20:23,781 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.464e+01 8.685e+01 9.383e+01 1.043e+02 1.576e+02, threshold=1.877e+02, percent-clipped=0.0 2023-11-24 14:20:24,355 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.59 vs. limit=15.0 2023-11-24 14:20:24,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2866686.6666666665, ans=15.0 2023-11-24 14:20:31,901 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2866686.6666666665, ans=0.0 2023-11-24 14:20:46,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2866820.0, ans=0.125 2023-11-24 14:20:49,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2866820.0, ans=0.1 2023-11-24 14:20:53,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2866820.0, ans=0.125 2023-11-24 14:20:58,549 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.66 vs. limit=15.0 2023-11-24 14:20:59,764 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9200, loss[loss=0.07315, simple_loss=0.08965, pruned_loss=0.0165, audio_tagging_loss=0.01182, over 13865.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09067, pruned_loss=0.01287, audio_tagging_loss=0.00867, over 3044284.12 frames. ], batch size: 53, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:21:04,882 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2866886.6666666665, ans=0.2 2023-11-24 14:21:15,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2866953.3333333335, ans=0.0 2023-11-24 14:21:18,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430050 2023-11-24 14:21:25,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2867020.0, ans=0.125 2023-11-24 14:21:25,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2867020.0, ans=0.1 2023-11-24 14:21:28,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2867020.0, ans=0.125 2023-11-24 14:21:32,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2867020.0, ans=0.125 2023-11-24 14:21:35,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2867020.0, ans=0.125 2023-11-24 14:21:40,196 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=14.16 vs. limit=15.0 2023-11-24 14:21:52,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2867153.3333333335, ans=0.2 2023-11-24 14:21:57,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2867153.3333333335, ans=0.125 2023-11-24 14:22:01,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2867220.0, ans=0.125 2023-11-24 14:22:02,250 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9250, loss[loss=0.05806, simple_loss=0.06792, pruned_loss=0.0135, audio_tagging_loss=0.0106, over 15730.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09057, pruned_loss=0.01289, audio_tagging_loss=0.008732, over 3050507.10 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:22:02,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2867220.0, ans=0.125 2023-11-24 14:22:04,877 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2867220.0, ans=0.2 2023-11-24 14:22:21,300 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430100 2023-11-24 14:22:30,814 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.050e+01 8.538e+01 9.195e+01 9.828e+01 1.249e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 14:22:36,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2867353.3333333335, ans=0.0 2023-11-24 14:22:44,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2867420.0, ans=0.1 2023-11-24 14:23:04,577 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9300, loss[loss=0.06814, simple_loss=0.09207, pruned_loss=0.01415, audio_tagging_loss=0.007953, over 15564.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09031, pruned_loss=0.01291, audio_tagging_loss=0.008882, over 3049641.03 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:23:20,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2867620.0, ans=0.125 2023-11-24 14:23:22,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430150 2023-11-24 14:23:25,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2867620.0, ans=0.2 2023-11-24 14:23:35,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2867686.6666666665, ans=0.0 2023-11-24 14:23:58,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2867820.0, ans=0.0 2023-11-24 14:24:06,051 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9350, loss[loss=0.07287, simple_loss=0.09457, pruned_loss=0.01663, audio_tagging_loss=0.008961, over 15235.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.0904, pruned_loss=0.01283, audio_tagging_loss=0.008862, over 3050915.86 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:24:06,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2867886.6666666665, ans=0.95 2023-11-24 14:24:17,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2867953.3333333335, ans=0.0 2023-11-24 14:24:25,385 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430200 2023-11-24 14:24:29,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2867953.3333333335, ans=0.2 2023-11-24 14:24:35,740 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.521e+01 8.556e+01 8.988e+01 9.800e+01 1.410e+02, threshold=1.798e+02, percent-clipped=0.0 2023-11-24 14:24:59,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2868153.3333333335, ans=0.125 2023-11-24 14:25:04,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2868153.3333333335, ans=0.0 2023-11-24 14:25:06,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2868153.3333333335, ans=0.125 2023-11-24 14:25:08,884 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9400, loss[loss=0.08148, simple_loss=0.09782, pruned_loss=0.02125, audio_tagging_loss=0.01132, over 15918.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09082, pruned_loss=0.01297, audio_tagging_loss=0.008911, over 3044844.94 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:25:14,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2868220.0, ans=0.125 2023-11-24 14:25:18,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2868220.0, ans=0.125 2023-11-24 14:25:18,445 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.31 vs. limit=15.0 2023-11-24 14:25:27,844 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430250 2023-11-24 14:25:29,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2868286.6666666665, ans=0.0 2023-11-24 14:25:35,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2868353.3333333335, ans=0.025 2023-11-24 14:25:35,992 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.01 vs. limit=15.0 2023-11-24 14:25:39,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2868353.3333333335, ans=0.0 2023-11-24 14:25:43,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2868353.3333333335, ans=0.2 2023-11-24 14:25:44,260 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0 2023-11-24 14:25:57,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=2868486.6666666665, ans=0.025 2023-11-24 14:26:05,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2868486.6666666665, ans=0.1 2023-11-24 14:26:06,004 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.37 vs. limit=12.0 2023-11-24 14:26:10,688 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:26:11,856 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9450, loss[loss=0.09672, simple_loss=0.1282, pruned_loss=0.02316, audio_tagging_loss=0.009443, over 14469.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09095, pruned_loss=0.01321, audio_tagging_loss=0.008961, over 3045350.73 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:26:22,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2868553.3333333335, ans=0.0 2023-11-24 14:26:30,264 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430300 2023-11-24 14:26:39,331 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.666e+01 9.464e+01 1.022e+02 1.388e+02, threshold=1.893e+02, percent-clipped=0.0 2023-11-24 14:27:08,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2868820.0, ans=0.2 2023-11-24 14:27:13,072 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9500, loss[loss=0.08044, simple_loss=0.1074, pruned_loss=0.01939, audio_tagging_loss=0.007352, over 15945.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09003, pruned_loss=0.01296, audio_tagging_loss=0.009033, over 3048919.85 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:27:19,062 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2868886.6666666665, ans=0.025 2023-11-24 14:27:31,255 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430350 2023-11-24 14:27:32,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2868953.3333333335, ans=0.125 2023-11-24 14:28:14,250 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9550, loss[loss=0.08395, simple_loss=0.1193, pruned_loss=0.01664, audio_tagging_loss=0.007637, over 16376.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09022, pruned_loss=0.01296, audio_tagging_loss=0.009156, over 3046783.49 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:28:19,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2869220.0, ans=0.125 2023-11-24 14:28:32,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2869286.6666666665, ans=0.0 2023-11-24 14:28:33,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430400 2023-11-24 14:28:43,140 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.513e+01 9.117e+01 9.826e+01 1.260e+02, threshold=1.823e+02, percent-clipped=0.0 2023-11-24 14:28:50,606 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2869420.0, ans=0.09899494936611666 2023-11-24 14:29:16,930 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9600, loss[loss=0.0501, simple_loss=0.07049, pruned_loss=0.006095, audio_tagging_loss=0.008758, over 15104.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09033, pruned_loss=0.01304, audio_tagging_loss=0.00915, over 3046862.87 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:29:35,235 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430450 2023-11-24 14:29:35,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_na.min_abs, batch_count=2869620.0, ans=0.02 2023-11-24 14:30:10,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2869820.0, ans=0.05 2023-11-24 14:30:19,697 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9650, loss[loss=0.08785, simple_loss=0.1245, pruned_loss=0.01967, audio_tagging_loss=0.005919, over 15141.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09008, pruned_loss=0.01297, audio_tagging_loss=0.00913, over 3046289.01 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:30:21,390 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0 2023-11-24 14:30:23,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2869886.6666666665, ans=0.125 2023-11-24 14:30:25,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2869886.6666666665, ans=0.0 2023-11-24 14:30:37,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430500 2023-11-24 14:30:49,802 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.279e+01 8.447e+01 9.267e+01 9.901e+01 1.317e+02, threshold=1.853e+02, percent-clipped=0.0 2023-11-24 14:31:09,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2870153.3333333335, ans=0.125 2023-11-24 14:31:09,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2870153.3333333335, ans=0.2 2023-11-24 14:31:22,790 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9700, loss[loss=0.08672, simple_loss=0.1159, pruned_loss=0.02054, audio_tagging_loss=0.008231, over 13959.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08925, pruned_loss=0.01275, audio_tagging_loss=0.009069, over 3045515.32 frames. ], batch size: 52, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:31:25,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2870220.0, ans=0.0 2023-11-24 14:31:42,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430550 2023-11-24 14:31:45,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=2870286.6666666665, ans=6.0 2023-11-24 14:32:13,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2870486.6666666665, ans=0.5 2023-11-24 14:32:25,797 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9750, loss[loss=0.07175, simple_loss=0.1018, pruned_loss=0.01493, audio_tagging_loss=0.005907, over 15989.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09053, pruned_loss=0.01306, audio_tagging_loss=0.008947, over 3052161.54 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:32:29,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2870553.3333333335, ans=0.0 2023-11-24 14:32:45,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2870620.0, ans=0.0 2023-11-24 14:32:46,534 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430600 2023-11-24 14:32:57,818 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.545e+01 8.594e+01 9.328e+01 1.010e+02 1.236e+02, threshold=1.866e+02, percent-clipped=0.0 2023-11-24 14:33:08,622 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2870753.3333333335, ans=0.1 2023-11-24 14:33:13,232 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2870753.3333333335, ans=0.125 2023-11-24 14:33:31,847 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9800, loss[loss=0.07712, simple_loss=0.1107, pruned_loss=0.01843, audio_tagging_loss=0.003351, over 15771.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09068, pruned_loss=0.01313, audio_tagging_loss=0.008831, over 3053860.31 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:33:33,540 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.97 vs. limit=6.0 2023-11-24 14:33:34,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2870886.6666666665, ans=0.04949747468305833 2023-11-24 14:33:42,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.95 vs. limit=22.5 2023-11-24 14:33:50,285 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430650 2023-11-24 14:33:59,239 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2871020.0, ans=0.125 2023-11-24 14:34:18,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2871086.6666666665, ans=0.125 2023-11-24 14:34:18,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2871086.6666666665, ans=0.125 2023-11-24 14:34:28,083 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:34:34,255 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9850, loss[loss=0.08003, simple_loss=0.1028, pruned_loss=0.01983, audio_tagging_loss=0.008818, over 15255.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09096, pruned_loss=0.01334, audio_tagging_loss=0.00874, over 3045657.50 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:34:53,810 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430700 2023-11-24 14:35:05,555 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.366e+01 9.070e+01 9.726e+01 1.220e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 14:35:19,009 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2871420.0, ans=0.0 2023-11-24 14:35:23,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2871486.6666666665, ans=0.2 2023-11-24 14:35:37,175 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9900, loss[loss=0.06892, simple_loss=0.09351, pruned_loss=0.01429, audio_tagging_loss=0.007876, over 15226.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09062, pruned_loss=0.01325, audio_tagging_loss=0.008715, over 3043148.55 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:35:40,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2871553.3333333335, ans=0.125 2023-11-24 14:35:44,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2871553.3333333335, ans=0.125 2023-11-24 14:35:56,908 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430750 2023-11-24 14:35:58,280 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2871620.0, ans=0.5 2023-11-24 14:36:13,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2871753.3333333335, ans=0.125 2023-11-24 14:36:16,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2871753.3333333335, ans=0.2 2023-11-24 14:36:26,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2871820.0, ans=0.2 2023-11-24 14:36:30,212 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0 2023-11-24 14:36:31,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2871820.0, ans=0.1 2023-11-24 14:36:41,005 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 9950, loss[loss=0.07003, simple_loss=0.09366, pruned_loss=0.0168, audio_tagging_loss=0.006395, over 14318.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09026, pruned_loss=0.01317, audio_tagging_loss=0.008783, over 3039984.45 frames. ], batch size: 53, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:36:59,240 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430800 2023-11-24 14:36:59,558 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2871953.3333333335, ans=0.04949747468305833 2023-11-24 14:37:05,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2872020.0, ans=0.0 2023-11-24 14:37:05,891 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2872020.0, ans=0.0 2023-11-24 14:37:10,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2872020.0, ans=0.125 2023-11-24 14:37:11,017 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.477e+01 9.166e+01 9.822e+01 1.194e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 14:37:16,411 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.95 vs. limit=15.0 2023-11-24 14:37:18,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.28 vs. limit=22.5 2023-11-24 14:37:44,595 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10000, loss[loss=0.09598, simple_loss=0.1375, pruned_loss=0.01969, audio_tagging_loss=0.007533, over 15104.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09103, pruned_loss=0.01313, audio_tagging_loss=0.008757, over 3048079.82 frames. ], batch size: 53, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:37:47,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2872220.0, ans=0.125 2023-11-24 14:38:00,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2872286.6666666665, ans=0.0 2023-11-24 14:38:04,141 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430850 2023-11-24 14:38:50,182 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10050, loss[loss=0.06137, simple_loss=0.0862, pruned_loss=0.01031, audio_tagging_loss=0.007962, over 14769.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.0904, pruned_loss=0.01299, audio_tagging_loss=0.008787, over 3038570.63 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:38:54,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2872553.3333333335, ans=0.1 2023-11-24 14:39:09,670 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430900 2023-11-24 14:39:11,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2872620.0, ans=0.2 2023-11-24 14:39:15,457 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2872686.6666666665, ans=0.1 2023-11-24 14:39:20,973 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.464e+01 8.558e+01 9.038e+01 9.635e+01 1.199e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 14:39:21,682 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.88 vs. limit=15.0 2023-11-24 14:39:23,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2872686.6666666665, ans=0.0 2023-11-24 14:39:34,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2872753.3333333335, ans=0.0 2023-11-24 14:39:41,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2872820.0, ans=0.05 2023-11-24 14:39:42,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2872820.0, ans=0.125 2023-11-24 14:39:48,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2872820.0, ans=0.125 2023-11-24 14:39:51,559 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0 2023-11-24 14:39:54,129 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10100, loss[loss=0.05865, simple_loss=0.07271, pruned_loss=0.01256, audio_tagging_loss=0.009738, over 14239.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.0914, pruned_loss=0.01316, audio_tagging_loss=0.008803, over 3036638.07 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:39:54,492 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2872886.6666666665, ans=0.0 2023-11-24 14:40:03,805 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2872886.6666666665, ans=0.1 2023-11-24 14:40:13,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 430950 2023-11-24 14:40:28,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2873020.0, ans=0.125 2023-11-24 14:40:46,857 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:40:47,022 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2873153.3333333335, ans=0.2 2023-11-24 14:40:48,796 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.44 vs. limit=12.0 2023-11-24 14:40:57,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2873220.0, ans=0.125 2023-11-24 14:40:58,565 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10150, loss[loss=0.05475, simple_loss=0.0705, pruned_loss=0.009519, audio_tagging_loss=0.009987, over 15113.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09114, pruned_loss=0.01322, audio_tagging_loss=0.00882, over 3040269.44 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:40:58,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2873220.0, ans=0.2 2023-11-24 14:41:17,739 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431000 2023-11-24 14:41:28,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2873353.3333333335, ans=0.0 2023-11-24 14:41:29,398 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.382e+01 8.738e+01 9.315e+01 1.037e+02 1.393e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 14:41:29,481 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:41:52,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2873486.6666666665, ans=0.125 2023-11-24 14:41:59,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2873486.6666666665, ans=0.125 2023-11-24 14:42:03,156 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10200, loss[loss=0.06951, simple_loss=0.09443, pruned_loss=0.01494, audio_tagging_loss=0.007359, over 15154.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09094, pruned_loss=0.01313, audio_tagging_loss=0.008901, over 3043605.04 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:42:17,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2873620.0, ans=0.0 2023-11-24 14:42:18,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2873620.0, ans=0.125 2023-11-24 14:42:21,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2873620.0, ans=0.0 2023-11-24 14:42:22,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431050 2023-11-24 14:42:27,542 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:42:30,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2873686.6666666665, ans=0.1 2023-11-24 14:42:32,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2873686.6666666665, ans=0.125 2023-11-24 14:42:46,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2873753.3333333335, ans=0.0 2023-11-24 14:42:46,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2873753.3333333335, ans=0.0 2023-11-24 14:42:56,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2873820.0, ans=0.125 2023-11-24 14:43:06,409 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10250, loss[loss=0.08363, simple_loss=0.1077, pruned_loss=0.01986, audio_tagging_loss=0.009948, over 15253.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.08946, pruned_loss=0.01291, audio_tagging_loss=0.009084, over 3046338.14 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 14:43:15,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0 2023-11-24 14:43:25,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431100 2023-11-24 14:43:25,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2873953.3333333335, ans=0.125 2023-11-24 14:43:26,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2873953.3333333335, ans=0.1 2023-11-24 14:43:36,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2874020.0, ans=0.125 2023-11-24 14:43:39,457 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.593e+01 9.288e+01 9.901e+01 1.156e+02, threshold=1.858e+02, percent-clipped=0.0 2023-11-24 14:43:43,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2874086.6666666665, ans=0.125 2023-11-24 14:43:51,465 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.60 vs. limit=15.0 2023-11-24 14:43:59,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2874153.3333333335, ans=0.125 2023-11-24 14:44:01,543 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2874153.3333333335, ans=0.2 2023-11-24 14:44:05,646 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.52 vs. limit=22.5 2023-11-24 14:44:10,548 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10300, loss[loss=0.0708, simple_loss=0.0976, pruned_loss=0.01129, audio_tagging_loss=0.0107, over 14809.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09038, pruned_loss=0.01313, audio_tagging_loss=0.008929, over 3047593.12 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 14:44:18,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2874220.0, ans=0.125 2023-11-24 14:44:20,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2874220.0, ans=22.5 2023-11-24 14:44:29,116 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431150 2023-11-24 14:44:34,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2874353.3333333335, ans=0.2 2023-11-24 14:45:12,589 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10350, loss[loss=0.09192, simple_loss=0.1248, pruned_loss=0.02105, audio_tagging_loss=0.008482, over 15253.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09128, pruned_loss=0.01322, audio_tagging_loss=0.008921, over 3045021.87 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 14:45:12,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2874553.3333333335, ans=0.0 2023-11-24 14:45:26,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2874620.0, ans=0.125 2023-11-24 14:45:32,530 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431200 2023-11-24 14:45:36,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2874620.0, ans=0.125 2023-11-24 14:45:42,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2874686.6666666665, ans=0.125 2023-11-24 14:45:46,001 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.769e+01 8.642e+01 9.293e+01 9.950e+01 1.145e+02, threshold=1.859e+02, percent-clipped=0.0 2023-11-24 14:46:08,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2874820.0, ans=0.0 2023-11-24 14:46:10,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2874820.0, ans=0.2 2023-11-24 14:46:17,054 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10400, loss[loss=0.09109, simple_loss=0.1306, pruned_loss=0.01985, audio_tagging_loss=0.005918, over 15920.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.0907, pruned_loss=0.01314, audio_tagging_loss=0.008978, over 3036794.29 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:46:17,456 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2874886.6666666665, ans=0.0 2023-11-24 14:46:36,187 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431250 2023-11-24 14:46:36,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2874953.3333333335, ans=0.125 2023-11-24 14:46:56,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2875086.6666666665, ans=0.2 2023-11-24 14:47:16,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=2875153.3333333335, ans=0.1 2023-11-24 14:47:19,070 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2875153.3333333335, ans=0.125 2023-11-24 14:47:21,119 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10450, loss[loss=0.06649, simple_loss=0.08887, pruned_loss=0.01185, audio_tagging_loss=0.01021, over 15064.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09088, pruned_loss=0.01316, audio_tagging_loss=0.008989, over 3044993.95 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:47:39,999 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431300 2023-11-24 14:47:41,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2875286.6666666665, ans=0.125 2023-11-24 14:47:41,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2875286.6666666665, ans=0.1 2023-11-24 14:47:54,048 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.463e+01 9.315e+01 9.862e+01 1.314e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 14:48:04,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0 2023-11-24 14:48:10,970 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.38 vs. limit=15.0 2023-11-24 14:48:14,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2875486.6666666665, ans=0.0 2023-11-24 14:48:15,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2875486.6666666665, ans=0.025 2023-11-24 14:48:24,015 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10500, loss[loss=0.07368, simple_loss=0.09767, pruned_loss=0.01568, audio_tagging_loss=0.009165, over 15530.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09055, pruned_loss=0.01305, audio_tagging_loss=0.008904, over 3042308.77 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:48:42,968 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431350 2023-11-24 14:48:48,265 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2875686.6666666665, ans=0.1 2023-11-24 14:48:56,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2875686.6666666665, ans=0.125 2023-11-24 14:49:16,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2875820.0, ans=0.125 2023-11-24 14:49:26,986 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10550, loss[loss=0.06515, simple_loss=0.08411, pruned_loss=0.01298, audio_tagging_loss=0.01011, over 13612.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09099, pruned_loss=0.01307, audio_tagging_loss=0.008886, over 3032932.53 frames. ], batch size: 53, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:49:30,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2875886.6666666665, ans=0.125 2023-11-24 14:49:31,991 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:49:45,342 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431400 2023-11-24 14:49:58,960 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.301e+01 8.652e+01 9.234e+01 9.954e+01 1.298e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 14:50:12,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2876086.6666666665, ans=0.0 2023-11-24 14:50:28,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.03 vs. limit=22.5 2023-11-24 14:50:29,157 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10600, loss[loss=0.08387, simple_loss=0.1156, pruned_loss=0.01633, audio_tagging_loss=0.009754, over 15231.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09057, pruned_loss=0.013, audio_tagging_loss=0.008917, over 3041342.79 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:50:29,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2876220.0, ans=0.2 2023-11-24 14:50:42,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2876286.6666666665, ans=0.0 2023-11-24 14:50:48,109 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431450 2023-11-24 14:51:06,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2876420.0, ans=0.125 2023-11-24 14:51:31,310 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10650, loss[loss=0.0806, simple_loss=0.1084, pruned_loss=0.01827, audio_tagging_loss=0.008123, over 15552.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09068, pruned_loss=0.01294, audio_tagging_loss=0.008889, over 3039630.80 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:51:51,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431500 2023-11-24 14:52:03,201 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.03 vs. limit=10.0 2023-11-24 14:52:04,980 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.032e+01 8.499e+01 9.165e+01 9.904e+01 1.171e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 14:52:13,172 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.31 vs. limit=10.0 2023-11-24 14:52:15,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.82 vs. limit=15.0 2023-11-24 14:52:17,400 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2876753.3333333335, ans=0.0 2023-11-24 14:52:27,004 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.96 vs. limit=12.0 2023-11-24 14:52:36,286 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10700, loss[loss=0.08749, simple_loss=0.1165, pruned_loss=0.01949, audio_tagging_loss=0.009763, over 16108.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09168, pruned_loss=0.01307, audio_tagging_loss=0.008845, over 3039416.48 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:52:37,767 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2876886.6666666665, ans=0.125 2023-11-24 14:52:37,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2876886.6666666665, ans=0.0 2023-11-24 14:52:55,515 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431550 2023-11-24 14:53:18,654 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:53:20,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2877086.6666666665, ans=0.025 2023-11-24 14:53:32,040 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:53:40,854 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10750, loss[loss=0.06681, simple_loss=0.09114, pruned_loss=0.01323, audio_tagging_loss=0.008013, over 14931.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09117, pruned_loss=0.013, audio_tagging_loss=0.008814, over 3039320.49 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 14:53:47,467 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2877220.0, ans=0.1 2023-11-24 14:53:49,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2877220.0, ans=0.125 2023-11-24 14:53:59,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431600 2023-11-24 14:54:11,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2877353.3333333335, ans=0.025 2023-11-24 14:54:14,503 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.628e+01 9.351e+01 9.677e+01 1.250e+02, threshold=1.870e+02, percent-clipped=0.0 2023-11-24 14:54:35,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2877486.6666666665, ans=0.125 2023-11-24 14:54:43,928 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10800, loss[loss=0.05035, simple_loss=0.06205, pruned_loss=0.009301, audio_tagging_loss=0.01002, over 16129.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09072, pruned_loss=0.01292, audio_tagging_loss=0.008792, over 3035485.84 frames. ], batch size: 63, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:54:51,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2877553.3333333335, ans=0.125 2023-11-24 14:54:57,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2877620.0, ans=0.125 2023-11-24 14:55:02,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431650 2023-11-24 14:55:05,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2877620.0, ans=0.125 2023-11-24 14:55:09,493 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.82 vs. limit=15.0 2023-11-24 14:55:10,352 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2877686.6666666665, ans=0.125 2023-11-24 14:55:12,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2877686.6666666665, ans=0.05 2023-11-24 14:55:16,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2877686.6666666665, ans=0.125 2023-11-24 14:55:36,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2877820.0, ans=0.0 2023-11-24 14:55:46,754 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10850, loss[loss=0.0585, simple_loss=0.08259, pruned_loss=0.008425, audio_tagging_loss=0.00878, over 15339.00 frames. ], tot_loss[loss=0.06621, simple_loss=0.08953, pruned_loss=0.01263, audio_tagging_loss=0.008823, over 3034144.14 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:55:51,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2877886.6666666665, ans=0.0 2023-11-24 14:55:56,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2877886.6666666665, ans=0.0 2023-11-24 14:56:05,721 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431700 2023-11-24 14:56:09,719 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.34 vs. limit=22.5 2023-11-24 14:56:16,580 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 14:56:18,680 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.559e+01 9.138e+01 9.969e+01 1.288e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 14:56:28,018 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.37 vs. limit=12.0 2023-11-24 14:56:29,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2878086.6666666665, ans=0.1 2023-11-24 14:56:36,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2878153.3333333335, ans=0.0 2023-11-24 14:56:45,772 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:56:49,799 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10900, loss[loss=0.0775, simple_loss=0.1093, pruned_loss=0.01466, audio_tagging_loss=0.008177, over 15967.00 frames. ], tot_loss[loss=0.06616, simple_loss=0.08927, pruned_loss=0.01261, audio_tagging_loss=0.008913, over 3041247.67 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:56:52,416 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2878220.0, ans=0.1 2023-11-24 14:56:54,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2878220.0, ans=0.125 2023-11-24 14:57:03,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2878286.6666666665, ans=0.125 2023-11-24 14:57:04,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.60 vs. limit=6.0 2023-11-24 14:57:07,894 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431750 2023-11-24 14:57:26,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.96 vs. limit=15.0 2023-11-24 14:57:38,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2878486.6666666665, ans=0.0 2023-11-24 14:57:51,586 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 10950, loss[loss=0.0702, simple_loss=0.08724, pruned_loss=0.01492, audio_tagging_loss=0.01167, over 14718.00 frames. ], tot_loss[loss=0.06624, simple_loss=0.08952, pruned_loss=0.01256, audio_tagging_loss=0.008919, over 3038661.74 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:58:00,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2878553.3333333335, ans=10.0 2023-11-24 14:58:10,009 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431800 2023-11-24 14:58:24,305 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.538e+01 9.236e+01 1.007e+02 1.326e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 14:58:53,775 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11000, loss[loss=0.06538, simple_loss=0.0943, pruned_loss=0.01017, audio_tagging_loss=0.008064, over 15707.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.0897, pruned_loss=0.01252, audio_tagging_loss=0.008981, over 3037603.04 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 14:58:59,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2878886.6666666665, ans=0.1 2023-11-24 14:59:05,657 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 14:59:12,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2878953.3333333335, ans=0.125 2023-11-24 14:59:13,494 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431850 2023-11-24 14:59:42,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2879153.3333333335, ans=0.0 2023-11-24 14:59:50,852 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2879153.3333333335, ans=0.0 2023-11-24 14:59:56,743 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11050, loss[loss=0.07668, simple_loss=0.1046, pruned_loss=0.01448, audio_tagging_loss=0.009905, over 15869.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09061, pruned_loss=0.01271, audio_tagging_loss=0.00901, over 3037137.19 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:00:02,479 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2879220.0, ans=0.2 2023-11-24 15:00:04,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2879220.0, ans=0.125 2023-11-24 15:00:15,369 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431900 2023-11-24 15:00:28,282 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.634e+01 9.324e+01 1.017e+02 1.244e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 15:00:38,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2879420.0, ans=0.1 2023-11-24 15:00:38,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.44 vs. limit=22.5 2023-11-24 15:00:59,021 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11100, loss[loss=0.0778, simple_loss=0.09461, pruned_loss=0.01866, audio_tagging_loss=0.01184, over 14234.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09096, pruned_loss=0.01291, audio_tagging_loss=0.009042, over 3035900.53 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:01:02,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2879553.3333333335, ans=0.125 2023-11-24 15:01:08,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2879553.3333333335, ans=0.125 2023-11-24 15:01:14,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2879620.0, ans=0.0 2023-11-24 15:01:17,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 431950 2023-11-24 15:01:51,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2879820.0, ans=0.125 2023-11-24 15:01:59,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2879886.6666666665, ans=0.0 2023-11-24 15:02:00,874 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11150, loss[loss=0.06625, simple_loss=0.09425, pruned_loss=0.0124, audio_tagging_loss=0.006723, over 15913.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09142, pruned_loss=0.01292, audio_tagging_loss=0.009109, over 3039255.13 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:02:04,900 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.93 vs. limit=22.5 2023-11-24 15:02:08,045 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2879886.6666666665, ans=0.0 2023-11-24 15:02:21,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432000 2023-11-24 15:02:36,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2880020.0, ans=0.125 2023-11-24 15:02:37,217 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.446e+01 9.218e+01 9.993e+01 1.214e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 15:02:38,638 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2880020.0, ans=0.125 2023-11-24 15:02:52,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2880153.3333333335, ans=0.09899494936611666 2023-11-24 15:03:04,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2880153.3333333335, ans=0.1 2023-11-24 15:03:07,438 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11200, loss[loss=0.06893, simple_loss=0.09665, pruned_loss=0.01241, audio_tagging_loss=0.008201, over 15337.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09173, pruned_loss=0.01297, audio_tagging_loss=0.009145, over 3047082.91 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:03:25,994 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432050 2023-11-24 15:04:00,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2880486.6666666665, ans=0.0 2023-11-24 15:04:04,617 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.85 vs. limit=15.0 2023-11-24 15:04:09,706 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11250, loss[loss=0.0717, simple_loss=0.09598, pruned_loss=0.01412, audio_tagging_loss=0.009592, over 16076.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09072, pruned_loss=0.01291, audio_tagging_loss=0.009207, over 3039437.45 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:04:13,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2880553.3333333335, ans=0.0 2023-11-24 15:04:27,476 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432100 2023-11-24 15:04:28,709 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2880620.0, ans=0.1 2023-11-24 15:04:41,559 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.423e+01 8.402e+01 9.085e+01 9.896e+01 1.491e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 15:05:10,111 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.21 vs. limit=12.0 2023-11-24 15:05:10,589 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11300, loss[loss=0.06751, simple_loss=0.1013, pruned_loss=0.009876, audio_tagging_loss=0.006986, over 15794.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.0908, pruned_loss=0.01285, audio_tagging_loss=0.009073, over 3038583.47 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:05:12,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2880886.6666666665, ans=0.0 2023-11-24 15:05:29,696 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432150 2023-11-24 15:05:42,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2881020.0, ans=0.2 2023-11-24 15:05:49,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2881086.6666666665, ans=0.125 2023-11-24 15:06:13,227 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11350, loss[loss=0.08607, simple_loss=0.1157, pruned_loss=0.01989, audio_tagging_loss=0.008317, over 16384.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09131, pruned_loss=0.01302, audio_tagging_loss=0.008899, over 3042908.40 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:06:15,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2881220.0, ans=0.1 2023-11-24 15:06:32,165 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432200 2023-11-24 15:06:32,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2881286.6666666665, ans=0.1 2023-11-24 15:06:45,422 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.608e+01 9.253e+01 9.930e+01 1.315e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 15:07:06,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.16 vs. limit=15.0 2023-11-24 15:07:08,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2881486.6666666665, ans=0.1 2023-11-24 15:07:16,364 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11400, loss[loss=0.05883, simple_loss=0.07328, pruned_loss=0.01045, audio_tagging_loss=0.01174, over 14698.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09075, pruned_loss=0.0129, audio_tagging_loss=0.008852, over 3044951.03 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:07:21,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=22.5 2023-11-24 15:07:34,326 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432250 2023-11-24 15:07:34,766 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=15.0 2023-11-24 15:07:42,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2881686.6666666665, ans=0.125 2023-11-24 15:07:58,087 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.32 vs. limit=10.0 2023-11-24 15:07:59,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2881753.3333333335, ans=0.0 2023-11-24 15:08:13,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2881820.0, ans=0.0 2023-11-24 15:08:18,293 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11450, loss[loss=0.09041, simple_loss=0.1226, pruned_loss=0.02262, audio_tagging_loss=0.006491, over 14872.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09157, pruned_loss=0.0131, audio_tagging_loss=0.008755, over 3043778.01 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:08:21,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2881886.6666666665, ans=0.1 2023-11-24 15:08:22,533 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.37 vs. limit=15.0 2023-11-24 15:08:23,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2881886.6666666665, ans=0.125 2023-11-24 15:08:37,398 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432300 2023-11-24 15:08:46,968 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.86 vs. limit=15.0 2023-11-24 15:08:51,397 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.204e+01 8.606e+01 9.263e+01 9.916e+01 1.290e+02, threshold=1.853e+02, percent-clipped=0.0 2023-11-24 15:08:52,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2882020.0, ans=0.125 2023-11-24 15:09:12,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2882153.3333333335, ans=0.0 2023-11-24 15:09:17,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.whiten.whitening_limit, batch_count=2882153.3333333335, ans=12.0 2023-11-24 15:09:20,632 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11500, loss[loss=0.06762, simple_loss=0.08977, pruned_loss=0.01364, audio_tagging_loss=0.009094, over 15263.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.092, pruned_loss=0.01326, audio_tagging_loss=0.008781, over 3042128.22 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 15:09:25,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2882220.0, ans=0.0 2023-11-24 15:09:39,326 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432350 2023-11-24 15:09:39,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2882286.6666666665, ans=0.0 2023-11-24 15:09:52,466 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:09:55,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2882420.0, ans=0.0 2023-11-24 15:10:21,970 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11550, loss[loss=0.07947, simple_loss=0.1137, pruned_loss=0.01546, audio_tagging_loss=0.007172, over 15758.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09175, pruned_loss=0.01323, audio_tagging_loss=0.008788, over 3043655.99 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 15:10:40,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432400 2023-11-24 15:10:40,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2882620.0, ans=0.125 2023-11-24 15:10:55,835 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.536e+01 9.084e+01 9.886e+01 1.170e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 15:10:58,568 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2882753.3333333335, ans=0.0 2023-11-24 15:11:01,155 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 15:11:09,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2882753.3333333335, ans=0.95 2023-11-24 15:11:18,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2882820.0, ans=0.125 2023-11-24 15:11:24,131 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11600, loss[loss=0.06335, simple_loss=0.08986, pruned_loss=0.01024, audio_tagging_loss=0.008188, over 14733.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09166, pruned_loss=0.01319, audio_tagging_loss=0.008735, over 3044138.60 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0 2023-11-24 15:11:36,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2882953.3333333335, ans=0.125 2023-11-24 15:11:41,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2882953.3333333335, ans=0.0 2023-11-24 15:11:43,017 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432450 2023-11-24 15:12:16,319 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:12:26,852 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11650, loss[loss=0.04349, simple_loss=0.05204, pruned_loss=0.004494, audio_tagging_loss=0.01298, over 15213.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09174, pruned_loss=0.01313, audio_tagging_loss=0.008715, over 3046787.98 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 15:12:27,580 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.35 vs. limit=15.0 2023-11-24 15:12:29,451 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2883220.0, ans=0.125 2023-11-24 15:12:34,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2883220.0, ans=0.125 2023-11-24 15:12:44,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2883286.6666666665, ans=0.2 2023-11-24 15:12:45,577 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432500 2023-11-24 15:12:52,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2883353.3333333335, ans=0.125 2023-11-24 15:13:01,375 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.376e+01 8.463e+01 9.086e+01 9.673e+01 1.142e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 15:13:09,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2883420.0, ans=0.125 2023-11-24 15:13:10,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2883420.0, ans=0.125 2023-11-24 15:13:15,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2883486.6666666665, ans=0.0 2023-11-24 15:13:25,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2883486.6666666665, ans=0.125 2023-11-24 15:13:28,775 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11700, loss[loss=0.07486, simple_loss=0.09682, pruned_loss=0.01813, audio_tagging_loss=0.008319, over 15016.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09191, pruned_loss=0.01323, audio_tagging_loss=0.008721, over 3046574.32 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 15:13:37,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2883553.3333333335, ans=0.0 2023-11-24 15:13:47,040 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432550 2023-11-24 15:13:48,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2883620.0, ans=0.125 2023-11-24 15:14:31,235 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11750, loss[loss=0.06032, simple_loss=0.08325, pruned_loss=0.00879, audio_tagging_loss=0.009902, over 14960.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09248, pruned_loss=0.01332, audio_tagging_loss=0.008805, over 3050530.89 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 15:14:48,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2883953.3333333335, ans=0.125 2023-11-24 15:14:50,164 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432600 2023-11-24 15:14:50,758 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.94 vs. limit=12.0 2023-11-24 15:14:53,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.13 vs. limit=15.0 2023-11-24 15:15:07,584 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.388e+01 9.250e+01 9.905e+01 1.592e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 15:15:09,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2884086.6666666665, ans=0.125 2023-11-24 15:15:16,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2884086.6666666665, ans=0.0 2023-11-24 15:15:19,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2884086.6666666665, ans=0.125 2023-11-24 15:15:26,841 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.70 vs. limit=15.0 2023-11-24 15:15:34,421 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11800, loss[loss=0.06309, simple_loss=0.08749, pruned_loss=0.00981, audio_tagging_loss=0.009531, over 15785.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09267, pruned_loss=0.01348, audio_tagging_loss=0.008818, over 3046643.52 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 15:15:49,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2884286.6666666665, ans=0.125 2023-11-24 15:15:52,878 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432650 2023-11-24 15:16:06,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2884353.3333333335, ans=0.025 2023-11-24 15:16:20,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2884420.0, ans=0.0 2023-11-24 15:16:23,549 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=2884486.6666666665, ans=0.2 2023-11-24 15:16:30,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2884486.6666666665, ans=0.09899494936611666 2023-11-24 15:16:33,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2884486.6666666665, ans=0.0 2023-11-24 15:16:36,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2884553.3333333335, ans=0.125 2023-11-24 15:16:36,950 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11850, loss[loss=0.08234, simple_loss=0.1145, pruned_loss=0.01853, audio_tagging_loss=0.00657, over 16095.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09316, pruned_loss=0.01367, audio_tagging_loss=0.008995, over 3053339.74 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 15:16:54,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432700 2023-11-24 15:16:59,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2884620.0, ans=0.07 2023-11-24 15:17:06,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2884686.6666666665, ans=0.125 2023-11-24 15:17:13,150 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.609e+01 8.479e+01 9.210e+01 9.675e+01 1.207e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-24 15:17:24,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2884753.3333333335, ans=0.0 2023-11-24 15:17:27,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2884820.0, ans=0.0 2023-11-24 15:17:36,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2884820.0, ans=0.125 2023-11-24 15:17:37,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2884886.6666666665, ans=0.05 2023-11-24 15:17:38,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0 2023-11-24 15:17:38,786 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11900, loss[loss=0.0648, simple_loss=0.08933, pruned_loss=0.01125, audio_tagging_loss=0.00889, over 14805.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09284, pruned_loss=0.01358, audio_tagging_loss=0.009069, over 3054673.44 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 15:17:56,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.91 vs. limit=22.5 2023-11-24 15:17:58,178 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432750 2023-11-24 15:18:40,536 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 11950, loss[loss=0.05708, simple_loss=0.07944, pruned_loss=0.00894, audio_tagging_loss=0.008418, over 14101.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09175, pruned_loss=0.01347, audio_tagging_loss=0.009202, over 3045637.56 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 8.0 2023-11-24 15:19:00,181 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432800 2023-11-24 15:19:09,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2885353.3333333335, ans=0.0 2023-11-24 15:19:16,919 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.459e+01 8.985e+01 9.590e+01 1.349e+02, threshold=1.797e+02, percent-clipped=0.0 2023-11-24 15:19:17,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2885420.0, ans=0.125 2023-11-24 15:19:29,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2885486.6666666665, ans=0.125 2023-11-24 15:19:41,564 INFO [train_asr.py:1221] (3/4) Epoch 36, batch 12000, loss[loss=0.09692, simple_loss=0.1384, pruned_loss=0.02109, audio_tagging_loss=0.006614, over 16038.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09214, pruned_loss=0.01347, audio_tagging_loss=0.009185, over 3047082.24 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 16.0 2023-11-24 15:19:41,565 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 15:20:14,326 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6024, 3.6720, 3.9322, 3.4566], device='cuda:3') 2023-11-24 15:20:23,119 INFO [train_asr.py:1253] (3/4) Epoch 36, validation: loss=0.05822, simple_loss=0.05085, pruned_loss=0.005219, audio_tagging_loss=0.02757, over 4681554.00 frames. 2023-11-24 15:20:23,120 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 15:20:35,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2885620.0, ans=0.125 2023-11-24 15:20:39,987 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432850 2023-11-24 15:20:45,192 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=22.5 2023-11-24 15:21:26,919 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 0, loss[loss=0.0763, simple_loss=0.08862, pruned_loss=0.01215, audio_tagging_loss=0.01984, over 14723.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.08862, pruned_loss=0.01215, audio_tagging_loss=0.01984, over 14723.00 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:21:26,920 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 15:22:03,067 INFO [train_asr.py:1253] (3/4) Epoch 37, validation: loss=0.05797, simple_loss=0.05085, pruned_loss=0.005252, audio_tagging_loss=0.02729, over 4681554.00 frames. 2023-11-24 15:22:03,067 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 15:22:04,486 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:22:05,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2885720.0, ans=0.125 2023-11-24 15:22:08,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2885720.0, ans=0.0 2023-11-24 15:22:14,398 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.25 vs. limit=15.0 2023-11-24 15:22:24,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2885786.6666666665, ans=0.125 2023-11-24 15:22:36,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2885853.3333333335, ans=0.0 2023-11-24 15:22:43,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2885920.0, ans=0.125 2023-11-24 15:22:46,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2885920.0, ans=0.125 2023-11-24 15:22:53,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432900 2023-11-24 15:23:01,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2885986.6666666665, ans=0.0 2023-11-24 15:23:01,669 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.43 vs. limit=15.0 2023-11-24 15:23:06,062 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 50, loss[loss=0.07815, simple_loss=0.09536, pruned_loss=0.01125, audio_tagging_loss=0.01922, over 15782.00 frames. ], tot_loss[loss=0.07744, simple_loss=0.09415, pruned_loss=0.0134, audio_tagging_loss=0.01696, over 698976.36 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:23:10,798 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.891e+01 9.190e+01 9.778e+01 1.072e+02 1.495e+02, threshold=1.956e+02, percent-clipped=0.0 2023-11-24 15:23:14,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2886053.3333333335, ans=0.1 2023-11-24 15:23:27,154 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2886120.0, ans=0.125 2023-11-24 15:23:55,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 432950 2023-11-24 15:24:05,898 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2886320.0, ans=0.2 2023-11-24 15:24:07,986 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 100, loss[loss=0.06934, simple_loss=0.08436, pruned_loss=0.01079, audio_tagging_loss=0.01637, over 16299.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09069, pruned_loss=0.01293, audio_tagging_loss=0.01661, over 1220645.54 frames. ], batch size: 63, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:24:14,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2886386.6666666665, ans=0.2 2023-11-24 15:24:25,039 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:24:25,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.54 vs. limit=22.5 2023-11-24 15:24:51,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.04 vs. limit=15.0 2023-11-24 15:24:52,034 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:24:57,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433000 2023-11-24 15:25:09,901 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 150, loss[loss=0.05472, simple_loss=0.0715, pruned_loss=0.007952, audio_tagging_loss=0.01102, over 14705.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09017, pruned_loss=0.01268, audio_tagging_loss=0.01483, over 1621361.03 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:25:10,525 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.07 vs. limit=10.0 2023-11-24 15:25:12,999 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.68 vs. limit=15.0 2023-11-24 15:25:16,639 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.469e+01 8.996e+01 9.560e+01 1.019e+02 1.193e+02, threshold=1.912e+02, percent-clipped=0.0 2023-11-24 15:25:18,774 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.38 vs. limit=22.5 2023-11-24 15:25:26,251 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.52 vs. limit=10.0 2023-11-24 15:25:32,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2886786.6666666665, ans=0.125 2023-11-24 15:25:43,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2886853.3333333335, ans=0.2 2023-11-24 15:25:45,528 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.81 vs. limit=22.5 2023-11-24 15:25:59,145 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433050 2023-11-24 15:26:04,089 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0 2023-11-24 15:26:12,773 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 200, loss[loss=0.0706, simple_loss=0.09339, pruned_loss=0.01565, audio_tagging_loss=0.008256, over 15609.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09025, pruned_loss=0.0125, audio_tagging_loss=0.01303, over 1940992.86 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:26:28,301 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.66 vs. limit=15.0 2023-11-24 15:26:37,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2887186.6666666665, ans=0.5 2023-11-24 15:27:02,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433100 2023-11-24 15:27:13,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2887320.0, ans=0.125 2023-11-24 15:27:15,140 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 250, loss[loss=0.07367, simple_loss=0.09594, pruned_loss=0.01816, audio_tagging_loss=0.007547, over 15303.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09164, pruned_loss=0.01291, audio_tagging_loss=0.01162, over 2194099.05 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:27:18,270 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.66 vs. limit=15.0 2023-11-24 15:27:20,925 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.613e+01 8.812e+01 9.417e+01 1.032e+02 1.666e+02, threshold=1.883e+02, percent-clipped=0.0 2023-11-24 15:27:49,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2887520.0, ans=0.125 2023-11-24 15:28:04,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433150 2023-11-24 15:28:16,220 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 300, loss[loss=0.05826, simple_loss=0.07315, pruned_loss=0.01073, audio_tagging_loss=0.01095, over 14077.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09149, pruned_loss=0.01284, audio_tagging_loss=0.01082, over 2385699.17 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:28:25,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2887720.0, ans=0.125 2023-11-24 15:28:32,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2887786.6666666665, ans=0.125 2023-11-24 15:28:35,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2887786.6666666665, ans=0.2 2023-11-24 15:28:36,444 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2887786.6666666665, ans=0.0 2023-11-24 15:28:39,148 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.92 vs. limit=22.5 2023-11-24 15:28:56,089 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2887920.0, ans=0.05 2023-11-24 15:29:03,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2887920.0, ans=0.0 2023-11-24 15:29:05,474 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433200 2023-11-24 15:29:14,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2887986.6666666665, ans=0.125 2023-11-24 15:29:16,544 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.09 vs. limit=15.0 2023-11-24 15:29:18,707 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 350, loss[loss=0.06831, simple_loss=0.1046, pruned_loss=0.009421, audio_tagging_loss=0.006615, over 16353.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09161, pruned_loss=0.01283, audio_tagging_loss=0.01025, over 2532214.61 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:29:25,207 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.488e+01 8.531e+01 9.197e+01 9.925e+01 1.241e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 15:29:36,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2888120.0, ans=0.125 2023-11-24 15:29:44,041 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2888186.6666666665, ans=0.125 2023-11-24 15:29:55,801 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:29:57,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2888253.3333333335, ans=0.0 2023-11-24 15:30:09,249 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433250 2023-11-24 15:30:16,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2888320.0, ans=0.0 2023-11-24 15:30:21,524 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 400, loss[loss=0.1017, simple_loss=0.1384, pruned_loss=0.02387, audio_tagging_loss=0.00861, over 16015.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.0912, pruned_loss=0.01273, audio_tagging_loss=0.009828, over 2642734.42 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:30:26,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2888386.6666666665, ans=0.125 2023-11-24 15:30:43,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2888453.3333333335, ans=0.125 2023-11-24 15:30:50,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2888520.0, ans=0.95 2023-11-24 15:31:11,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433300 2023-11-24 15:31:23,320 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 450, loss[loss=0.05561, simple_loss=0.07239, pruned_loss=0.007298, audio_tagging_loss=0.01212, over 15209.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.091, pruned_loss=0.01248, audio_tagging_loss=0.009546, over 2729841.87 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:31:23,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2888720.0, ans=0.0 2023-11-24 15:31:25,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2888720.0, ans=0.0 2023-11-24 15:31:30,278 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.409e+01 9.030e+01 9.754e+01 1.188e+02, threshold=1.806e+02, percent-clipped=0.0 2023-11-24 15:31:37,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2888786.6666666665, ans=0.125 2023-11-24 15:31:44,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2888786.6666666665, ans=0.125 2023-11-24 15:32:04,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0 2023-11-24 15:32:12,970 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433350 2023-11-24 15:32:25,462 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 500, loss[loss=0.06337, simple_loss=0.0754, pruned_loss=0.01466, audio_tagging_loss=0.01101, over 14783.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09008, pruned_loss=0.01245, audio_tagging_loss=0.009479, over 2792381.84 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:32:29,747 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2889053.3333333335, ans=0.125 2023-11-24 15:32:33,727 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.87 vs. limit=15.0 2023-11-24 15:32:36,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2889053.3333333335, ans=0.0 2023-11-24 15:32:45,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2889120.0, ans=0.125 2023-11-24 15:32:52,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2889186.6666666665, ans=0.07 2023-11-24 15:32:57,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2889186.6666666665, ans=0.07 2023-11-24 15:33:16,213 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433400 2023-11-24 15:33:16,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2889320.0, ans=0.1 2023-11-24 15:33:28,911 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 550, loss[loss=0.0664, simple_loss=0.09049, pruned_loss=0.01069, audio_tagging_loss=0.01046, over 16885.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.08979, pruned_loss=0.01247, audio_tagging_loss=0.009348, over 2851005.38 frames. ], batch size: 63, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:33:29,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2889386.6666666665, ans=0.1 2023-11-24 15:33:35,036 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.81 vs. limit=15.0 2023-11-24 15:33:36,634 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.403e+01 9.014e+01 9.757e+01 1.245e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-24 15:33:37,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2889386.6666666665, ans=0.125 2023-11-24 15:33:46,771 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.80 vs. limit=15.0 2023-11-24 15:33:48,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2889453.3333333335, ans=0.0 2023-11-24 15:34:09,151 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.98 vs. limit=15.0 2023-11-24 15:34:18,821 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433450 2023-11-24 15:34:20,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2889653.3333333335, ans=0.0 2023-11-24 15:34:23,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2889653.3333333335, ans=0.1 2023-11-24 15:34:30,540 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 600, loss[loss=0.06787, simple_loss=0.08909, pruned_loss=0.01195, audio_tagging_loss=0.01137, over 15051.00 frames. ], tot_loss[loss=0.06622, simple_loss=0.08875, pruned_loss=0.01244, audio_tagging_loss=0.009403, over 2894919.50 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:35:03,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2889853.3333333335, ans=0.125 2023-11-24 15:35:12,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2889920.0, ans=0.125 2023-11-24 15:35:13,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.81 vs. limit=15.0 2023-11-24 15:35:18,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2889920.0, ans=0.0 2023-11-24 15:35:20,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433500 2023-11-24 15:35:33,127 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 650, loss[loss=0.04923, simple_loss=0.06277, pruned_loss=0.007058, audio_tagging_loss=0.01079, over 13967.00 frames. ], tot_loss[loss=0.06571, simple_loss=0.08821, pruned_loss=0.01226, audio_tagging_loss=0.009347, over 2920648.85 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:35:40,749 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 8.381e+01 9.134e+01 9.904e+01 1.320e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 15:35:45,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.45 vs. limit=10.0 2023-11-24 15:36:01,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2890186.6666666665, ans=0.1 2023-11-24 15:36:12,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2890253.3333333335, ans=0.125 2023-11-24 15:36:13,425 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.18 vs. limit=6.0 2023-11-24 15:36:23,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433550 2023-11-24 15:36:35,538 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 700, loss[loss=0.08556, simple_loss=0.1221, pruned_loss=0.01743, audio_tagging_loss=0.00711, over 16353.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.0902, pruned_loss=0.01246, audio_tagging_loss=0.009147, over 2959838.07 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:36:35,725 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2890386.6666666665, ans=0.0 2023-11-24 15:36:53,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2890453.3333333335, ans=0.125 2023-11-24 15:36:58,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2890453.3333333335, ans=0.0 2023-11-24 15:37:02,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2890520.0, ans=0.0 2023-11-24 15:37:04,978 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.76 vs. limit=15.0 2023-11-24 15:37:20,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.37 vs. limit=15.0 2023-11-24 15:37:25,569 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433600 2023-11-24 15:37:38,631 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 750, loss[loss=0.06753, simple_loss=0.08481, pruned_loss=0.01324, audio_tagging_loss=0.01188, over 15654.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.09042, pruned_loss=0.01248, audio_tagging_loss=0.009109, over 2984767.35 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:37:43,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2890720.0, ans=0.125 2023-11-24 15:37:45,697 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.386e+01 8.622e+01 9.264e+01 1.020e+02 2.352e+02, threshold=1.853e+02, percent-clipped=1.0 2023-11-24 15:37:49,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2890786.6666666665, ans=0.125 2023-11-24 15:38:10,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2890853.3333333335, ans=0.0 2023-11-24 15:38:13,279 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2890853.3333333335, ans=0.125 2023-11-24 15:38:28,490 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433650 2023-11-24 15:38:38,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2890986.6666666665, ans=0.125 2023-11-24 15:38:40,583 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 800, loss[loss=0.05919, simple_loss=0.07801, pruned_loss=0.01109, audio_tagging_loss=0.009095, over 14533.00 frames. ], tot_loss[loss=0.06634, simple_loss=0.0896, pruned_loss=0.01244, audio_tagging_loss=0.009106, over 2992125.33 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:38:46,675 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.12 vs. limit=10.0 2023-11-24 15:38:57,657 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:39:11,909 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2891186.6666666665, ans=0.0 2023-11-24 15:39:26,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2891253.3333333335, ans=0.125 2023-11-24 15:39:31,226 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433700 2023-11-24 15:39:32,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2891320.0, ans=0.0 2023-11-24 15:39:43,391 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 850, loss[loss=0.09317, simple_loss=0.1373, pruned_loss=0.01804, audio_tagging_loss=0.006503, over 16193.00 frames. ], tot_loss[loss=0.06592, simple_loss=0.08897, pruned_loss=0.01227, audio_tagging_loss=0.009172, over 3008125.61 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:39:51,002 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.494e+01 9.223e+01 9.765e+01 1.207e+02, threshold=1.845e+02, percent-clipped=0.0 2023-11-24 15:39:57,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2891453.3333333335, ans=0.125 2023-11-24 15:40:30,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2891586.6666666665, ans=0.1 2023-11-24 15:40:32,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2891653.3333333335, ans=0.0 2023-11-24 15:40:33,387 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433750 2023-11-24 15:40:33,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2891653.3333333335, ans=0.0 2023-11-24 15:40:45,661 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 900, loss[loss=0.0577, simple_loss=0.06552, pruned_loss=0.01168, audio_tagging_loss=0.01325, over 14426.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08937, pruned_loss=0.01245, audio_tagging_loss=0.009312, over 3019574.49 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:40:47,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2891720.0, ans=0.2 2023-11-24 15:40:58,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2891786.6666666665, ans=10.0 2023-11-24 15:41:00,691 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2891786.6666666665, ans=0.1 2023-11-24 15:41:03,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2891786.6666666665, ans=0.1 2023-11-24 15:41:31,752 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2891920.0, ans=0.0 2023-11-24 15:41:35,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433800 2023-11-24 15:41:48,028 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 950, loss[loss=0.0666, simple_loss=0.08852, pruned_loss=0.01257, audio_tagging_loss=0.009762, over 15175.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09048, pruned_loss=0.01269, audio_tagging_loss=0.009154, over 3027802.59 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:41:57,422 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.340e+01 8.595e+01 9.194e+01 9.952e+01 1.132e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 15:41:58,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2892053.3333333335, ans=0.125 2023-11-24 15:42:02,820 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.47 vs. limit=12.0 2023-11-24 15:42:23,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2892186.6666666665, ans=0.125 2023-11-24 15:42:38,331 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433850 2023-11-24 15:42:40,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2892320.0, ans=0.0 2023-11-24 15:42:40,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2892320.0, ans=0.0 2023-11-24 15:42:51,329 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1000, loss[loss=0.06798, simple_loss=0.09043, pruned_loss=0.01475, audio_tagging_loss=0.008022, over 15798.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09089, pruned_loss=0.01297, audio_tagging_loss=0.008995, over 3030124.24 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:43:01,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2892386.6666666665, ans=0.1 2023-11-24 15:43:17,455 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 15:43:38,731 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2892586.6666666665, ans=0.125 2023-11-24 15:43:40,853 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433900 2023-11-24 15:43:48,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2892653.3333333335, ans=0.125 2023-11-24 15:43:52,917 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1050, loss[loss=0.06118, simple_loss=0.08367, pruned_loss=0.01178, audio_tagging_loss=0.007567, over 15545.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.09041, pruned_loss=0.01284, audio_tagging_loss=0.008915, over 3031684.64 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:44:01,782 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.504e+01 9.256e+01 1.007e+02 1.365e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 15:44:02,201 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2892720.0, ans=0.0 2023-11-24 15:44:29,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2892920.0, ans=0.0 2023-11-24 15:44:42,852 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 433950 2023-11-24 15:44:51,112 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0 2023-11-24 15:44:55,278 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1100, loss[loss=0.08339, simple_loss=0.1181, pruned_loss=0.01776, audio_tagging_loss=0.006572, over 15053.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09033, pruned_loss=0.01291, audio_tagging_loss=0.008863, over 3038276.17 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:44:57,704 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 15:45:19,352 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.74 vs. limit=15.0 2023-11-24 15:45:33,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.72 vs. limit=6.0 2023-11-24 15:45:34,301 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.72 vs. limit=15.0 2023-11-24 15:45:40,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=2893253.3333333335, ans=15.0 2023-11-24 15:45:45,340 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434000 2023-11-24 15:45:58,261 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1150, loss[loss=0.05009, simple_loss=0.05825, pruned_loss=0.01026, audio_tagging_loss=0.0107, over 14103.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09082, pruned_loss=0.01299, audio_tagging_loss=0.008762, over 3040468.26 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:46:06,383 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.608e+01 9.158e+01 9.709e+01 1.539e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 15:46:21,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=13.87 vs. limit=22.5 2023-11-24 15:46:26,803 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.89 vs. limit=15.0 2023-11-24 15:46:38,586 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.59 vs. limit=15.0 2023-11-24 15:46:48,194 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434050 2023-11-24 15:47:00,399 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1200, loss[loss=0.06682, simple_loss=0.09631, pruned_loss=0.0105, audio_tagging_loss=0.008172, over 14496.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09147, pruned_loss=0.01306, audio_tagging_loss=0.008735, over 3040449.29 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:47:00,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2893720.0, ans=0.0 2023-11-24 15:47:19,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2893786.6666666665, ans=0.125 2023-11-24 15:47:50,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434100 2023-11-24 15:48:01,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2894053.3333333335, ans=0.2 2023-11-24 15:48:01,997 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1250, loss[loss=0.04774, simple_loss=0.0596, pruned_loss=0.007049, audio_tagging_loss=0.01089, over 14688.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09089, pruned_loss=0.01294, audio_tagging_loss=0.008718, over 3038615.27 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:48:05,256 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.73 vs. limit=15.0 2023-11-24 15:48:11,438 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.496e+01 8.998e+01 9.780e+01 2.107e+02, threshold=1.800e+02, percent-clipped=1.0 2023-11-24 15:48:28,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2894186.6666666665, ans=0.125 2023-11-24 15:48:29,282 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.37 vs. limit=22.5 2023-11-24 15:48:51,834 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434150 2023-11-24 15:49:00,857 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0 2023-11-24 15:49:05,246 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1300, loss[loss=0.07951, simple_loss=0.1053, pruned_loss=0.01702, audio_tagging_loss=0.009842, over 15469.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08965, pruned_loss=0.01278, audio_tagging_loss=0.008861, over 3037191.70 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:49:13,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2894386.6666666665, ans=0.025 2023-11-24 15:49:22,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2894453.3333333335, ans=0.125 2023-11-24 15:49:54,818 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434200 2023-11-24 15:50:07,483 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1350, loss[loss=0.06582, simple_loss=0.08335, pruned_loss=0.01374, audio_tagging_loss=0.01041, over 15004.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08996, pruned_loss=0.01303, audio_tagging_loss=0.008896, over 3036705.39 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:50:15,716 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.208e+01 8.839e+01 9.831e+01 1.172e+02, threshold=1.768e+02, percent-clipped=0.0 2023-11-24 15:50:33,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2894853.3333333335, ans=0.5 2023-11-24 15:50:43,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2894920.0, ans=0.125 2023-11-24 15:50:47,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2894920.0, ans=0.125 2023-11-24 15:50:51,220 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 15:50:56,506 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434250 2023-11-24 15:50:57,107 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.72 vs. limit=15.0 2023-11-24 15:51:04,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.whiten.whitening_limit, batch_count=2894986.6666666665, ans=15.0 2023-11-24 15:51:07,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2895053.3333333335, ans=0.0 2023-11-24 15:51:08,232 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1400, loss[loss=0.08544, simple_loss=0.1148, pruned_loss=0.01803, audio_tagging_loss=0.009991, over 15434.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.08959, pruned_loss=0.01295, audio_tagging_loss=0.009032, over 3038736.17 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:51:11,319 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.23 vs. limit=15.0 2023-11-24 15:51:23,057 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.44 vs. limit=12.0 2023-11-24 15:51:43,648 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:51:48,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2895253.3333333335, ans=0.0 2023-11-24 15:51:49,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2895253.3333333335, ans=0.125 2023-11-24 15:51:54,226 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2895253.3333333335, ans=0.2 2023-11-24 15:51:57,615 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434300 2023-11-24 15:52:05,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2895320.0, ans=0.0 2023-11-24 15:52:10,534 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1450, loss[loss=0.07509, simple_loss=0.1077, pruned_loss=0.01293, audio_tagging_loss=0.008296, over 14483.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09088, pruned_loss=0.01323, audio_tagging_loss=0.009052, over 3039570.66 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:52:13,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2895386.6666666665, ans=0.2 2023-11-24 15:52:20,309 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.122e+01 8.571e+01 9.242e+01 1.000e+02 1.352e+02, threshold=1.848e+02, percent-clipped=0.0 2023-11-24 15:52:39,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2895520.0, ans=0.125 2023-11-24 15:53:00,140 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434350 2023-11-24 15:53:01,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2895653.3333333335, ans=0.125 2023-11-24 15:53:05,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.41 vs. limit=22.5 2023-11-24 15:53:12,008 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1500, loss[loss=0.08008, simple_loss=0.1068, pruned_loss=0.01632, audio_tagging_loss=0.01035, over 14902.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09072, pruned_loss=0.01317, audio_tagging_loss=0.00918, over 3042349.17 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:53:13,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2895720.0, ans=0.0 2023-11-24 15:53:16,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2895720.0, ans=0.125 2023-11-24 15:53:21,268 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:53:35,466 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:53:44,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2895853.3333333335, ans=0.125 2023-11-24 15:53:50,714 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2895920.0, ans=0.125 2023-11-24 15:53:54,457 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0 2023-11-24 15:53:57,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2895920.0, ans=0.0 2023-11-24 15:54:01,274 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434400 2023-11-24 15:54:04,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2895986.6666666665, ans=0.125 2023-11-24 15:54:07,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2895986.6666666665, ans=0.125 2023-11-24 15:54:08,327 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:54:09,615 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2895986.6666666665, ans=0.125 2023-11-24 15:54:14,017 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1550, loss[loss=0.05725, simple_loss=0.06821, pruned_loss=0.01084, audio_tagging_loss=0.01231, over 15067.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09044, pruned_loss=0.01318, audio_tagging_loss=0.009102, over 3044797.34 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:54:23,511 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.499e+01 8.776e+01 9.350e+01 1.010e+02 1.250e+02, threshold=1.870e+02, percent-clipped=0.0 2023-11-24 15:54:44,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2896186.6666666665, ans=0.2 2023-11-24 15:54:59,756 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.25 vs. limit=15.0 2023-11-24 15:55:03,748 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434450 2023-11-24 15:55:05,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2896320.0, ans=0.125 2023-11-24 15:55:16,194 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1600, loss[loss=0.06335, simple_loss=0.08744, pruned_loss=0.01141, audio_tagging_loss=0.008218, over 14934.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09132, pruned_loss=0.01333, audio_tagging_loss=0.009162, over 3045118.47 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:55:28,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2896453.3333333335, ans=0.0 2023-11-24 15:55:30,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2896453.3333333335, ans=0.2 2023-11-24 15:55:38,851 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.38 vs. limit=15.0 2023-11-24 15:56:02,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2896586.6666666665, ans=0.0 2023-11-24 15:56:02,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2896586.6666666665, ans=0.125 2023-11-24 15:56:05,679 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434500 2023-11-24 15:56:07,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2896653.3333333335, ans=0.2 2023-11-24 15:56:08,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2896653.3333333335, ans=0.125 2023-11-24 15:56:16,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2896653.3333333335, ans=0.125 2023-11-24 15:56:18,320 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1650, loss[loss=0.06763, simple_loss=0.0807, pruned_loss=0.01757, audio_tagging_loss=0.009716, over 14804.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.0905, pruned_loss=0.01317, audio_tagging_loss=0.009197, over 3040730.09 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 15:56:28,157 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.503e+01 8.575e+01 9.140e+01 1.003e+02 1.202e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 15:56:55,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2896920.0, ans=0.2 2023-11-24 15:57:08,122 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434550 2023-11-24 15:57:20,453 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1700, loss[loss=0.07206, simple_loss=0.1046, pruned_loss=0.013, audio_tagging_loss=0.006736, over 15094.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09002, pruned_loss=0.01313, audio_tagging_loss=0.009202, over 3044835.42 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:57:24,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2897053.3333333335, ans=0.125 2023-11-24 15:57:36,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2897120.0, ans=0.5 2023-11-24 15:58:06,871 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.12 vs. limit=15.0 2023-11-24 15:58:10,031 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434600 2023-11-24 15:58:14,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2897320.0, ans=0.125 2023-11-24 15:58:22,746 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1750, loss[loss=0.06622, simple_loss=0.09439, pruned_loss=0.01272, audio_tagging_loss=0.006297, over 14798.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.08983, pruned_loss=0.01295, audio_tagging_loss=0.009077, over 3048188.57 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:58:22,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2897386.6666666665, ans=0.0 2023-11-24 15:58:33,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2897386.6666666665, ans=0.0 2023-11-24 15:58:33,877 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 8.384e+01 9.014e+01 9.814e+01 1.863e+02, threshold=1.803e+02, percent-clipped=1.0 2023-11-24 15:58:44,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2897453.3333333335, ans=0.5 2023-11-24 15:58:55,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2897520.0, ans=0.0 2023-11-24 15:58:59,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2897586.6666666665, ans=0.0 2023-11-24 15:59:07,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2897586.6666666665, ans=0.2 2023-11-24 15:59:12,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434650 2023-11-24 15:59:12,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2897653.3333333335, ans=0.125 2023-11-24 15:59:13,006 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.27 vs. limit=6.0 2023-11-24 15:59:22,766 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 15:59:24,849 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1800, loss[loss=0.08232, simple_loss=0.1205, pruned_loss=0.01151, audio_tagging_loss=0.01054, over 15550.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.08969, pruned_loss=0.01294, audio_tagging_loss=0.00898, over 3041079.18 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 15:59:35,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2897720.0, ans=0.0 2023-11-24 15:59:41,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2897786.6666666665, ans=0.125 2023-11-24 15:59:52,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2897853.3333333335, ans=0.125 2023-11-24 16:00:15,185 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434700 2023-11-24 16:00:15,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2897986.6666666665, ans=0.125 2023-11-24 16:00:17,675 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2897986.6666666665, ans=0.125 2023-11-24 16:00:25,329 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2897986.6666666665, ans=0.125 2023-11-24 16:00:27,544 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1850, loss[loss=0.07141, simple_loss=0.1104, pruned_loss=0.009668, audio_tagging_loss=0.006555, over 14414.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09031, pruned_loss=0.01309, audio_tagging_loss=0.008932, over 3041551.52 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:00:32,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2898053.3333333335, ans=0.2 2023-11-24 16:00:38,270 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.639e+01 8.698e+01 9.278e+01 9.936e+01 1.415e+02, threshold=1.856e+02, percent-clipped=0.0 2023-11-24 16:00:56,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2898186.6666666665, ans=0.125 2023-11-24 16:01:06,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.35 vs. limit=6.0 2023-11-24 16:01:10,199 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.14 vs. limit=22.5 2023-11-24 16:01:17,895 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434750 2023-11-24 16:01:29,991 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1900, loss[loss=0.0583, simple_loss=0.08294, pruned_loss=0.01148, audio_tagging_loss=0.005349, over 15211.00 frames. ], tot_loss[loss=0.06649, simple_loss=0.08942, pruned_loss=0.01274, audio_tagging_loss=0.009043, over 3042170.47 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:01:35,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2898386.6666666665, ans=0.0 2023-11-24 16:01:42,301 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.55 vs. limit=15.0 2023-11-24 16:01:49,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2898453.3333333335, ans=0.0 2023-11-24 16:02:20,237 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434800 2023-11-24 16:02:32,978 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 1950, loss[loss=0.0512, simple_loss=0.0622, pruned_loss=0.01101, audio_tagging_loss=0.009094, over 14988.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.08984, pruned_loss=0.01284, audio_tagging_loss=0.008974, over 3046924.60 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:02:34,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2898720.0, ans=0.0 2023-11-24 16:02:43,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2898720.0, ans=0.0 2023-11-24 16:02:44,688 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.590e+01 9.514e+01 1.026e+02 1.248e+02, threshold=1.903e+02, percent-clipped=0.0 2023-11-24 16:02:55,671 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2898786.6666666665, ans=0.125 2023-11-24 16:03:22,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434850 2023-11-24 16:03:27,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2898986.6666666665, ans=0.025 2023-11-24 16:03:35,533 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2000, loss[loss=0.06903, simple_loss=0.09852, pruned_loss=0.01207, audio_tagging_loss=0.007706, over 15716.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.08996, pruned_loss=0.01292, audio_tagging_loss=0.008967, over 3039182.73 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 16:03:48,878 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2899120.0, ans=0.2 2023-11-24 16:03:51,253 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2899120.0, ans=0.125 2023-11-24 16:04:25,327 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434900 2023-11-24 16:04:36,759 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2050, loss[loss=0.08235, simple_loss=0.1174, pruned_loss=0.01766, audio_tagging_loss=0.005964, over 15527.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.0902, pruned_loss=0.01292, audio_tagging_loss=0.008846, over 3034750.39 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:04:42,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2899386.6666666665, ans=0.125 2023-11-24 16:04:45,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2899386.6666666665, ans=0.0 2023-11-24 16:04:49,655 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.636e+01 9.159e+01 9.953e+01 1.239e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 16:04:54,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2899453.3333333335, ans=0.125 2023-11-24 16:04:58,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2899453.3333333335, ans=0.95 2023-11-24 16:05:07,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2899520.0, ans=0.2 2023-11-24 16:05:10,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2899520.0, ans=0.0 2023-11-24 16:05:16,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2899586.6666666665, ans=0.0 2023-11-24 16:05:22,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2899586.6666666665, ans=0.125 2023-11-24 16:05:26,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2899653.3333333335, ans=10.0 2023-11-24 16:05:27,057 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 434950 2023-11-24 16:05:39,837 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2100, loss[loss=0.0669, simple_loss=0.08953, pruned_loss=0.01409, audio_tagging_loss=0.008045, over 14803.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09087, pruned_loss=0.01299, audio_tagging_loss=0.008835, over 3033859.24 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:05:56,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2899786.6666666665, ans=0.125 2023-11-24 16:06:01,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2899786.6666666665, ans=0.0 2023-11-24 16:06:07,503 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2899853.3333333335, ans=0.025 2023-11-24 16:06:10,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2899853.3333333335, ans=0.2 2023-11-24 16:06:29,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435000 2023-11-24 16:06:42,425 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2150, loss[loss=0.08441, simple_loss=0.1187, pruned_loss=0.01825, audio_tagging_loss=0.006812, over 16552.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09126, pruned_loss=0.01318, audio_tagging_loss=0.00884, over 3042258.32 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:06:43,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2900053.3333333335, ans=0.1 2023-11-24 16:06:54,775 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.565e+01 9.008e+01 9.643e+01 1.315e+02, threshold=1.802e+02, percent-clipped=0.0 2023-11-24 16:07:01,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2900120.0, ans=0.0 2023-11-24 16:07:07,530 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2900186.6666666665, ans=0.1 2023-11-24 16:07:09,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.71 vs. limit=22.5 2023-11-24 16:07:15,383 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2900186.6666666665, ans=0.09899494936611666 2023-11-24 16:07:19,834 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:07:32,301 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435050 2023-11-24 16:07:32,465 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2900320.0, ans=0.125 2023-11-24 16:07:33,812 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 16:07:44,814 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2200, loss[loss=0.05647, simple_loss=0.07427, pruned_loss=0.01025, audio_tagging_loss=0.009086, over 16520.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09122, pruned_loss=0.01305, audio_tagging_loss=0.008895, over 3039462.51 frames. ], batch size: 66, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:08:22,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2900586.6666666665, ans=0.0 2023-11-24 16:08:33,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.76 vs. limit=15.0 2023-11-24 16:08:34,460 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435100 2023-11-24 16:08:47,280 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2250, loss[loss=0.06978, simple_loss=0.09895, pruned_loss=0.01205, audio_tagging_loss=0.008252, over 15334.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09164, pruned_loss=0.01303, audio_tagging_loss=0.008792, over 3039817.14 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:08:59,270 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.499e+01 9.314e+01 1.008e+02 1.259e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 16:09:13,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2900853.3333333335, ans=0.125 2023-11-24 16:09:20,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.30 vs. limit=12.0 2023-11-24 16:09:37,553 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435150 2023-11-24 16:09:37,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2900986.6666666665, ans=0.0 2023-11-24 16:09:41,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2900986.6666666665, ans=0.2 2023-11-24 16:09:49,111 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2300, loss[loss=0.08085, simple_loss=0.1173, pruned_loss=0.01599, audio_tagging_loss=0.006198, over 15398.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09121, pruned_loss=0.0131, audio_tagging_loss=0.008885, over 3043045.53 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:10:17,278 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0 2023-11-24 16:10:39,101 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435200 2023-11-24 16:10:40,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2901320.0, ans=0.1 2023-11-24 16:10:44,194 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:10:51,720 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2350, loss[loss=0.06316, simple_loss=0.08194, pruned_loss=0.0144, audio_tagging_loss=0.007798, over 14811.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09027, pruned_loss=0.01289, audio_tagging_loss=0.008949, over 3039738.87 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:11:04,102 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.491e+01 9.098e+01 9.735e+01 1.165e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-24 16:11:08,795 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.30 vs. limit=15.0 2023-11-24 16:11:11,098 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.31 vs. limit=15.0 2023-11-24 16:11:28,317 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.46 vs. limit=22.5 2023-11-24 16:11:33,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2901586.6666666665, ans=0.125 2023-11-24 16:11:35,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2901586.6666666665, ans=0.0 2023-11-24 16:11:41,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435250 2023-11-24 16:11:41,825 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.43 vs. limit=15.0 2023-11-24 16:11:53,581 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2400, loss[loss=0.06654, simple_loss=0.08448, pruned_loss=0.014, audio_tagging_loss=0.0103, over 14591.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.08995, pruned_loss=0.01282, audio_tagging_loss=0.009071, over 3036488.90 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:11:58,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2901720.0, ans=0.04949747468305833 2023-11-24 16:12:06,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2901786.6666666665, ans=0.0 2023-11-24 16:12:21,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2901853.3333333335, ans=0.125 2023-11-24 16:12:27,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2901853.3333333335, ans=0.125 2023-11-24 16:12:28,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2901853.3333333335, ans=0.1 2023-11-24 16:12:35,412 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 16:12:44,410 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435300 2023-11-24 16:12:48,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2901986.6666666665, ans=0.0 2023-11-24 16:12:56,205 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2450, loss[loss=0.06326, simple_loss=0.08364, pruned_loss=0.0111, audio_tagging_loss=0.01034, over 15586.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.0905, pruned_loss=0.01313, audio_tagging_loss=0.009153, over 3037250.50 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:13:09,834 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.445e+01 8.651e+01 9.072e+01 9.865e+01 1.248e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 16:13:17,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2902120.0, ans=0.0 2023-11-24 16:13:19,106 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2902120.0, ans=0.125 2023-11-24 16:13:20,369 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2902186.6666666665, ans=0.1 2023-11-24 16:13:46,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435350 2023-11-24 16:13:58,424 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2500, loss[loss=0.06516, simple_loss=0.09307, pruned_loss=0.008515, audio_tagging_loss=0.01011, over 14631.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.08961, pruned_loss=0.01279, audio_tagging_loss=0.009234, over 3043669.69 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:14:06,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2902386.6666666665, ans=0.125 2023-11-24 16:14:15,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2902453.3333333335, ans=0.1 2023-11-24 16:14:24,302 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=2902520.0, ans=0.2 2023-11-24 16:14:28,208 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.20 vs. limit=6.0 2023-11-24 16:14:34,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2902520.0, ans=0.0 2023-11-24 16:14:48,516 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435400 2023-11-24 16:14:58,368 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.76 vs. limit=15.0 2023-11-24 16:15:01,953 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2550, loss[loss=0.07409, simple_loss=0.1049, pruned_loss=0.01574, audio_tagging_loss=0.005898, over 16367.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09011, pruned_loss=0.01303, audio_tagging_loss=0.009089, over 3044293.08 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:15:15,544 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.436e+01 8.712e+01 9.254e+01 9.868e+01 2.546e+02, threshold=1.851e+02, percent-clipped=1.0 2023-11-24 16:15:18,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2902786.6666666665, ans=0.125 2023-11-24 16:15:32,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2902853.3333333335, ans=0.125 2023-11-24 16:15:40,154 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.00 vs. limit=6.0 2023-11-24 16:15:51,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435450 2023-11-24 16:15:52,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2902986.6666666665, ans=0.125 2023-11-24 16:16:03,760 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2600, loss[loss=0.04283, simple_loss=0.0512, pruned_loss=0.005973, audio_tagging_loss=0.01126, over 16857.00 frames. ], tot_loss[loss=0.06617, simple_loss=0.08882, pruned_loss=0.01272, audio_tagging_loss=0.00903, over 3050499.39 frames. ], batch size: 67, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:16:08,842 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2903053.3333333335, ans=0.125 2023-11-24 16:16:25,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2903120.0, ans=0.1 2023-11-24 16:16:25,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2903120.0, ans=0.07 2023-11-24 16:16:45,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2903253.3333333335, ans=0.07 2023-11-24 16:16:48,689 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2903253.3333333335, ans=0.2 2023-11-24 16:16:53,226 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435500 2023-11-24 16:17:05,541 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2650, loss[loss=0.05635, simple_loss=0.07707, pruned_loss=0.007458, audio_tagging_loss=0.01036, over 14268.00 frames. ], tot_loss[loss=0.06635, simple_loss=0.08925, pruned_loss=0.01277, audio_tagging_loss=0.008951, over 3049126.74 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:17:10,720 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.53 vs. limit=15.0 2023-11-24 16:17:18,503 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.439e+01 8.953e+01 1.003e+02 3.059e+02, threshold=1.791e+02, percent-clipped=1.0 2023-11-24 16:17:21,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2903453.3333333335, ans=0.125 2023-11-24 16:17:54,380 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435550 2023-11-24 16:17:55,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2903653.3333333335, ans=0.2 2023-11-24 16:17:57,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.77 vs. limit=15.0 2023-11-24 16:18:06,109 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2700, loss[loss=0.05561, simple_loss=0.07674, pruned_loss=0.007805, audio_tagging_loss=0.009432, over 16228.00 frames. ], tot_loss[loss=0.06604, simple_loss=0.0891, pruned_loss=0.01267, audio_tagging_loss=0.008822, over 3052247.95 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:18:10,921 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2903720.0, ans=0.125 2023-11-24 16:18:13,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2903720.0, ans=0.0 2023-11-24 16:18:27,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2903786.6666666665, ans=0.1 2023-11-24 16:18:34,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2903853.3333333335, ans=0.0 2023-11-24 16:18:38,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.79 vs. limit=15.0 2023-11-24 16:18:56,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435600 2023-11-24 16:19:09,899 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2750, loss[loss=0.04287, simple_loss=0.05781, pruned_loss=0.003137, audio_tagging_loss=0.01082, over 14790.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09007, pruned_loss=0.01308, audio_tagging_loss=0.008855, over 3052988.80 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 8.0 2023-11-24 16:19:24,582 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.485e+01 8.630e+01 9.216e+01 9.892e+01 1.188e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 16:19:27,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2904120.0, ans=0.125 2023-11-24 16:19:47,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2904253.3333333335, ans=0.1 2023-11-24 16:19:55,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2904253.3333333335, ans=0.125 2023-11-24 16:19:59,730 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435650 2023-11-24 16:20:02,113 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:20:12,154 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2800, loss[loss=0.07572, simple_loss=0.1017, pruned_loss=0.01699, audio_tagging_loss=0.007898, over 15658.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.0897, pruned_loss=0.013, audio_tagging_loss=0.00881, over 3050474.09 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:20:18,424 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2904386.6666666665, ans=0.1 2023-11-24 16:20:25,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2904453.3333333335, ans=0.125 2023-11-24 16:20:29,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2904453.3333333335, ans=0.125 2023-11-24 16:20:31,999 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2904453.3333333335, ans=0.0 2023-11-24 16:20:37,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2904520.0, ans=0.125 2023-11-24 16:20:50,490 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2904586.6666666665, ans=0.2 2023-11-24 16:21:01,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435700 2023-11-24 16:21:10,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2904653.3333333335, ans=0.125 2023-11-24 16:21:13,368 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2850, loss[loss=0.05501, simple_loss=0.0757, pruned_loss=0.008675, audio_tagging_loss=0.008484, over 14630.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.08955, pruned_loss=0.01291, audio_tagging_loss=0.0089, over 3046950.83 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:21:16,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2904720.0, ans=0.2 2023-11-24 16:21:22,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2904720.0, ans=0.07 2023-11-24 16:21:28,569 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 16:21:29,327 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.333e+01 9.111e+01 9.785e+01 2.308e+02, threshold=1.822e+02, percent-clipped=1.0 2023-11-24 16:21:37,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.01 vs. limit=6.0 2023-11-24 16:21:49,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2904853.3333333335, ans=0.1 2023-11-24 16:21:51,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2904920.0, ans=0.2 2023-11-24 16:21:55,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2904920.0, ans=0.125 2023-11-24 16:21:57,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2904920.0, ans=0.125 2023-11-24 16:22:03,266 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435750 2023-11-24 16:22:08,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2904986.6666666665, ans=0.0 2023-11-24 16:22:10,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2904986.6666666665, ans=0.0 2023-11-24 16:22:16,935 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2900, loss[loss=0.07845, simple_loss=0.1006, pruned_loss=0.01915, audio_tagging_loss=0.008985, over 15734.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09032, pruned_loss=0.01294, audio_tagging_loss=0.008921, over 3052708.79 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:22:27,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2905053.3333333335, ans=0.125 2023-11-24 16:23:05,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2905320.0, ans=0.09899494936611666 2023-11-24 16:23:06,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435800 2023-11-24 16:23:19,021 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 2950, loss[loss=0.07297, simple_loss=0.09555, pruned_loss=0.01633, audio_tagging_loss=0.008855, over 15742.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09174, pruned_loss=0.01313, audio_tagging_loss=0.008876, over 3051310.99 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:23:21,874 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.46 vs. limit=15.0 2023-11-24 16:23:28,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2905386.6666666665, ans=0.0 2023-11-24 16:23:33,479 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.426e+01 9.166e+01 9.683e+01 1.219e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 16:23:44,066 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2905520.0, ans=0.125 2023-11-24 16:23:52,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2905520.0, ans=0.2 2023-11-24 16:23:54,871 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.18 vs. limit=22.5 2023-11-24 16:24:02,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2905586.6666666665, ans=0.125 2023-11-24 16:24:09,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435850 2023-11-24 16:24:09,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2905653.3333333335, ans=0.0 2023-11-24 16:24:16,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2905653.3333333335, ans=0.0 2023-11-24 16:24:21,011 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3000, loss[loss=0.05079, simple_loss=0.06827, pruned_loss=0.008184, audio_tagging_loss=0.008474, over 15627.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09202, pruned_loss=0.01325, audio_tagging_loss=0.008938, over 3058068.25 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:24:21,011 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 16:24:46,636 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.5109, 3.3968, 3.7376, 3.5747], device='cuda:3') 2023-11-24 16:25:01,962 INFO [train_asr.py:1253] (3/4) Epoch 37, validation: loss=0.05757, simple_loss=0.05085, pruned_loss=0.005185, audio_tagging_loss=0.02697, over 4681554.00 frames. 2023-11-24 16:25:01,962 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 16:25:18,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2905786.6666666665, ans=0.125 2023-11-24 16:25:18,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2905786.6666666665, ans=0.2 2023-11-24 16:25:18,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2905786.6666666665, ans=0.0 2023-11-24 16:25:24,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2905786.6666666665, ans=0.125 2023-11-24 16:25:36,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2905853.3333333335, ans=0.125 2023-11-24 16:25:37,028 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.77 vs. limit=6.0 2023-11-24 16:25:44,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2905920.0, ans=0.0 2023-11-24 16:25:51,911 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435900 2023-11-24 16:25:55,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2905986.6666666665, ans=0.2 2023-11-24 16:26:04,082 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3050, loss[loss=0.07367, simple_loss=0.101, pruned_loss=0.01439, audio_tagging_loss=0.008776, over 14953.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09171, pruned_loss=0.01307, audio_tagging_loss=0.009039, over 3052806.42 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:26:17,328 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 16:26:18,211 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.443e+01 9.096e+01 9.825e+01 1.321e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 16:26:22,876 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2906120.0, ans=0.125 2023-11-24 16:26:26,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2906120.0, ans=0.2 2023-11-24 16:26:32,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2906186.6666666665, ans=0.125 2023-11-24 16:26:40,227 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:26:54,069 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 435950 2023-11-24 16:26:54,165 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2906320.0, ans=0.0 2023-11-24 16:27:05,782 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3100, loss[loss=0.05923, simple_loss=0.0815, pruned_loss=0.009794, audio_tagging_loss=0.008685, over 14932.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09234, pruned_loss=0.01327, audio_tagging_loss=0.009027, over 3045421.35 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:27:09,673 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2906386.6666666665, ans=0.125 2023-11-24 16:27:10,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2906386.6666666665, ans=0.125 2023-11-24 16:27:22,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2906453.3333333335, ans=0.0 2023-11-24 16:27:52,744 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.48 vs. limit=15.0 2023-11-24 16:27:55,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436000 2023-11-24 16:28:08,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2906653.3333333335, ans=0.125 2023-11-24 16:28:10,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2906720.0, ans=0.0 2023-11-24 16:28:12,031 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3150, loss[loss=0.06484, simple_loss=0.08338, pruned_loss=0.01622, audio_tagging_loss=0.006926, over 15467.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09189, pruned_loss=0.01314, audio_tagging_loss=0.009064, over 3042782.21 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:28:13,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2906720.0, ans=0.0 2023-11-24 16:28:14,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2906720.0, ans=0.2 2023-11-24 16:28:27,193 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.617e+01 9.268e+01 9.907e+01 1.441e+02, threshold=1.854e+02, percent-clipped=0.0 2023-11-24 16:28:35,872 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2906853.3333333335, ans=0.125 2023-11-24 16:28:46,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2906853.3333333335, ans=0.2 2023-11-24 16:28:48,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2906920.0, ans=0.0 2023-11-24 16:29:01,666 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436050 2023-11-24 16:29:04,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2906986.6666666665, ans=0.125 2023-11-24 16:29:05,480 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2906986.6666666665, ans=0.1 2023-11-24 16:29:12,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2906986.6666666665, ans=0.125 2023-11-24 16:29:14,528 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3200, loss[loss=0.05477, simple_loss=0.0735, pruned_loss=0.01001, audio_tagging_loss=0.008008, over 15169.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09168, pruned_loss=0.01314, audio_tagging_loss=0.009143, over 3035327.42 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 16:30:04,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436100 2023-11-24 16:30:09,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2907320.0, ans=0.0 2023-11-24 16:30:16,185 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3250, loss[loss=0.06408, simple_loss=0.07676, pruned_loss=0.01403, audio_tagging_loss=0.01167, over 15222.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09157, pruned_loss=0.01321, audio_tagging_loss=0.009194, over 3039560.90 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 16:30:28,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0 2023-11-24 16:30:31,298 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.491e+01 9.014e+01 9.810e+01 1.302e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-24 16:30:35,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2907453.3333333335, ans=0.2 2023-11-24 16:30:44,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2907520.0, ans=0.125 2023-11-24 16:31:05,845 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436150 2023-11-24 16:31:08,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2907653.3333333335, ans=0.2 2023-11-24 16:31:18,083 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3300, loss[loss=0.07731, simple_loss=0.09934, pruned_loss=0.01693, audio_tagging_loss=0.01071, over 16212.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09155, pruned_loss=0.01321, audio_tagging_loss=0.009255, over 3048488.06 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 16:31:38,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.47 vs. limit=10.0 2023-11-24 16:31:41,250 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.12 vs. limit=10.0 2023-11-24 16:31:47,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2907853.3333333335, ans=0.0 2023-11-24 16:32:07,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436200 2023-11-24 16:32:21,580 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3350, loss[loss=0.0641, simple_loss=0.08181, pruned_loss=0.01375, audio_tagging_loss=0.009449, over 16153.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09155, pruned_loss=0.01314, audio_tagging_loss=0.009206, over 3045808.72 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 16:32:34,166 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0 2023-11-24 16:32:35,789 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.766e+01 9.367e+01 1.008e+02 1.183e+02, threshold=1.873e+02, percent-clipped=0.0 2023-11-24 16:32:46,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2908186.6666666665, ans=0.125 2023-11-24 16:32:58,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2908253.3333333335, ans=0.125 2023-11-24 16:33:11,484 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436250 2023-11-24 16:33:14,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.28 vs. limit=22.5 2023-11-24 16:33:23,273 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3400, loss[loss=0.06574, simple_loss=0.08807, pruned_loss=0.01316, audio_tagging_loss=0.008545, over 14150.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09115, pruned_loss=0.01298, audio_tagging_loss=0.009024, over 3043864.85 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0 2023-11-24 16:33:41,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2908453.3333333335, ans=0.125 2023-11-24 16:34:13,204 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436300 2023-11-24 16:34:13,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2908653.3333333335, ans=0.125 2023-11-24 16:34:26,327 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3450, loss[loss=0.06644, simple_loss=0.08972, pruned_loss=0.01273, audio_tagging_loss=0.008855, over 15645.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.0908, pruned_loss=0.01292, audio_tagging_loss=0.008966, over 3046482.70 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:34:37,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2908786.6666666665, ans=0.95 2023-11-24 16:34:39,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2908786.6666666665, ans=0.125 2023-11-24 16:34:42,910 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.653e+01 9.241e+01 9.942e+01 2.012e+02, threshold=1.848e+02, percent-clipped=1.0 2023-11-24 16:35:16,733 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436350 2023-11-24 16:35:29,772 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3500, loss[loss=0.07439, simple_loss=0.1059, pruned_loss=0.01491, audio_tagging_loss=0.006535, over 14247.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09078, pruned_loss=0.01298, audio_tagging_loss=0.008888, over 3046548.98 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0 2023-11-24 16:35:30,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2909053.3333333335, ans=0.0 2023-11-24 16:35:51,139 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 16:36:00,995 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:36:12,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2909253.3333333335, ans=0.2 2023-11-24 16:36:19,198 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2909320.0, ans=0.125 2023-11-24 16:36:20,161 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436400 2023-11-24 16:36:33,049 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3550, loss[loss=0.081, simple_loss=0.1119, pruned_loss=0.01746, audio_tagging_loss=0.007573, over 15463.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09127, pruned_loss=0.0131, audio_tagging_loss=0.008849, over 3055350.13 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:36:37,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2909386.6666666665, ans=0.2 2023-11-24 16:36:49,076 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.431e+01 8.760e+01 9.470e+01 1.011e+02 1.264e+02, threshold=1.894e+02, percent-clipped=0.0 2023-11-24 16:36:49,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2909453.3333333335, ans=0.125 2023-11-24 16:37:01,068 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.57 vs. limit=5.0 2023-11-24 16:37:01,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2909520.0, ans=0.1 2023-11-24 16:37:03,354 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.02 vs. limit=22.5 2023-11-24 16:37:22,969 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436450 2023-11-24 16:37:35,218 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3600, loss[loss=0.05954, simple_loss=0.07792, pruned_loss=0.01137, audio_tagging_loss=0.009214, over 14310.00 frames. ], tot_loss[loss=0.06636, simple_loss=0.08955, pruned_loss=0.01264, audio_tagging_loss=0.008944, over 3048941.46 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 16:37:43,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2909720.0, ans=0.0 2023-11-24 16:37:50,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2909786.6666666665, ans=0.05 2023-11-24 16:37:51,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2909786.6666666665, ans=0.1 2023-11-24 16:37:59,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2909853.3333333335, ans=0.125 2023-11-24 16:38:05,321 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2909853.3333333335, ans=0.1 2023-11-24 16:38:17,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2909920.0, ans=0.0 2023-11-24 16:38:24,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2909986.6666666665, ans=0.0 2023-11-24 16:38:25,041 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436500 2023-11-24 16:38:33,533 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0 2023-11-24 16:38:37,892 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3650, loss[loss=0.0701, simple_loss=0.09122, pruned_loss=0.01523, audio_tagging_loss=0.009263, over 15215.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09063, pruned_loss=0.01285, audio_tagging_loss=0.00885, over 3044324.46 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 16:38:46,585 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2910053.3333333335, ans=0.5 2023-11-24 16:38:54,063 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.295e+01 9.068e+01 9.649e+01 1.086e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 16:38:59,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2910120.0, ans=0.1 2023-11-24 16:39:00,501 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.10 vs. limit=22.5 2023-11-24 16:39:27,861 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436550 2023-11-24 16:39:28,103 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2910320.0, ans=0.125 2023-11-24 16:39:38,752 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.53 vs. limit=22.5 2023-11-24 16:39:39,415 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3700, loss[loss=0.05513, simple_loss=0.07819, pruned_loss=0.007714, audio_tagging_loss=0.008321, over 14315.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09044, pruned_loss=0.01286, audio_tagging_loss=0.008901, over 3052166.40 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:39:56,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2910453.3333333335, ans=0.2 2023-11-24 16:40:17,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2910586.6666666665, ans=0.125 2023-11-24 16:40:19,488 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2910586.6666666665, ans=0.125 2023-11-24 16:40:20,685 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2910586.6666666665, ans=0.125 2023-11-24 16:40:29,932 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436600 2023-11-24 16:40:30,074 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2910653.3333333335, ans=0.0 2023-11-24 16:40:39,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2910653.3333333335, ans=0.0 2023-11-24 16:40:43,395 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3750, loss[loss=0.06713, simple_loss=0.09084, pruned_loss=0.01002, audio_tagging_loss=0.01169, over 16063.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09128, pruned_loss=0.01304, audio_tagging_loss=0.008986, over 3056462.03 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:40:47,630 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.74 vs. limit=15.0 2023-11-24 16:40:56,124 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2910786.6666666665, ans=0.125 2023-11-24 16:41:01,144 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.745e+01 9.267e+01 9.947e+01 1.281e+02, threshold=1.853e+02, percent-clipped=0.0 2023-11-24 16:41:04,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2910786.6666666665, ans=0.125 2023-11-24 16:41:07,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2910853.3333333335, ans=0.125 2023-11-24 16:41:09,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2910853.3333333335, ans=0.1 2023-11-24 16:41:21,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2910920.0, ans=0.125 2023-11-24 16:41:25,473 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:41:25,771 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2910920.0, ans=0.1 2023-11-24 16:41:28,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2910920.0, ans=0.125 2023-11-24 16:41:28,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2910920.0, ans=0.5 2023-11-24 16:41:33,349 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436650 2023-11-24 16:41:33,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2910986.6666666665, ans=0.125 2023-11-24 16:41:45,783 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3800, loss[loss=0.07285, simple_loss=0.09294, pruned_loss=0.01521, audio_tagging_loss=0.01117, over 14556.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09107, pruned_loss=0.01308, audio_tagging_loss=0.009007, over 3055115.40 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:42:18,487 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2911186.6666666665, ans=0.0 2023-11-24 16:42:23,207 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2911253.3333333335, ans=0.125 2023-11-24 16:42:26,772 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2911253.3333333335, ans=0.125 2023-11-24 16:42:36,220 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436700 2023-11-24 16:42:48,431 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3850, loss[loss=0.05229, simple_loss=0.06818, pruned_loss=0.006994, audio_tagging_loss=0.0112, over 15798.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09116, pruned_loss=0.01295, audio_tagging_loss=0.008957, over 3054135.08 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:42:51,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2911386.6666666665, ans=0.0 2023-11-24 16:42:52,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2911386.6666666665, ans=0.125 2023-11-24 16:43:06,192 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.966e+01 8.511e+01 9.199e+01 9.867e+01 1.160e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-24 16:43:10,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2911453.3333333335, ans=0.1 2023-11-24 16:43:15,490 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.93 vs. limit=15.0 2023-11-24 16:43:25,892 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2911586.6666666665, ans=0.125 2023-11-24 16:43:26,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2911586.6666666665, ans=0.125 2023-11-24 16:43:38,389 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436750 2023-11-24 16:43:43,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2911653.3333333335, ans=0.125 2023-11-24 16:43:48,579 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 16:43:50,757 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3900, loss[loss=0.0742, simple_loss=0.1019, pruned_loss=0.01489, audio_tagging_loss=0.008355, over 15597.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09063, pruned_loss=0.01291, audio_tagging_loss=0.009036, over 3049573.29 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:43:54,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2911720.0, ans=0.0 2023-11-24 16:43:54,238 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2911720.0, ans=0.125 2023-11-24 16:44:05,288 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2911786.6666666665, ans=0.0 2023-11-24 16:44:36,341 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.45 vs. limit=15.0 2023-11-24 16:44:40,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2911986.6666666665, ans=0.125 2023-11-24 16:44:41,525 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436800 2023-11-24 16:44:45,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2911986.6666666665, ans=0.125 2023-11-24 16:44:54,255 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 3950, loss[loss=0.07172, simple_loss=0.09851, pruned_loss=0.01501, audio_tagging_loss=0.007458, over 15714.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09075, pruned_loss=0.01293, audio_tagging_loss=0.009077, over 3045527.77 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:45:00,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.94 vs. limit=22.5 2023-11-24 16:45:11,320 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.639e+01 9.043e+01 9.863e+01 1.208e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-24 16:45:11,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2912120.0, ans=0.125 2023-11-24 16:45:37,258 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.51 vs. limit=15.0 2023-11-24 16:45:41,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2912253.3333333335, ans=0.0 2023-11-24 16:45:44,228 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436850 2023-11-24 16:45:54,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2912320.0, ans=0.0 2023-11-24 16:45:56,549 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4000, loss[loss=0.07337, simple_loss=0.09178, pruned_loss=0.01682, audio_tagging_loss=0.01066, over 13991.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09031, pruned_loss=0.01283, audio_tagging_loss=0.009187, over 3043831.37 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 16:46:31,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2912520.0, ans=0.125 2023-11-24 16:46:34,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2912586.6666666665, ans=0.125 2023-11-24 16:46:39,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2912586.6666666665, ans=10.0 2023-11-24 16:46:46,509 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436900 2023-11-24 16:46:47,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.76 vs. limit=6.0 2023-11-24 16:46:58,175 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4050, loss[loss=0.04613, simple_loss=0.06022, pruned_loss=0.006978, audio_tagging_loss=0.00904, over 14837.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09034, pruned_loss=0.01295, audio_tagging_loss=0.00922, over 3039968.97 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 16:47:00,529 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:47:07,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2912720.0, ans=0.125 2023-11-24 16:47:08,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2912720.0, ans=0.0 2023-11-24 16:47:12,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2912786.6666666665, ans=0.125 2023-11-24 16:47:16,514 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.690e+01 9.276e+01 1.004e+02 1.184e+02, threshold=1.855e+02, percent-clipped=0.0 2023-11-24 16:47:19,656 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.62 vs. limit=15.0 2023-11-24 16:47:32,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2912853.3333333335, ans=0.0 2023-11-24 16:47:47,928 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 436950 2023-11-24 16:47:58,614 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2912986.6666666665, ans=0.125 2023-11-24 16:48:01,375 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4100, loss[loss=0.09831, simple_loss=0.1305, pruned_loss=0.02469, audio_tagging_loss=0.008351, over 15882.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09068, pruned_loss=0.013, audio_tagging_loss=0.0092, over 3046216.75 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 16:48:51,499 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437000 2023-11-24 16:48:59,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2913320.0, ans=0.1 2023-11-24 16:49:04,132 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4150, loss[loss=0.07158, simple_loss=0.09361, pruned_loss=0.01522, audio_tagging_loss=0.009554, over 14779.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09131, pruned_loss=0.01316, audio_tagging_loss=0.009017, over 3056153.24 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:49:08,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2913386.6666666665, ans=0.125 2023-11-24 16:49:22,042 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.740e+01 9.379e+01 9.988e+01 1.190e+02, threshold=1.876e+02, percent-clipped=0.0 2023-11-24 16:49:40,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2913586.6666666665, ans=0.125 2023-11-24 16:49:47,795 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:49:53,883 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437050 2023-11-24 16:50:00,467 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.32 vs. limit=10.0 2023-11-24 16:50:03,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2913653.3333333335, ans=0.125 2023-11-24 16:50:05,831 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4200, loss[loss=0.07674, simple_loss=0.1066, pruned_loss=0.0164, audio_tagging_loss=0.007056, over 15638.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09202, pruned_loss=0.01326, audio_tagging_loss=0.008824, over 3048967.69 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:50:07,869 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0 2023-11-24 16:50:11,516 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.93 vs. limit=22.5 2023-11-24 16:50:13,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.89 vs. limit=15.0 2023-11-24 16:50:24,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2913786.6666666665, ans=0.07 2023-11-24 16:50:38,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2913853.3333333335, ans=0.0 2023-11-24 16:50:47,176 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.32 vs. limit=12.0 2023-11-24 16:50:56,132 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437100 2023-11-24 16:51:04,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2913986.6666666665, ans=0.125 2023-11-24 16:51:08,512 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4250, loss[loss=0.06752, simple_loss=0.08744, pruned_loss=0.01258, audio_tagging_loss=0.01123, over 13923.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.0914, pruned_loss=0.01301, audio_tagging_loss=0.008764, over 3047239.06 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:51:19,308 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2914053.3333333335, ans=0.0 2023-11-24 16:51:27,406 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.332e+01 8.707e+01 9.618e+01 1.026e+02 1.343e+02, threshold=1.924e+02, percent-clipped=0.0 2023-11-24 16:51:29,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2914120.0, ans=0.0 2023-11-24 16:51:30,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2914120.0, ans=15.0 2023-11-24 16:51:31,644 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.44 vs. limit=6.0 2023-11-24 16:51:53,046 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2914253.3333333335, ans=0.0 2023-11-24 16:51:57,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437150 2023-11-24 16:52:10,477 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4300, loss[loss=0.07145, simple_loss=0.09546, pruned_loss=0.01689, audio_tagging_loss=0.006829, over 14643.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09147, pruned_loss=0.01307, audio_tagging_loss=0.008788, over 3051694.38 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 16:52:15,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2914386.6666666665, ans=0.1 2023-11-24 16:52:46,477 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2914586.6666666665, ans=0.0 2023-11-24 16:52:50,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2914586.6666666665, ans=0.125 2023-11-24 16:52:54,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2914586.6666666665, ans=0.07 2023-11-24 16:53:00,522 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437200 2023-11-24 16:53:03,760 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.04 vs. limit=12.0 2023-11-24 16:53:11,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2914720.0, ans=0.125 2023-11-24 16:53:12,552 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4350, loss[loss=0.07287, simple_loss=0.08735, pruned_loss=0.01869, audio_tagging_loss=0.01051, over 16613.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09071, pruned_loss=0.01303, audio_tagging_loss=0.008837, over 3044032.81 frames. ], batch size: 64, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 16:53:27,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.59 vs. limit=22.5 2023-11-24 16:53:28,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2914786.6666666665, ans=0.125 2023-11-24 16:53:32,388 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.693e+01 9.378e+01 1.028e+02 1.311e+02, threshold=1.876e+02, percent-clipped=0.0 2023-11-24 16:53:36,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2914853.3333333335, ans=0.125 2023-11-24 16:53:38,732 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.28 vs. limit=5.0 2023-11-24 16:53:56,530 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0 2023-11-24 16:54:01,899 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437250 2023-11-24 16:54:13,959 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4400, loss[loss=0.06303, simple_loss=0.08534, pruned_loss=0.01227, audio_tagging_loss=0.008091, over 14893.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.0906, pruned_loss=0.01303, audio_tagging_loss=0.008871, over 3047445.32 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 16:54:33,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2915120.0, ans=0.125 2023-11-24 16:54:50,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten.whitening_limit, batch_count=2915253.3333333335, ans=22.5 2023-11-24 16:55:00,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2915253.3333333335, ans=0.125 2023-11-24 16:55:03,455 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437300 2023-11-24 16:55:16,833 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4450, loss[loss=0.07195, simple_loss=0.09644, pruned_loss=0.01596, audio_tagging_loss=0.00776, over 15903.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09065, pruned_loss=0.01293, audio_tagging_loss=0.008832, over 3051004.39 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 16:55:36,772 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.463e+01 9.107e+01 9.644e+01 1.325e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 16:56:01,404 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2915586.6666666665, ans=0.125 2023-11-24 16:56:04,864 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0 2023-11-24 16:56:05,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2915653.3333333335, ans=0.125 2023-11-24 16:56:06,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437350 2023-11-24 16:56:18,387 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4500, loss[loss=0.06876, simple_loss=0.09531, pruned_loss=0.01357, audio_tagging_loss=0.007534, over 16141.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09128, pruned_loss=0.01308, audio_tagging_loss=0.008732, over 3052601.58 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 16:56:25,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2915720.0, ans=0.125 2023-11-24 16:56:46,033 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.06 vs. limit=22.5 2023-11-24 16:56:58,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2915920.0, ans=0.015 2023-11-24 16:57:05,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2915920.0, ans=0.1 2023-11-24 16:57:07,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437400 2023-11-24 16:57:20,280 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4550, loss[loss=0.06624, simple_loss=0.09789, pruned_loss=0.01195, audio_tagging_loss=0.005341, over 16115.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09041, pruned_loss=0.01288, audio_tagging_loss=0.008826, over 3052993.02 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 4.0 2023-11-24 16:57:20,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2916053.3333333335, ans=0.125 2023-11-24 16:57:25,808 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.44 vs. limit=10.0 2023-11-24 16:57:27,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2916053.3333333335, ans=0.0 2023-11-24 16:57:30,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2916053.3333333335, ans=0.0 2023-11-24 16:57:34,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2916120.0, ans=0.125 2023-11-24 16:57:35,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2916120.0, ans=0.125 2023-11-24 16:57:43,341 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.438e+01 8.634e+01 9.216e+01 9.790e+01 1.251e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 16:58:06,787 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 16:58:09,524 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2916320.0, ans=0.125 2023-11-24 16:58:10,360 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437450 2023-11-24 16:58:23,350 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4600, loss[loss=0.05576, simple_loss=0.06731, pruned_loss=0.008508, audio_tagging_loss=0.0136, over 15440.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09073, pruned_loss=0.01304, audio_tagging_loss=0.008856, over 3046455.05 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 16:58:52,435 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.55 vs. limit=22.5 2023-11-24 16:59:03,594 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.17 vs. limit=22.5 2023-11-24 16:59:12,973 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437500 2023-11-24 16:59:15,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2916653.3333333335, ans=0.1 2023-11-24 16:59:17,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2916653.3333333335, ans=0.04949747468305833 2023-11-24 16:59:22,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2916653.3333333335, ans=0.125 2023-11-24 16:59:25,201 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4650, loss[loss=0.05392, simple_loss=0.07322, pruned_loss=0.00926, audio_tagging_loss=0.008047, over 15529.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.09006, pruned_loss=0.01283, audio_tagging_loss=0.00899, over 3047373.21 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 16:59:33,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2916720.0, ans=0.125 2023-11-24 16:59:41,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2916786.6666666665, ans=0.125 2023-11-24 16:59:42,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2916786.6666666665, ans=0.0 2023-11-24 16:59:46,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2916786.6666666665, ans=0.125 2023-11-24 16:59:46,887 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.291e+01 9.050e+01 9.890e+01 1.291e+02, threshold=1.810e+02, percent-clipped=0.0 2023-11-24 16:59:57,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2916853.3333333335, ans=0.025 2023-11-24 17:00:04,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=2916920.0, ans=0.2 2023-11-24 17:00:14,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437550 2023-11-24 17:00:16,087 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2916986.6666666665, ans=0.1 2023-11-24 17:00:27,362 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4700, loss[loss=0.08592, simple_loss=0.1194, pruned_loss=0.01805, audio_tagging_loss=0.008161, over 14569.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09072, pruned_loss=0.01299, audio_tagging_loss=0.008985, over 3047866.98 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:00:51,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2917186.6666666665, ans=0.0 2023-11-24 17:00:54,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=5.66 vs. limit=15.0 2023-11-24 17:01:17,483 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437600 2023-11-24 17:01:30,268 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4750, loss[loss=0.05262, simple_loss=0.06727, pruned_loss=0.009446, audio_tagging_loss=0.009538, over 14395.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09022, pruned_loss=0.01272, audio_tagging_loss=0.009219, over 3046154.45 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:01:42,818 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 17:01:52,693 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.496e+01 8.751e+01 9.417e+01 1.046e+02 1.255e+02, threshold=1.883e+02, percent-clipped=0.0 2023-11-24 17:02:05,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2917520.0, ans=0.125 2023-11-24 17:02:09,117 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2917586.6666666665, ans=0.0 2023-11-24 17:02:10,584 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.07 vs. limit=15.0 2023-11-24 17:02:19,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2917653.3333333335, ans=0.2 2023-11-24 17:02:20,688 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437650 2023-11-24 17:02:27,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2917653.3333333335, ans=0.0 2023-11-24 17:02:29,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2917653.3333333335, ans=0.0 2023-11-24 17:02:32,349 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4800, loss[loss=0.05259, simple_loss=0.06501, pruned_loss=0.007488, audio_tagging_loss=0.0126, over 15859.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09073, pruned_loss=0.01273, audio_tagging_loss=0.00925, over 3051047.75 frames. ], batch size: 63, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:03:21,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437700 2023-11-24 17:03:26,371 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.79 vs. limit=15.0 2023-11-24 17:03:34,101 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4850, loss[loss=0.07145, simple_loss=0.0983, pruned_loss=0.0128, audio_tagging_loss=0.009507, over 15366.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.0907, pruned_loss=0.01274, audio_tagging_loss=0.00924, over 3048489.32 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:03:37,720 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.40 vs. limit=10.0 2023-11-24 17:03:55,662 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.69 vs. limit=12.0 2023-11-24 17:03:58,341 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.659e+01 9.301e+01 9.915e+01 1.482e+02, threshold=1.860e+02, percent-clipped=0.0 2023-11-24 17:04:03,593 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2918186.6666666665, ans=0.0 2023-11-24 17:04:24,732 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437750 2023-11-24 17:04:36,860 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4900, loss[loss=0.04747, simple_loss=0.06441, pruned_loss=0.006037, audio_tagging_loss=0.009227, over 13912.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.08991, pruned_loss=0.01281, audio_tagging_loss=0.009244, over 3052570.23 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:04:42,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2918386.6666666665, ans=0.125 2023-11-24 17:05:02,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2918520.0, ans=0.125 2023-11-24 17:05:05,096 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2918520.0, ans=0.0 2023-11-24 17:05:07,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2918520.0, ans=0.2 2023-11-24 17:05:12,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2918520.0, ans=0.0 2023-11-24 17:05:14,102 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2918586.6666666665, ans=0.125 2023-11-24 17:05:18,782 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2918586.6666666665, ans=0.125 2023-11-24 17:05:27,734 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437800 2023-11-24 17:05:39,986 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 4950, loss[loss=0.05468, simple_loss=0.07028, pruned_loss=0.01, audio_tagging_loss=0.009535, over 14570.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09079, pruned_loss=0.01292, audio_tagging_loss=0.008983, over 3047097.46 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:06:04,043 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.424e+01 9.084e+01 9.605e+01 1.394e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 17:06:06,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2918853.3333333335, ans=0.0 2023-11-24 17:06:28,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2918920.0, ans=0.125 2023-11-24 17:06:30,247 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437850 2023-11-24 17:06:42,660 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5000, loss[loss=0.05826, simple_loss=0.07535, pruned_loss=0.01147, audio_tagging_loss=0.009113, over 16140.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09058, pruned_loss=0.01288, audio_tagging_loss=0.008846, over 3037157.42 frames. ], batch size: 62, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:06:46,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2919053.3333333335, ans=0.125 2023-11-24 17:06:58,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2919120.0, ans=0.125 2023-11-24 17:07:09,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2919186.6666666665, ans=0.0 2023-11-24 17:07:21,242 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2919253.3333333335, ans=0.125 2023-11-24 17:07:30,599 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.12 vs. limit=15.0 2023-11-24 17:07:32,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437900 2023-11-24 17:07:35,869 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.10 vs. limit=22.5 2023-11-24 17:07:43,229 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.83 vs. limit=15.0 2023-11-24 17:07:45,492 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5050, loss[loss=0.07155, simple_loss=0.101, pruned_loss=0.0117, audio_tagging_loss=0.009363, over 15227.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09089, pruned_loss=0.01294, audio_tagging_loss=0.008727, over 3032515.16 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:07:55,120 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.36 vs. limit=15.0 2023-11-24 17:08:03,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2919453.3333333335, ans=0.125 2023-11-24 17:08:08,405 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.782e+01 8.595e+01 9.107e+01 9.818e+01 1.374e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 17:08:19,254 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.05 vs. limit=15.0 2023-11-24 17:08:31,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2919586.6666666665, ans=0.0 2023-11-24 17:08:35,338 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 437950 2023-11-24 17:08:41,114 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.85 vs. limit=15.0 2023-11-24 17:08:44,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2919653.3333333335, ans=0.0 2023-11-24 17:08:47,801 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5100, loss[loss=0.05733, simple_loss=0.07552, pruned_loss=0.01023, audio_tagging_loss=0.009339, over 15254.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.0903, pruned_loss=0.0129, audio_tagging_loss=0.008725, over 3033924.00 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:08:52,663 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2919720.0, ans=0.1 2023-11-24 17:09:19,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2919853.3333333335, ans=0.125 2023-11-24 17:09:37,366 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438000 2023-11-24 17:09:45,238 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.86 vs. limit=15.0 2023-11-24 17:09:49,272 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5150, loss[loss=0.0554, simple_loss=0.07108, pruned_loss=0.01076, audio_tagging_loss=0.009105, over 14309.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.09056, pruned_loss=0.013, audio_tagging_loss=0.008676, over 3037542.13 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:10:01,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2920120.0, ans=0.125 2023-11-24 17:10:06,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2920120.0, ans=0.0 2023-11-24 17:10:13,400 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.605e+01 9.201e+01 1.005e+02 1.322e+02, threshold=1.840e+02, percent-clipped=0.0 2023-11-24 17:10:39,129 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438050 2023-11-24 17:10:52,001 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5200, loss[loss=0.07217, simple_loss=0.1035, pruned_loss=0.01466, audio_tagging_loss=0.005755, over 15849.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09079, pruned_loss=0.01302, audio_tagging_loss=0.008709, over 3041350.87 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:10:52,246 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2920386.6666666665, ans=0.2 2023-11-24 17:11:07,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2920453.3333333335, ans=0.125 2023-11-24 17:11:20,781 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2920520.0, ans=0.07 2023-11-24 17:11:25,590 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2920520.0, ans=0.2 2023-11-24 17:11:38,630 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2920586.6666666665, ans=0.5 2023-11-24 17:11:42,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438100 2023-11-24 17:11:42,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2920653.3333333335, ans=0.125 2023-11-24 17:11:55,069 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5250, loss[loss=0.0686, simple_loss=0.07828, pruned_loss=0.01496, audio_tagging_loss=0.0145, over 15665.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09119, pruned_loss=0.01314, audio_tagging_loss=0.008537, over 3045366.89 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:12:02,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2920720.0, ans=0.125 2023-11-24 17:12:04,751 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2920720.0, ans=0.125 2023-11-24 17:12:17,929 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.449e+01 8.931e+01 9.765e+01 1.225e+02, threshold=1.786e+02, percent-clipped=0.0 2023-11-24 17:12:21,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0 2023-11-24 17:12:41,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2920920.0, ans=0.1 2023-11-24 17:12:45,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438150 2023-11-24 17:12:48,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.04 vs. limit=22.5 2023-11-24 17:12:57,278 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5300, loss[loss=0.05273, simple_loss=0.07199, pruned_loss=0.009178, audio_tagging_loss=0.007554, over 14383.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.09028, pruned_loss=0.01293, audio_tagging_loss=0.008626, over 3038879.51 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:13:45,496 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=15.0 2023-11-24 17:13:47,410 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438200 2023-11-24 17:14:00,125 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5350, loss[loss=0.0674, simple_loss=0.08838, pruned_loss=0.01447, audio_tagging_loss=0.008744, over 15551.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09052, pruned_loss=0.01296, audio_tagging_loss=0.008599, over 3034024.52 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:14:12,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2921453.3333333335, ans=0.125 2023-11-24 17:14:13,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2921453.3333333335, ans=0.125 2023-11-24 17:14:20,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2921453.3333333335, ans=0.0 2023-11-24 17:14:24,405 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.013e+01 8.487e+01 9.193e+01 9.991e+01 1.472e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 17:14:29,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2921520.0, ans=0.1 2023-11-24 17:14:49,949 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438250 2023-11-24 17:14:59,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2921653.3333333335, ans=0.0 2023-11-24 17:15:03,528 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5400, loss[loss=0.0361, simple_loss=0.03951, pruned_loss=0.00531, audio_tagging_loss=0.01103, over 14513.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09112, pruned_loss=0.01297, audio_tagging_loss=0.008677, over 3039214.25 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:15:21,742 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2921786.6666666665, ans=0.125 2023-11-24 17:15:22,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2921786.6666666665, ans=0.0 2023-11-24 17:15:25,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2921786.6666666665, ans=0.125 2023-11-24 17:15:28,820 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2921853.3333333335, ans=0.2 2023-11-24 17:15:32,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2921853.3333333335, ans=0.125 2023-11-24 17:15:51,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2921920.0, ans=0.2 2023-11-24 17:15:53,243 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438300 2023-11-24 17:16:04,911 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5450, loss[loss=0.04223, simple_loss=0.05352, pruned_loss=0.005533, audio_tagging_loss=0.009939, over 14174.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09157, pruned_loss=0.01307, audio_tagging_loss=0.008682, over 3036896.27 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:16:22,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2922120.0, ans=0.125 2023-11-24 17:16:28,923 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.393e+01 9.258e+01 1.002e+02 1.486e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 17:16:51,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2922253.3333333335, ans=0.2 2023-11-24 17:16:53,684 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2922253.3333333335, ans=0.2 2023-11-24 17:16:54,205 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.76 vs. limit=15.0 2023-11-24 17:16:55,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438350 2023-11-24 17:17:00,386 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.87 vs. limit=10.0 2023-11-24 17:17:06,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2922320.0, ans=0.0 2023-11-24 17:17:08,722 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5500, loss[loss=0.05495, simple_loss=0.06733, pruned_loss=0.009311, audio_tagging_loss=0.01198, over 14604.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09147, pruned_loss=0.013, audio_tagging_loss=0.008797, over 3044820.81 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:17:58,579 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438400 2023-11-24 17:18:08,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.83 vs. limit=10.0 2023-11-24 17:18:11,630 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5550, loss[loss=0.06731, simple_loss=0.09103, pruned_loss=0.01285, audio_tagging_loss=0.008944, over 15331.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09226, pruned_loss=0.01332, audio_tagging_loss=0.008767, over 3043948.70 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:18:34,541 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.462e+01 8.681e+01 9.449e+01 1.002e+02 1.118e+02, threshold=1.890e+02, percent-clipped=0.0 2023-11-24 17:18:45,020 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2922853.3333333335, ans=0.0 2023-11-24 17:18:55,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2922920.0, ans=0.09899494936611666 2023-11-24 17:19:01,996 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438450 2023-11-24 17:19:02,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2922986.6666666665, ans=0.125 2023-11-24 17:19:13,631 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5600, loss[loss=0.05419, simple_loss=0.07354, pruned_loss=0.007184, audio_tagging_loss=0.01024, over 15896.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09235, pruned_loss=0.01317, audio_tagging_loss=0.008875, over 3037679.74 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:19:13,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2923053.3333333335, ans=0.125 2023-11-24 17:19:14,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.28 vs. limit=12.0 2023-11-24 17:19:40,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.35 vs. limit=8.0 2023-11-24 17:19:43,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2923186.6666666665, ans=0.125 2023-11-24 17:19:51,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2923253.3333333335, ans=0.2 2023-11-24 17:19:54,140 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2923253.3333333335, ans=0.1 2023-11-24 17:19:57,414 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 17:20:03,527 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438500 2023-11-24 17:20:04,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2923320.0, ans=0.2 2023-11-24 17:20:06,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2923320.0, ans=0.2 2023-11-24 17:20:13,132 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2923320.0, ans=0.125 2023-11-24 17:20:15,352 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5650, loss[loss=0.05299, simple_loss=0.07863, pruned_loss=0.004762, audio_tagging_loss=0.008914, over 14658.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.0922, pruned_loss=0.01305, audio_tagging_loss=0.009005, over 3042251.56 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:20:18,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2923386.6666666665, ans=0.0 2023-11-24 17:20:22,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.26 vs. limit=22.5 2023-11-24 17:20:26,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2923386.6666666665, ans=0.07 2023-11-24 17:20:28,962 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2923453.3333333335, ans=0.1 2023-11-24 17:20:31,379 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.00 vs. limit=12.0 2023-11-24 17:20:37,975 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.18 vs. limit=22.5 2023-11-24 17:20:40,182 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.503e+01 9.097e+01 9.762e+01 1.252e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 17:20:40,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0 2023-11-24 17:21:05,877 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438550 2023-11-24 17:21:18,239 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5700, loss[loss=0.08116, simple_loss=0.1206, pruned_loss=0.01702, audio_tagging_loss=0.003828, over 15683.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09222, pruned_loss=0.01309, audio_tagging_loss=0.008959, over 3046846.52 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:21:26,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2923720.0, ans=0.2 2023-11-24 17:21:33,219 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.48 vs. limit=22.5 2023-11-24 17:21:44,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2923853.3333333335, ans=0.125 2023-11-24 17:21:44,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2923853.3333333335, ans=0.125 2023-11-24 17:21:53,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2923853.3333333335, ans=0.125 2023-11-24 17:21:59,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2923920.0, ans=0.1 2023-11-24 17:22:07,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2923986.6666666665, ans=0.125 2023-11-24 17:22:08,405 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438600 2023-11-24 17:22:18,426 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2923986.6666666665, ans=0.1 2023-11-24 17:22:20,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2924053.3333333335, ans=0.1 2023-11-24 17:22:21,772 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5750, loss[loss=0.07316, simple_loss=0.09571, pruned_loss=0.01562, audio_tagging_loss=0.009685, over 15489.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09121, pruned_loss=0.01293, audio_tagging_loss=0.008972, over 3048601.48 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:22:23,325 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2924053.3333333335, ans=0.125 2023-11-24 17:22:35,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2924120.0, ans=0.125 2023-11-24 17:22:44,764 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.490e+01 8.615e+01 9.261e+01 9.857e+01 1.309e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 17:22:50,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2924186.6666666665, ans=0.09899494936611666 2023-11-24 17:22:55,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2924186.6666666665, ans=0.125 2023-11-24 17:23:01,755 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.33 vs. limit=15.0 2023-11-24 17:23:04,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2924253.3333333335, ans=0.0 2023-11-24 17:23:06,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2924253.3333333335, ans=0.0 2023-11-24 17:23:11,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438650 2023-11-24 17:23:15,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2924320.0, ans=0.1 2023-11-24 17:23:23,082 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5800, loss[loss=0.0696, simple_loss=0.09494, pruned_loss=0.01306, audio_tagging_loss=0.009072, over 15798.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09071, pruned_loss=0.01304, audio_tagging_loss=0.009011, over 3063165.73 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:23:30,865 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2924386.6666666665, ans=0.1 2023-11-24 17:23:32,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2924386.6666666665, ans=0.0 2023-11-24 17:24:00,796 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2924586.6666666665, ans=0.125 2023-11-24 17:24:12,473 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438700 2023-11-24 17:24:18,044 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2924653.3333333335, ans=0.125 2023-11-24 17:24:22,307 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.05 vs. limit=22.5 2023-11-24 17:24:25,424 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5850, loss[loss=0.0762, simple_loss=0.09238, pruned_loss=0.01641, audio_tagging_loss=0.01361, over 15365.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09085, pruned_loss=0.01291, audio_tagging_loss=0.008912, over 3061758.36 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:24:25,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2924720.0, ans=0.0 2023-11-24 17:24:26,273 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.41 vs. limit=12.0 2023-11-24 17:24:34,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2924720.0, ans=0.0 2023-11-24 17:24:44,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2924786.6666666665, ans=0.125 2023-11-24 17:24:50,121 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.501e+01 8.938e+01 9.729e+01 1.237e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-24 17:24:52,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2924853.3333333335, ans=0.125 2023-11-24 17:24:54,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2924853.3333333335, ans=0.125 2023-11-24 17:25:04,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2924920.0, ans=0.0 2023-11-24 17:25:15,080 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438750 2023-11-24 17:25:24,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.49 vs. limit=15.0 2023-11-24 17:25:25,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2924986.6666666665, ans=0.1 2023-11-24 17:25:27,328 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5900, loss[loss=0.05676, simple_loss=0.07336, pruned_loss=0.01187, audio_tagging_loss=0.008204, over 15160.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09079, pruned_loss=0.0129, audio_tagging_loss=0.008888, over 3064281.71 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:25:48,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2925120.0, ans=0.0 2023-11-24 17:26:16,683 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438800 2023-11-24 17:26:29,254 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 5950, loss[loss=0.08, simple_loss=0.1061, pruned_loss=0.01859, audio_tagging_loss=0.008354, over 15265.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09081, pruned_loss=0.01278, audio_tagging_loss=0.008856, over 3057138.99 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:26:31,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2925386.6666666665, ans=0.1 2023-11-24 17:26:32,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2925386.6666666665, ans=0.125 2023-11-24 17:26:35,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2925386.6666666665, ans=0.0 2023-11-24 17:26:54,497 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.448e+01 8.647e+01 9.220e+01 9.867e+01 1.210e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 17:27:08,293 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.90 vs. limit=15.0 2023-11-24 17:27:18,862 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438850 2023-11-24 17:27:30,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2925720.0, ans=0.125 2023-11-24 17:27:31,688 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6000, loss[loss=0.08767, simple_loss=0.1166, pruned_loss=0.02409, audio_tagging_loss=0.005255, over 14917.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09127, pruned_loss=0.01295, audio_tagging_loss=0.00879, over 3051240.27 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:27:31,689 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 17:27:57,240 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.4735, 2.8880, 2.5534, 3.1703, 2.8373, 2.9032, 2.9082, 2.7896], device='cuda:3') 2023-11-24 17:28:13,997 INFO [train_asr.py:1253] (3/4) Epoch 37, validation: loss=0.05829, simple_loss=0.05083, pruned_loss=0.00526, audio_tagging_loss=0.02761, over 4681554.00 frames. 2023-11-24 17:28:13,998 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 17:28:35,612 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2925786.6666666665, ans=0.2 2023-11-24 17:28:36,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2925786.6666666665, ans=0.2 2023-11-24 17:28:42,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2925853.3333333335, ans=0.2 2023-11-24 17:28:48,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.83 vs. limit=15.0 2023-11-24 17:28:59,043 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 17:29:03,778 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438900 2023-11-24 17:29:12,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2925986.6666666665, ans=0.2 2023-11-24 17:29:16,131 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6050, loss[loss=0.05257, simple_loss=0.07085, pruned_loss=0.00825, audio_tagging_loss=0.008902, over 14138.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09124, pruned_loss=0.01295, audio_tagging_loss=0.008795, over 3051877.52 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:29:37,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2926120.0, ans=0.1 2023-11-24 17:29:40,437 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2926186.6666666665, ans=0.0 2023-11-24 17:29:41,480 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.423e+01 8.485e+01 9.097e+01 9.869e+01 1.305e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 17:29:51,278 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2926186.6666666665, ans=0.125 2023-11-24 17:29:55,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2926253.3333333335, ans=0.125 2023-11-24 17:30:06,417 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 438950 2023-11-24 17:30:18,683 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6100, loss[loss=0.07066, simple_loss=0.1008, pruned_loss=0.01267, audio_tagging_loss=0.007613, over 14137.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09084, pruned_loss=0.0128, audio_tagging_loss=0.008785, over 3051488.01 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:30:20,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2926386.6666666665, ans=0.2 2023-11-24 17:30:20,772 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.06 vs. limit=15.0 2023-11-24 17:30:40,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2926453.3333333335, ans=0.125 2023-11-24 17:30:42,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2926520.0, ans=0.125 2023-11-24 17:30:48,779 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.96 vs. limit=15.0 2023-11-24 17:30:53,941 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 17:31:08,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439000 2023-11-24 17:31:09,190 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.66 vs. limit=10.0 2023-11-24 17:31:21,193 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6150, loss[loss=0.08795, simple_loss=0.1235, pruned_loss=0.01743, audio_tagging_loss=0.008747, over 15608.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09076, pruned_loss=0.01267, audio_tagging_loss=0.00877, over 3053685.41 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:31:46,190 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.702e+01 9.525e+01 1.018e+02 1.166e+02, threshold=1.905e+02, percent-clipped=0.0 2023-11-24 17:31:46,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2926853.3333333335, ans=0.1 2023-11-24 17:32:04,319 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.79 vs. limit=22.5 2023-11-24 17:32:05,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2926920.0, ans=0.1 2023-11-24 17:32:10,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439050 2023-11-24 17:32:13,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2926986.6666666665, ans=0.2 2023-11-24 17:32:19,347 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 17:32:22,656 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6200, loss[loss=0.08671, simple_loss=0.1279, pruned_loss=0.01621, audio_tagging_loss=0.006558, over 16207.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.09068, pruned_loss=0.01261, audio_tagging_loss=0.008844, over 3052564.29 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:32:49,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2927186.6666666665, ans=0.0 2023-11-24 17:33:12,151 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439100 2023-11-24 17:33:18,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=2927320.0, ans=15.0 2023-11-24 17:33:25,879 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6250, loss[loss=0.06148, simple_loss=0.08704, pruned_loss=0.01042, audio_tagging_loss=0.007541, over 15707.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.0906, pruned_loss=0.01269, audio_tagging_loss=0.008926, over 3053259.87 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 17:33:32,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2927386.6666666665, ans=0.2 2023-11-24 17:33:50,394 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 8.621e+01 9.212e+01 1.010e+02 1.296e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-24 17:34:12,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2927586.6666666665, ans=0.0 2023-11-24 17:34:15,163 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439150 2023-11-24 17:34:25,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2927653.3333333335, ans=0.0 2023-11-24 17:34:27,450 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6300, loss[loss=0.07153, simple_loss=0.1018, pruned_loss=0.01045, audio_tagging_loss=0.01017, over 16197.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09043, pruned_loss=0.01268, audio_tagging_loss=0.009013, over 3055325.66 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:34:59,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2927853.3333333335, ans=0.1 2023-11-24 17:35:00,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.28 vs. limit=6.0 2023-11-24 17:35:01,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2927853.3333333335, ans=0.125 2023-11-24 17:35:17,219 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439200 2023-11-24 17:35:17,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2927986.6666666665, ans=0.125 2023-11-24 17:35:29,267 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6350, loss[loss=0.07765, simple_loss=0.1118, pruned_loss=0.01195, audio_tagging_loss=0.009807, over 15455.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.0913, pruned_loss=0.01279, audio_tagging_loss=0.009065, over 3055907.87 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:35:41,633 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.62 vs. limit=15.0 2023-11-24 17:35:56,864 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.531e+01 9.107e+01 9.829e+01 2.915e+02, threshold=1.821e+02, percent-clipped=1.0 2023-11-24 17:36:11,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2928253.3333333335, ans=0.125 2023-11-24 17:36:15,519 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2928253.3333333335, ans=0.125 2023-11-24 17:36:18,731 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439250 2023-11-24 17:36:18,853 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2928320.0, ans=0.1 2023-11-24 17:36:27,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2928320.0, ans=0.0 2023-11-24 17:36:31,545 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6400, loss[loss=0.07177, simple_loss=0.09581, pruned_loss=0.01327, audio_tagging_loss=0.01059, over 15757.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.0915, pruned_loss=0.0129, audio_tagging_loss=0.009036, over 3061650.29 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:36:34,724 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2928386.6666666665, ans=0.0 2023-11-24 17:36:35,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2928386.6666666665, ans=0.125 2023-11-24 17:36:41,653 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2928386.6666666665, ans=0.1 2023-11-24 17:37:03,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2928520.0, ans=0.0 2023-11-24 17:37:05,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2928520.0, ans=0.125 2023-11-24 17:37:06,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2928586.6666666665, ans=0.1 2023-11-24 17:37:18,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2928586.6666666665, ans=0.09899494936611666 2023-11-24 17:37:19,113 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.46 vs. limit=15.0 2023-11-24 17:37:21,576 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439300 2023-11-24 17:37:26,586 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2928653.3333333335, ans=0.1 2023-11-24 17:37:33,832 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6450, loss[loss=0.06016, simple_loss=0.07988, pruned_loss=0.00992, audio_tagging_loss=0.0103, over 14536.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09062, pruned_loss=0.01279, audio_tagging_loss=0.009145, over 3067673.91 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:37:36,708 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0 2023-11-24 17:37:41,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2928720.0, ans=0.0 2023-11-24 17:37:44,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.28 vs. limit=6.0 2023-11-24 17:37:47,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2928786.6666666665, ans=0.125 2023-11-24 17:37:51,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2928786.6666666665, ans=0.0 2023-11-24 17:37:52,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2928786.6666666665, ans=0.125 2023-11-24 17:37:54,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2928786.6666666665, ans=0.0 2023-11-24 17:37:54,084 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2928786.6666666665, ans=0.0 2023-11-24 17:38:00,170 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.603e+01 9.218e+01 1.012e+02 1.215e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 17:38:08,135 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2928853.3333333335, ans=0.125 2023-11-24 17:38:20,377 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2928920.0, ans=0.125 2023-11-24 17:38:22,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439350 2023-11-24 17:38:34,357 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6500, loss[loss=0.06729, simple_loss=0.09483, pruned_loss=0.01216, audio_tagging_loss=0.00771, over 15559.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.08963, pruned_loss=0.01269, audio_tagging_loss=0.009155, over 3061022.53 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:38:38,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.66 vs. limit=22.5 2023-11-24 17:38:41,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2929053.3333333335, ans=0.125 2023-11-24 17:39:05,140 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0 2023-11-24 17:39:06,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2929186.6666666665, ans=0.0 2023-11-24 17:39:06,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.46 vs. limit=15.0 2023-11-24 17:39:09,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2929186.6666666665, ans=0.125 2023-11-24 17:39:19,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2929253.3333333335, ans=0.125 2023-11-24 17:39:24,221 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439400 2023-11-24 17:39:24,396 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2929320.0, ans=0.125 2023-11-24 17:39:34,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2929320.0, ans=0.1 2023-11-24 17:39:36,737 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6550, loss[loss=0.05622, simple_loss=0.07897, pruned_loss=0.01001, audio_tagging_loss=0.00672, over 14578.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.08999, pruned_loss=0.01274, audio_tagging_loss=0.008979, over 3057287.20 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:39:38,316 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2929386.6666666665, ans=0.125 2023-11-24 17:39:41,399 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.33 vs. limit=15.0 2023-11-24 17:39:42,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2929386.6666666665, ans=0.0 2023-11-24 17:39:55,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2929453.3333333335, ans=0.0 2023-11-24 17:40:05,707 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.772e+01 9.362e+01 9.895e+01 1.833e+02, threshold=1.872e+02, percent-clipped=0.0 2023-11-24 17:40:11,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2929520.0, ans=0.0 2023-11-24 17:40:17,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.16 vs. limit=15.0 2023-11-24 17:40:21,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.76 vs. limit=10.0 2023-11-24 17:40:26,641 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439450 2023-11-24 17:40:28,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2929653.3333333335, ans=0.125 2023-11-24 17:40:30,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2929653.3333333335, ans=0.09899494936611666 2023-11-24 17:40:39,585 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6600, loss[loss=0.05988, simple_loss=0.07214, pruned_loss=0.01239, audio_tagging_loss=0.01141, over 15060.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09037, pruned_loss=0.01293, audio_tagging_loss=0.008893, over 3045656.49 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:40:49,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2929720.0, ans=0.125 2023-11-24 17:41:01,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2929786.6666666665, ans=0.2 2023-11-24 17:41:10,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2929853.3333333335, ans=0.0 2023-11-24 17:41:12,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2929853.3333333335, ans=0.125 2023-11-24 17:41:18,105 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2929920.0, ans=0.09899494936611666 2023-11-24 17:41:29,666 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439500 2023-11-24 17:41:41,329 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6650, loss[loss=0.05533, simple_loss=0.07578, pruned_loss=0.009748, audio_tagging_loss=0.007688, over 14984.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.08974, pruned_loss=0.01281, audio_tagging_loss=0.008783, over 3048749.17 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:41:47,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2930053.3333333335, ans=0.0 2023-11-24 17:41:58,860 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2930120.0, ans=0.125 2023-11-24 17:42:10,032 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.532e+01 9.137e+01 1.001e+02 1.205e+02, threshold=1.827e+02, percent-clipped=0.0 2023-11-24 17:42:14,535 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2930186.6666666665, ans=0.125 2023-11-24 17:42:21,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2930253.3333333335, ans=0.0 2023-11-24 17:42:26,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2930253.3333333335, ans=0.125 2023-11-24 17:42:26,308 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.52 vs. limit=15.0 2023-11-24 17:42:31,774 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439550 2023-11-24 17:42:44,159 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6700, loss[loss=0.06183, simple_loss=0.08387, pruned_loss=0.01007, audio_tagging_loss=0.009819, over 15925.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.09045, pruned_loss=0.0129, audio_tagging_loss=0.008652, over 3052342.14 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:43:07,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2930453.3333333335, ans=0.1 2023-11-24 17:43:14,528 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2930520.0, ans=0.125 2023-11-24 17:43:15,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2930520.0, ans=0.125 2023-11-24 17:43:28,177 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 17:43:33,920 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439600 2023-11-24 17:43:38,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_na.min_abs, batch_count=2930653.3333333335, ans=0.02 2023-11-24 17:43:46,513 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6750, loss[loss=0.09928, simple_loss=0.143, pruned_loss=0.02107, audio_tagging_loss=0.006732, over 15872.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.09048, pruned_loss=0.01283, audio_tagging_loss=0.008595, over 3049332.77 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0 2023-11-24 17:43:55,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2930720.0, ans=0.2 2023-11-24 17:44:01,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2930786.6666666665, ans=0.1 2023-11-24 17:44:15,656 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.260e+01 8.901e+01 9.551e+01 1.159e+02, threshold=1.780e+02, percent-clipped=0.0 2023-11-24 17:44:31,168 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2930920.0, ans=0.125 2023-11-24 17:44:37,004 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439650 2023-11-24 17:44:49,439 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6800, loss[loss=0.07289, simple_loss=0.1025, pruned_loss=0.01198, audio_tagging_loss=0.009667, over 16004.00 frames. ], tot_loss[loss=0.06628, simple_loss=0.08969, pruned_loss=0.01272, audio_tagging_loss=0.008717, over 3049070.30 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:44:52,023 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2931053.3333333335, ans=0.04949747468305833 2023-11-24 17:44:54,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2931053.3333333335, ans=0.125 2023-11-24 17:45:39,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439700 2023-11-24 17:45:43,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2931320.0, ans=10.0 2023-11-24 17:45:43,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2931320.0, ans=0.0 2023-11-24 17:45:51,918 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6850, loss[loss=0.06639, simple_loss=0.09053, pruned_loss=0.01313, audio_tagging_loss=0.007987, over 15168.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.08961, pruned_loss=0.0129, audio_tagging_loss=0.008786, over 3038892.02 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:46:01,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2931386.6666666665, ans=0.125 2023-11-24 17:46:14,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2931453.3333333335, ans=0.1 2023-11-24 17:46:20,782 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.12 vs. limit=15.0 2023-11-24 17:46:21,181 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.185e+01 8.495e+01 8.934e+01 9.864e+01 1.145e+02, threshold=1.787e+02, percent-clipped=0.0 2023-11-24 17:46:27,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2931520.0, ans=0.0 2023-11-24 17:46:39,084 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.27 vs. limit=6.0 2023-11-24 17:46:42,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439750 2023-11-24 17:46:47,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2931653.3333333335, ans=0.125 2023-11-24 17:46:51,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2931653.3333333335, ans=0.1 2023-11-24 17:46:55,047 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6900, loss[loss=0.06207, simple_loss=0.08792, pruned_loss=0.009178, audio_tagging_loss=0.008933, over 15561.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09056, pruned_loss=0.01288, audio_tagging_loss=0.008757, over 3045256.93 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:47:20,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2931853.3333333335, ans=0.125 2023-11-24 17:47:30,197 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0 2023-11-24 17:47:42,858 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 17:47:45,323 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439800 2023-11-24 17:47:54,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2931986.6666666665, ans=0.07 2023-11-24 17:47:55,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2931986.6666666665, ans=0.125 2023-11-24 17:47:57,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2932053.3333333335, ans=0.05 2023-11-24 17:47:58,606 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 6950, loss[loss=0.081, simple_loss=0.1172, pruned_loss=0.01656, audio_tagging_loss=0.005829, over 15871.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.09061, pruned_loss=0.01283, audio_tagging_loss=0.008771, over 3037722.94 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:48:07,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2932053.3333333335, ans=0.09899494936611666 2023-11-24 17:48:08,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.66 vs. limit=15.0 2023-11-24 17:48:14,547 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.35 vs. limit=15.0 2023-11-24 17:48:22,397 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2932186.6666666665, ans=0.1 2023-11-24 17:48:27,201 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.428e+01 8.692e+01 9.069e+01 1.003e+02 1.264e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 17:48:27,604 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2932186.6666666665, ans=0.0 2023-11-24 17:48:33,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2932186.6666666665, ans=0.125 2023-11-24 17:48:35,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2932253.3333333335, ans=0.2 2023-11-24 17:48:37,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2932253.3333333335, ans=0.2 2023-11-24 17:48:41,391 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.70 vs. limit=15.0 2023-11-24 17:48:48,754 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439850 2023-11-24 17:48:49,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2932320.0, ans=0.125 2023-11-24 17:48:54,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2932320.0, ans=0.0 2023-11-24 17:48:56,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2932320.0, ans=0.1 2023-11-24 17:49:00,517 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7000, loss[loss=0.05175, simple_loss=0.06725, pruned_loss=0.008619, audio_tagging_loss=0.009503, over 14401.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09104, pruned_loss=0.01294, audio_tagging_loss=0.008805, over 3030538.37 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:49:18,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2932453.3333333335, ans=0.125 2023-11-24 17:49:21,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2932453.3333333335, ans=0.125 2023-11-24 17:49:28,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2932520.0, ans=0.125 2023-11-24 17:49:33,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2932520.0, ans=0.1 2023-11-24 17:49:49,128 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=15.35 vs. limit=15.0 2023-11-24 17:49:51,106 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439900 2023-11-24 17:49:56,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2932653.3333333335, ans=0.1 2023-11-24 17:49:57,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2932653.3333333335, ans=0.05 2023-11-24 17:50:03,453 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7050, loss[loss=0.08027, simple_loss=0.1051, pruned_loss=0.02028, audio_tagging_loss=0.007421, over 14601.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09169, pruned_loss=0.0132, audio_tagging_loss=0.008841, over 3038234.56 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:50:06,656 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.69 vs. limit=10.0 2023-11-24 17:50:23,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2932786.6666666665, ans=0.125 2023-11-24 17:50:25,345 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.30 vs. limit=6.0 2023-11-24 17:50:31,914 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.508e+01 9.071e+01 9.763e+01 1.227e+02, threshold=1.814e+02, percent-clipped=0.0 2023-11-24 17:50:46,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2932920.0, ans=0.0 2023-11-24 17:50:47,764 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2932920.0, ans=0.1 2023-11-24 17:50:53,365 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 439950 2023-11-24 17:51:05,589 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7100, loss[loss=0.07388, simple_loss=0.09718, pruned_loss=0.01823, audio_tagging_loss=0.007059, over 16059.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09144, pruned_loss=0.01322, audio_tagging_loss=0.008912, over 3040720.33 frames. ], batch size: 62, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:51:17,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2933120.0, ans=0.0 2023-11-24 17:51:33,924 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.87 vs. limit=15.0 2023-11-24 17:51:55,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440000 2023-11-24 17:52:04,029 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.91 vs. limit=15.0 2023-11-24 17:52:12,087 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7150, loss[loss=0.06725, simple_loss=0.09435, pruned_loss=0.01102, audio_tagging_loss=0.009052, over 14495.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09094, pruned_loss=0.01312, audio_tagging_loss=0.008972, over 3042124.10 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:52:13,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2933386.6666666665, ans=0.0 2023-11-24 17:52:14,597 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2933386.6666666665, ans=0.2 2023-11-24 17:52:14,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2933386.6666666665, ans=0.1 2023-11-24 17:52:20,259 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.01 vs. limit=6.0 2023-11-24 17:52:34,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2933453.3333333335, ans=0.125 2023-11-24 17:52:39,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2933520.0, ans=0.1 2023-11-24 17:52:40,900 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.362e+01 8.675e+01 9.215e+01 9.966e+01 1.271e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 17:52:46,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2933520.0, ans=0.04949747468305833 2023-11-24 17:53:01,840 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440050 2023-11-24 17:53:07,301 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2933653.3333333335, ans=0.125 2023-11-24 17:53:14,519 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7200, loss[loss=0.07707, simple_loss=0.105, pruned_loss=0.01554, audio_tagging_loss=0.009023, over 16535.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09112, pruned_loss=0.01309, audio_tagging_loss=0.009039, over 3045248.66 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:53:15,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.43 vs. limit=22.5 2023-11-24 17:53:18,455 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2933720.0, ans=10.0 2023-11-24 17:53:46,275 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.10 vs. limit=10.0 2023-11-24 17:53:47,969 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2933853.3333333335, ans=0.125 2023-11-24 17:53:49,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2933853.3333333335, ans=0.0 2023-11-24 17:53:52,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2933920.0, ans=0.0 2023-11-24 17:53:59,715 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2933920.0, ans=0.04949747468305833 2023-11-24 17:54:04,231 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440100 2023-11-24 17:54:16,762 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7250, loss[loss=0.08493, simple_loss=0.1197, pruned_loss=0.01889, audio_tagging_loss=0.006198, over 15145.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09117, pruned_loss=0.0131, audio_tagging_loss=0.009091, over 3038756.62 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:54:23,191 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=15.0 2023-11-24 17:54:24,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2934053.3333333335, ans=0.125 2023-11-24 17:54:35,708 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2934120.0, ans=0.0 2023-11-24 17:54:46,353 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.74 vs. limit=10.0 2023-11-24 17:54:47,441 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.141e+01 8.488e+01 9.107e+01 9.916e+01 1.399e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 17:54:56,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.17 vs. limit=10.0 2023-11-24 17:55:06,886 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440150 2023-11-24 17:55:11,947 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2934320.0, ans=0.025 2023-11-24 17:55:13,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.06 vs. limit=22.5 2023-11-24 17:55:18,561 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7300, loss[loss=0.05721, simple_loss=0.06933, pruned_loss=0.01121, audio_tagging_loss=0.01133, over 15236.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09113, pruned_loss=0.0129, audio_tagging_loss=0.008975, over 3035742.80 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:55:38,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2934453.3333333335, ans=0.125 2023-11-24 17:56:04,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2934586.6666666665, ans=0.0 2023-11-24 17:56:05,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2934586.6666666665, ans=0.125 2023-11-24 17:56:07,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2934653.3333333335, ans=0.125 2023-11-24 17:56:07,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2934653.3333333335, ans=0.125 2023-11-24 17:56:09,163 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440200 2023-11-24 17:56:22,503 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7350, loss[loss=0.05502, simple_loss=0.07795, pruned_loss=0.00927, audio_tagging_loss=0.006775, over 15456.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09154, pruned_loss=0.01304, audio_tagging_loss=0.008792, over 3043837.21 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:56:24,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2934720.0, ans=0.0 2023-11-24 17:56:30,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2934720.0, ans=0.025 2023-11-24 17:56:37,324 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2934786.6666666665, ans=0.0 2023-11-24 17:56:51,150 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.582e+01 8.969e+01 9.550e+01 1.265e+02, threshold=1.794e+02, percent-clipped=0.0 2023-11-24 17:56:55,010 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2934853.3333333335, ans=0.125 2023-11-24 17:56:55,366 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.82 vs. limit=15.0 2023-11-24 17:56:58,981 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2934920.0, ans=0.125 2023-11-24 17:56:58,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2934920.0, ans=10.0 2023-11-24 17:57:11,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440250 2023-11-24 17:57:24,028 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7400, loss[loss=0.07675, simple_loss=0.1031, pruned_loss=0.0177, audio_tagging_loss=0.007496, over 15702.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09165, pruned_loss=0.0131, audio_tagging_loss=0.008577, over 3037948.77 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:57:24,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.49 vs. limit=22.5 2023-11-24 17:57:48,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2935186.6666666665, ans=0.0 2023-11-24 17:57:52,362 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2935186.6666666665, ans=0.125 2023-11-24 17:57:57,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2935186.6666666665, ans=0.125 2023-11-24 17:57:59,483 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.45 vs. limit=15.0 2023-11-24 17:58:02,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2935253.3333333335, ans=0.1 2023-11-24 17:58:07,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2935253.3333333335, ans=0.0 2023-11-24 17:58:12,284 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 17:58:14,554 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440300 2023-11-24 17:58:26,490 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7450, loss[loss=0.06963, simple_loss=0.09597, pruned_loss=0.01255, audio_tagging_loss=0.009094, over 14659.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.0912, pruned_loss=0.01309, audio_tagging_loss=0.008563, over 3035770.75 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:58:28,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2935386.6666666665, ans=0.1 2023-11-24 17:58:31,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2935386.6666666665, ans=0.0 2023-11-24 17:58:56,709 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.405e+01 8.534e+01 9.145e+01 9.758e+01 1.407e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 17:59:15,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2935653.3333333335, ans=0.125 2023-11-24 17:59:16,388 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440350 2023-11-24 17:59:28,789 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7500, loss[loss=0.06385, simple_loss=0.102, pruned_loss=0.006714, audio_tagging_loss=0.006155, over 15356.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09088, pruned_loss=0.01307, audio_tagging_loss=0.008603, over 3038129.98 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 17:59:43,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2935786.6666666665, ans=0.125 2023-11-24 17:59:53,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2935853.3333333335, ans=10.0 2023-11-24 17:59:56,884 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.33 vs. limit=15.0 2023-11-24 18:00:03,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2935853.3333333335, ans=0.0 2023-11-24 18:00:18,918 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440400 2023-11-24 18:00:31,112 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7550, loss[loss=0.05297, simple_loss=0.0661, pruned_loss=0.008744, audio_tagging_loss=0.01118, over 15352.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09058, pruned_loss=0.01313, audio_tagging_loss=0.008648, over 3042643.92 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 18:00:46,346 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.57 vs. limit=10.0 2023-11-24 18:00:47,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2936120.0, ans=0.1 2023-11-24 18:00:58,542 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0 2023-11-24 18:01:00,294 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.354e+01 8.524e+01 9.162e+01 9.767e+01 1.233e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 18:01:03,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2936186.6666666665, ans=0.125 2023-11-24 18:01:06,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2936186.6666666665, ans=0.0 2023-11-24 18:01:21,336 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440450 2023-11-24 18:01:32,931 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7600, loss[loss=0.07815, simple_loss=0.109, pruned_loss=0.01639, audio_tagging_loss=0.007258, over 15828.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.09025, pruned_loss=0.01307, audio_tagging_loss=0.008651, over 3044426.35 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:01:46,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2936453.3333333335, ans=0.1 2023-11-24 18:01:52,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2936453.3333333335, ans=0.1 2023-11-24 18:02:04,839 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2936520.0, ans=0.125 2023-11-24 18:02:16,319 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2936586.6666666665, ans=0.125 2023-11-24 18:02:23,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440500 2023-11-24 18:02:35,611 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7650, loss[loss=0.07246, simple_loss=0.1022, pruned_loss=0.01131, audio_tagging_loss=0.01006, over 14775.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.09038, pruned_loss=0.01298, audio_tagging_loss=0.008724, over 3048937.63 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:02:57,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2936786.6666666665, ans=0.2 2023-11-24 18:03:03,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2936853.3333333335, ans=0.125 2023-11-24 18:03:05,430 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.749e+01 9.173e+01 9.808e+01 1.366e+02, threshold=1.835e+02, percent-clipped=0.0 2023-11-24 18:03:13,085 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2936920.0, ans=0.125 2023-11-24 18:03:22,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2936920.0, ans=0.0 2023-11-24 18:03:25,541 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440550 2023-11-24 18:03:28,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2936986.6666666665, ans=0.0 2023-11-24 18:03:31,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2936986.6666666665, ans=0.125 2023-11-24 18:03:37,740 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7700, loss[loss=0.06753, simple_loss=0.0811, pruned_loss=0.0167, audio_tagging_loss=0.01028, over 14047.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09119, pruned_loss=0.01312, audio_tagging_loss=0.008821, over 3046267.59 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:03:44,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2937053.3333333335, ans=0.125 2023-11-24 18:03:50,599 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.57 vs. limit=15.0 2023-11-24 18:03:51,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2937120.0, ans=0.1 2023-11-24 18:03:55,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2937120.0, ans=0.1 2023-11-24 18:04:00,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2937186.6666666665, ans=0.125 2023-11-24 18:04:04,380 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2937186.6666666665, ans=0.0 2023-11-24 18:04:27,258 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440600 2023-11-24 18:04:27,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2937320.0, ans=0.2 2023-11-24 18:04:39,842 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7750, loss[loss=0.07626, simple_loss=0.1064, pruned_loss=0.01737, audio_tagging_loss=0.005712, over 15664.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09124, pruned_loss=0.01303, audio_tagging_loss=0.008858, over 3047653.24 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:04:46,641 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.60 vs. limit=15.0 2023-11-24 18:04:49,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2937386.6666666665, ans=0.0 2023-11-24 18:05:01,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2937453.3333333335, ans=0.125 2023-11-24 18:05:10,509 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.466e+01 8.710e+01 9.479e+01 9.926e+01 1.240e+02, threshold=1.896e+02, percent-clipped=0.0 2023-11-24 18:05:10,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2937520.0, ans=0.0 2023-11-24 18:05:30,269 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440650 2023-11-24 18:05:42,086 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7800, loss[loss=0.06028, simple_loss=0.08097, pruned_loss=0.01113, audio_tagging_loss=0.008666, over 14872.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09056, pruned_loss=0.01285, audio_tagging_loss=0.008944, over 3045539.83 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:05:56,926 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2937786.6666666665, ans=0.125 2023-11-24 18:06:12,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2937853.3333333335, ans=0.125 2023-11-24 18:06:20,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2937920.0, ans=0.07 2023-11-24 18:06:22,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2937920.0, ans=0.125 2023-11-24 18:06:30,312 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0 2023-11-24 18:06:32,221 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440700 2023-11-24 18:06:45,218 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7850, loss[loss=0.1053, simple_loss=0.1464, pruned_loss=0.02476, audio_tagging_loss=0.007371, over 16553.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09078, pruned_loss=0.01279, audio_tagging_loss=0.009032, over 3048747.19 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:06:51,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2938053.3333333335, ans=0.125 2023-11-24 18:06:56,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2938120.0, ans=0.1 2023-11-24 18:07:11,110 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2938186.6666666665, ans=0.1 2023-11-24 18:07:14,189 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.363e+01 8.824e+01 9.632e+01 1.042e+02 1.245e+02, threshold=1.926e+02, percent-clipped=0.0 2023-11-24 18:07:15,703 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2938186.6666666665, ans=0.05 2023-11-24 18:07:34,114 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=2938320.0, ans=15.0 2023-11-24 18:07:34,888 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440750 2023-11-24 18:07:36,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.91 vs. limit=15.0 2023-11-24 18:07:47,156 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7900, loss[loss=0.06573, simple_loss=0.0868, pruned_loss=0.01101, audio_tagging_loss=0.01132, over 14960.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09043, pruned_loss=0.01279, audio_tagging_loss=0.009219, over 3046867.29 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:07:54,996 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.73 vs. limit=22.5 2023-11-24 18:07:59,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2938453.3333333335, ans=0.125 2023-11-24 18:08:05,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2938453.3333333335, ans=0.1 2023-11-24 18:08:18,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2938520.0, ans=0.0 2023-11-24 18:08:18,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.84 vs. limit=15.0 2023-11-24 18:08:27,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2938586.6666666665, ans=0.0 2023-11-24 18:08:33,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2938586.6666666665, ans=0.125 2023-11-24 18:08:37,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440800 2023-11-24 18:08:48,997 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 7950, loss[loss=0.07334, simple_loss=0.1061, pruned_loss=0.01247, audio_tagging_loss=0.007821, over 15349.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09176, pruned_loss=0.013, audio_tagging_loss=0.009144, over 3047993.34 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:08:49,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2938720.0, ans=0.1 2023-11-24 18:08:52,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2938720.0, ans=0.125 2023-11-24 18:09:04,171 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:09:18,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.49 vs. limit=22.5 2023-11-24 18:09:20,727 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.553e+01 8.522e+01 9.138e+01 9.780e+01 1.184e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 18:09:22,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2938853.3333333335, ans=0.1 2023-11-24 18:09:29,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2938920.0, ans=0.0 2023-11-24 18:09:30,989 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.45 vs. limit=15.0 2023-11-24 18:09:38,592 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440850 2023-11-24 18:09:51,411 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8000, loss[loss=0.05763, simple_loss=0.07388, pruned_loss=0.01141, audio_tagging_loss=0.009282, over 14603.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09095, pruned_loss=0.01275, audio_tagging_loss=0.009201, over 3050833.84 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:10:20,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2939186.6666666665, ans=0.125 2023-11-24 18:10:31,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2939253.3333333335, ans=0.125 2023-11-24 18:10:35,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2939253.3333333335, ans=0.2 2023-11-24 18:10:41,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440900 2023-11-24 18:10:51,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2939320.0, ans=0.09899494936611666 2023-11-24 18:10:53,885 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8050, loss[loss=0.05967, simple_loss=0.07506, pruned_loss=0.01137, audio_tagging_loss=0.01077, over 13957.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09046, pruned_loss=0.01268, audio_tagging_loss=0.009245, over 3045450.19 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 32.0 2023-11-24 18:11:08,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2939453.3333333335, ans=0.2 2023-11-24 18:11:17,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2939520.0, ans=0.125 2023-11-24 18:11:25,409 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.434e+01 8.492e+01 9.262e+01 1.003e+02 1.241e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 18:11:36,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2939586.6666666665, ans=0.1 2023-11-24 18:11:43,202 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 440950 2023-11-24 18:11:43,831 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.17 vs. limit=15.0 2023-11-24 18:11:50,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2939653.3333333335, ans=0.125 2023-11-24 18:11:51,646 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2939653.3333333335, ans=0.125 2023-11-24 18:11:54,751 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8100, loss[loss=0.05927, simple_loss=0.07286, pruned_loss=0.01321, audio_tagging_loss=0.009625, over 14513.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09004, pruned_loss=0.01276, audio_tagging_loss=0.009146, over 3037696.27 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 18:11:57,946 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.39 vs. limit=5.0 2023-11-24 18:12:20,954 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.31 vs. limit=22.5 2023-11-24 18:12:39,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2939920.0, ans=0.0 2023-11-24 18:12:41,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2939920.0, ans=22.5 2023-11-24 18:12:44,271 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441000 2023-11-24 18:12:47,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2939986.6666666665, ans=0.2 2023-11-24 18:12:51,776 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.74 vs. limit=15.0 2023-11-24 18:12:57,406 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8150, loss[loss=0.07298, simple_loss=0.0959, pruned_loss=0.01682, audio_tagging_loss=0.00821, over 14952.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09054, pruned_loss=0.01282, audio_tagging_loss=0.008955, over 3037845.93 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 18:13:21,181 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2940186.6666666665, ans=0.1 2023-11-24 18:13:29,019 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.802e+01 9.331e+01 9.892e+01 1.682e+02, threshold=1.866e+02, percent-clipped=0.0 2023-11-24 18:13:44,608 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2940253.3333333335, ans=0.125 2023-11-24 18:13:46,737 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441050 2023-11-24 18:13:53,204 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2940320.0, ans=0.125 2023-11-24 18:13:59,373 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8200, loss[loss=0.06856, simple_loss=0.09544, pruned_loss=0.01305, audio_tagging_loss=0.007795, over 14958.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09076, pruned_loss=0.01293, audio_tagging_loss=0.008837, over 3042820.69 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 18:13:59,415 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:14:04,749 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:14:15,647 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.50 vs. limit=22.5 2023-11-24 18:14:16,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2940453.3333333335, ans=0.125 2023-11-24 18:14:29,384 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2940520.0, ans=0.035 2023-11-24 18:14:35,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2940586.6666666665, ans=15.0 2023-11-24 18:14:36,669 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2940586.6666666665, ans=0.2 2023-11-24 18:14:40,405 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2940586.6666666665, ans=0.125 2023-11-24 18:14:47,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2940653.3333333335, ans=0.0 2023-11-24 18:14:48,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441100 2023-11-24 18:14:50,284 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:14:50,707 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.42 vs. limit=6.0 2023-11-24 18:14:57,072 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.57 vs. limit=6.0 2023-11-24 18:15:01,164 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8250, loss[loss=0.07581, simple_loss=0.0991, pruned_loss=0.01721, audio_tagging_loss=0.009046, over 16284.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.091, pruned_loss=0.013, audio_tagging_loss=0.008801, over 3052828.64 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 18:15:05,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2940720.0, ans=0.0 2023-11-24 18:15:10,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2940720.0, ans=0.0 2023-11-24 18:15:33,534 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.363e+01 8.417e+01 9.119e+01 9.803e+01 1.778e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-24 18:15:34,983 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2940853.3333333335, ans=0.05 2023-11-24 18:15:36,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=2940853.3333333335, ans=10.0 2023-11-24 18:15:47,723 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0 2023-11-24 18:15:50,720 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441150 2023-11-24 18:16:02,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2941053.3333333335, ans=0.0 2023-11-24 18:16:03,869 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8300, loss[loss=0.06449, simple_loss=0.08114, pruned_loss=0.01462, audio_tagging_loss=0.009298, over 16255.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09096, pruned_loss=0.01293, audio_tagging_loss=0.008777, over 3057114.93 frames. ], batch size: 64, lr: 1.83e-03, grad_scale: 16.0 2023-11-24 18:16:15,536 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.10 vs. limit=22.5 2023-11-24 18:16:27,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2941186.6666666665, ans=0.0 2023-11-24 18:16:37,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2941186.6666666665, ans=0.2 2023-11-24 18:16:45,566 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2941253.3333333335, ans=0.125 2023-11-24 18:16:54,699 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441200 2023-11-24 18:17:07,484 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8350, loss[loss=0.0684, simple_loss=0.07874, pruned_loss=0.01645, audio_tagging_loss=0.01258, over 14219.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09137, pruned_loss=0.01291, audio_tagging_loss=0.008806, over 3045235.29 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:17:16,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.13 vs. limit=15.0 2023-11-24 18:17:28,626 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:17:32,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2941520.0, ans=0.0 2023-11-24 18:17:32,863 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2941520.0, ans=0.1 2023-11-24 18:17:37,647 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2941520.0, ans=0.125 2023-11-24 18:17:40,296 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.269e+01 8.581e+01 9.277e+01 1.007e+02 1.908e+02, threshold=1.855e+02, percent-clipped=1.0 2023-11-24 18:17:46,948 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.12 vs. limit=10.0 2023-11-24 18:17:52,393 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2941586.6666666665, ans=0.125 2023-11-24 18:17:57,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441250 2023-11-24 18:17:57,750 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2941653.3333333335, ans=0.95 2023-11-24 18:17:59,054 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.48 vs. limit=15.0 2023-11-24 18:17:59,989 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2941653.3333333335, ans=0.125 2023-11-24 18:18:09,320 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8400, loss[loss=0.04298, simple_loss=0.05715, pruned_loss=0.005938, audio_tagging_loss=0.008463, over 14110.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09069, pruned_loss=0.01287, audio_tagging_loss=0.008844, over 3045942.69 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:18:12,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2941720.0, ans=0.125 2023-11-24 18:18:14,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2941720.0, ans=0.125 2023-11-24 18:18:32,756 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.66 vs. limit=15.0 2023-11-24 18:18:42,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2941853.3333333335, ans=0.0 2023-11-24 18:18:44,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2941853.3333333335, ans=0.2 2023-11-24 18:18:45,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2941920.0, ans=0.125 2023-11-24 18:18:47,741 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.19 vs. limit=22.5 2023-11-24 18:18:59,403 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441300 2023-11-24 18:19:11,674 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8450, loss[loss=0.06418, simple_loss=0.08367, pruned_loss=0.01138, audio_tagging_loss=0.01097, over 15361.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09069, pruned_loss=0.01276, audio_tagging_loss=0.008817, over 3048270.40 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:19:34,602 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2942120.0, ans=0.1 2023-11-24 18:19:43,712 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.476e+01 8.739e+01 9.324e+01 1.024e+02 1.265e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 18:20:02,310 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441350 2023-11-24 18:20:03,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2942320.0, ans=0.05 2023-11-24 18:20:13,961 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8500, loss[loss=0.06163, simple_loss=0.08775, pruned_loss=0.01098, audio_tagging_loss=0.006775, over 14210.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.09066, pruned_loss=0.01281, audio_tagging_loss=0.008743, over 3051230.66 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:20:31,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2942453.3333333335, ans=0.125 2023-11-24 18:20:37,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2942453.3333333335, ans=0.035 2023-11-24 18:20:47,419 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2942520.0, ans=0.2 2023-11-24 18:20:54,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2942586.6666666665, ans=0.2 2023-11-24 18:20:56,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2942586.6666666665, ans=0.2 2023-11-24 18:20:58,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2942586.6666666665, ans=0.125 2023-11-24 18:21:05,368 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441400 2023-11-24 18:21:05,489 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2942653.3333333335, ans=0.04949747468305833 2023-11-24 18:21:17,309 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.whiten.whitening_limit, batch_count=2942720.0, ans=15.0 2023-11-24 18:21:17,647 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8550, loss[loss=0.06537, simple_loss=0.09715, pruned_loss=0.009908, audio_tagging_loss=0.006885, over 15795.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09079, pruned_loss=0.01289, audio_tagging_loss=0.008778, over 3047710.59 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:21:27,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2942720.0, ans=0.2 2023-11-24 18:21:36,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2942786.6666666665, ans=0.125 2023-11-24 18:21:52,034 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 8.580e+01 9.059e+01 9.638e+01 1.247e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 18:21:58,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2942920.0, ans=0.1 2023-11-24 18:22:07,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441450 2023-11-24 18:22:21,121 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8600, loss[loss=0.08033, simple_loss=0.1252, pruned_loss=0.01307, audio_tagging_loss=0.00465, over 14933.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09122, pruned_loss=0.01314, audio_tagging_loss=0.008798, over 3045317.25 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:22:41,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2943120.0, ans=0.125 2023-11-24 18:23:12,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441500 2023-11-24 18:23:19,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2943320.0, ans=0.125 2023-11-24 18:23:21,383 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.44 vs. limit=22.5 2023-11-24 18:23:22,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2943320.0, ans=0.05 2023-11-24 18:23:25,585 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8650, loss[loss=0.06512, simple_loss=0.08688, pruned_loss=0.01129, audio_tagging_loss=0.0104, over 15891.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09115, pruned_loss=0.01294, audio_tagging_loss=0.008857, over 3055555.50 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:23:30,540 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.53 vs. limit=15.0 2023-11-24 18:23:45,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2943453.3333333335, ans=0.125 2023-11-24 18:23:49,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2943520.0, ans=0.0 2023-11-24 18:23:53,397 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:23:59,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2943520.0, ans=0.0 2023-11-24 18:23:59,833 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.252e+01 8.635e+01 9.102e+01 9.726e+01 1.317e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-24 18:24:14,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2943586.6666666665, ans=0.1 2023-11-24 18:24:15,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2943653.3333333335, ans=0.1 2023-11-24 18:24:16,896 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441550 2023-11-24 18:24:17,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2943653.3333333335, ans=0.125 2023-11-24 18:24:25,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2943653.3333333335, ans=0.07 2023-11-24 18:24:28,898 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8700, loss[loss=0.06164, simple_loss=0.08495, pruned_loss=0.009943, audio_tagging_loss=0.00922, over 15210.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09089, pruned_loss=0.01289, audio_tagging_loss=0.008975, over 3055940.07 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:24:57,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.39 vs. limit=22.5 2023-11-24 18:25:06,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2943920.0, ans=0.125 2023-11-24 18:25:19,319 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441600 2023-11-24 18:25:31,363 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8750, loss[loss=0.08464, simple_loss=0.1179, pruned_loss=0.01847, audio_tagging_loss=0.007247, over 15611.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09101, pruned_loss=0.01302, audio_tagging_loss=0.009004, over 3059395.15 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:25:48,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2944120.0, ans=0.125 2023-11-24 18:26:05,770 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.362e+01 8.931e+01 9.559e+01 1.034e+02 1.434e+02, threshold=1.912e+02, percent-clipped=0.0 2023-11-24 18:26:16,195 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.77 vs. limit=6.0 2023-11-24 18:26:22,248 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441650 2023-11-24 18:26:23,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2944320.0, ans=0.125 2023-11-24 18:26:34,977 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8800, loss[loss=0.08456, simple_loss=0.1084, pruned_loss=0.02273, audio_tagging_loss=0.007657, over 15243.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.092, pruned_loss=0.01331, audio_tagging_loss=0.009075, over 3053411.09 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:26:54,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2944453.3333333335, ans=0.2 2023-11-24 18:27:04,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2944520.0, ans=0.125 2023-11-24 18:27:19,302 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.60 vs. limit=15.0 2023-11-24 18:27:20,635 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0 2023-11-24 18:27:22,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2944586.6666666665, ans=0.125 2023-11-24 18:27:25,839 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441700 2023-11-24 18:27:26,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2944653.3333333335, ans=0.0 2023-11-24 18:27:30,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2944653.3333333335, ans=0.0 2023-11-24 18:27:38,742 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8850, loss[loss=0.07758, simple_loss=0.1083, pruned_loss=0.01668, audio_tagging_loss=0.006744, over 15600.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09227, pruned_loss=0.01324, audio_tagging_loss=0.009021, over 3051219.62 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:27:42,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2944720.0, ans=0.125 2023-11-24 18:27:46,595 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.59 vs. limit=22.5 2023-11-24 18:27:49,690 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:27:54,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2944786.6666666665, ans=0.125 2023-11-24 18:27:58,987 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2944786.6666666665, ans=0.125 2023-11-24 18:28:01,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2944786.6666666665, ans=0.95 2023-11-24 18:28:12,407 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.596e+01 9.206e+01 9.875e+01 1.365e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-24 18:28:15,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2944920.0, ans=0.125 2023-11-24 18:28:16,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2944920.0, ans=0.1 2023-11-24 18:28:18,397 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.49 vs. limit=15.0 2023-11-24 18:28:28,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441750 2023-11-24 18:28:36,832 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0 2023-11-24 18:28:40,888 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8900, loss[loss=0.06842, simple_loss=0.09757, pruned_loss=0.01301, audio_tagging_loss=0.006632, over 15631.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09315, pruned_loss=0.01343, audio_tagging_loss=0.008936, over 3058987.43 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:28:45,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2945053.3333333335, ans=0.125 2023-11-24 18:28:50,391 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2945053.3333333335, ans=0.05 2023-11-24 18:28:55,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2945120.0, ans=0.125 2023-11-24 18:29:01,219 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2945120.0, ans=0.2 2023-11-24 18:29:15,264 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2945186.6666666665, ans=0.2 2023-11-24 18:29:15,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2945186.6666666665, ans=0.125 2023-11-24 18:29:33,146 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441800 2023-11-24 18:29:36,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2945320.0, ans=0.025 2023-11-24 18:29:46,779 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 8950, loss[loss=0.05365, simple_loss=0.07069, pruned_loss=0.00922, audio_tagging_loss=0.009086, over 14756.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09208, pruned_loss=0.01323, audio_tagging_loss=0.008909, over 3055849.60 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:29:49,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2945386.6666666665, ans=0.125 2023-11-24 18:30:03,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.05 vs. limit=15.0 2023-11-24 18:30:10,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2945520.0, ans=0.0 2023-11-24 18:30:12,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2945520.0, ans=0.0 2023-11-24 18:30:16,748 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2945520.0, ans=0.0 2023-11-24 18:30:19,297 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2945520.0, ans=0.125 2023-11-24 18:30:20,104 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.368e+01 8.707e+01 9.494e+01 1.019e+02 1.254e+02, threshold=1.899e+02, percent-clipped=0.0 2023-11-24 18:30:20,356 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2945520.0, ans=0.2 2023-11-24 18:30:37,503 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441850 2023-11-24 18:30:45,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2945653.3333333335, ans=0.0 2023-11-24 18:30:50,113 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9000, loss[loss=0.07599, simple_loss=0.1042, pruned_loss=0.01672, audio_tagging_loss=0.007171, over 16590.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09231, pruned_loss=0.01339, audio_tagging_loss=0.008859, over 3063524.57 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:30:50,114 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 18:31:24,098 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2712, 4.2600, 4.4632, 4.4382], device='cuda:3') 2023-11-24 18:31:31,485 INFO [train_asr.py:1253] (3/4) Epoch 37, validation: loss=0.05871, simple_loss=0.05072, pruned_loss=0.005135, audio_tagging_loss=0.02821, over 4681554.00 frames. 2023-11-24 18:31:31,485 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 18:31:37,180 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.90 vs. limit=15.0 2023-11-24 18:31:43,435 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2945786.6666666665, ans=0.125 2023-11-24 18:31:46,116 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0 2023-11-24 18:31:56,390 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.47 vs. limit=8.0 2023-11-24 18:32:14,072 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2945920.0, ans=0.0 2023-11-24 18:32:22,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441900 2023-11-24 18:32:30,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2945986.6666666665, ans=0.0 2023-11-24 18:32:34,939 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9050, loss[loss=0.07931, simple_loss=0.112, pruned_loss=0.01686, audio_tagging_loss=0.006445, over 15292.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09222, pruned_loss=0.01313, audio_tagging_loss=0.008766, over 3059005.75 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:32:39,657 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.97 vs. limit=22.5 2023-11-24 18:32:45,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2946053.3333333335, ans=22.5 2023-11-24 18:32:53,236 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.41 vs. limit=15.0 2023-11-24 18:32:54,061 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2946120.0, ans=0.0 2023-11-24 18:33:03,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2946186.6666666665, ans=0.1 2023-11-24 18:33:09,592 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.312e+01 8.396e+01 9.037e+01 9.861e+01 1.283e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 18:33:21,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2946253.3333333335, ans=0.1 2023-11-24 18:33:24,532 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 441950 2023-11-24 18:33:31,221 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.36 vs. limit=5.0 2023-11-24 18:33:37,555 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9100, loss[loss=0.07587, simple_loss=0.1039, pruned_loss=0.01598, audio_tagging_loss=0.007925, over 14888.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09257, pruned_loss=0.01327, audio_tagging_loss=0.008688, over 3060826.47 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:33:49,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2946453.3333333335, ans=0.125 2023-11-24 18:34:06,825 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.82 vs. limit=22.5 2023-11-24 18:34:26,640 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2946653.3333333335, ans=0.0 2023-11-24 18:34:27,590 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442000 2023-11-24 18:34:30,517 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2946653.3333333335, ans=0.125 2023-11-24 18:34:38,097 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.28 vs. limit=10.0 2023-11-24 18:34:39,638 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9150, loss[loss=0.07386, simple_loss=0.1023, pruned_loss=0.01418, audio_tagging_loss=0.008512, over 15925.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09186, pruned_loss=0.01328, audio_tagging_loss=0.008748, over 3055604.95 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:34:42,874 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2946720.0, ans=0.125 2023-11-24 18:35:11,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2946853.3333333335, ans=0.0 2023-11-24 18:35:15,206 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.510e+01 9.159e+01 9.829e+01 1.251e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 18:35:16,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2946920.0, ans=0.2 2023-11-24 18:35:30,170 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442050 2023-11-24 18:35:42,492 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9200, loss[loss=0.08341, simple_loss=0.1112, pruned_loss=0.01868, audio_tagging_loss=0.009129, over 15503.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09151, pruned_loss=0.01325, audio_tagging_loss=0.008663, over 3053693.77 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:36:24,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2947253.3333333335, ans=0.125 2023-11-24 18:36:32,592 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442100 2023-11-24 18:36:35,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.95 vs. limit=22.5 2023-11-24 18:36:45,619 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9250, loss[loss=0.07312, simple_loss=0.103, pruned_loss=0.01416, audio_tagging_loss=0.007438, over 16594.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.0906, pruned_loss=0.01306, audio_tagging_loss=0.008629, over 3054834.25 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:37:02,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2947453.3333333335, ans=0.125 2023-11-24 18:37:12,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2947520.0, ans=0.0 2023-11-24 18:37:15,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2947520.0, ans=0.05 2023-11-24 18:37:18,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.27 vs. limit=22.5 2023-11-24 18:37:19,952 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.271e+01 8.726e+01 9.340e+01 9.852e+01 1.345e+02, threshold=1.868e+02, percent-clipped=0.0 2023-11-24 18:37:29,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2947586.6666666665, ans=0.0 2023-11-24 18:37:35,658 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442150 2023-11-24 18:37:37,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2947653.3333333335, ans=0.125 2023-11-24 18:37:47,424 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9300, loss[loss=0.07888, simple_loss=0.1036, pruned_loss=0.01735, audio_tagging_loss=0.009738, over 14485.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.0903, pruned_loss=0.01306, audio_tagging_loss=0.008694, over 3044578.96 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:37:59,653 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=14.61 vs. limit=22.5 2023-11-24 18:38:09,011 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2947786.6666666665, ans=0.0 2023-11-24 18:38:14,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2947853.3333333335, ans=0.125 2023-11-24 18:38:27,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2947920.0, ans=0.125 2023-11-24 18:38:35,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2947920.0, ans=0.125 2023-11-24 18:38:37,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442200 2023-11-24 18:38:46,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2947986.6666666665, ans=0.2 2023-11-24 18:38:51,073 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9350, loss[loss=0.0457, simple_loss=0.0545, pruned_loss=0.006539, audio_tagging_loss=0.01191, over 15337.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09057, pruned_loss=0.01306, audio_tagging_loss=0.008711, over 3053631.38 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:39:04,836 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2948120.0, ans=0.1 2023-11-24 18:39:12,882 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.72 vs. limit=6.0 2023-11-24 18:39:26,933 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.625e+01 9.145e+01 9.875e+01 1.374e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 18:39:33,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2948253.3333333335, ans=0.0 2023-11-24 18:39:37,867 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.83 vs. limit=10.0 2023-11-24 18:39:40,962 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442250 2023-11-24 18:39:53,306 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9400, loss[loss=0.07425, simple_loss=0.09375, pruned_loss=0.01622, audio_tagging_loss=0.01116, over 16199.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09127, pruned_loss=0.01321, audio_tagging_loss=0.008707, over 3051493.77 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:40:16,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2948453.3333333335, ans=0.125 2023-11-24 18:40:38,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2948586.6666666665, ans=0.125 2023-11-24 18:40:38,521 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2948586.6666666665, ans=0.0 2023-11-24 18:40:44,062 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442300 2023-11-24 18:40:49,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2948653.3333333335, ans=0.1 2023-11-24 18:40:52,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2948653.3333333335, ans=0.1 2023-11-24 18:40:53,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2948653.3333333335, ans=0.07 2023-11-24 18:40:54,223 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:40:56,584 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9450, loss[loss=0.06918, simple_loss=0.0947, pruned_loss=0.01584, audio_tagging_loss=0.005991, over 15160.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09155, pruned_loss=0.01312, audio_tagging_loss=0.008767, over 3049613.92 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:40:59,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2948720.0, ans=0.0 2023-11-24 18:41:12,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2948786.6666666665, ans=0.1 2023-11-24 18:41:13,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2948786.6666666665, ans=0.1 2023-11-24 18:41:20,534 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2948853.3333333335, ans=0.0 2023-11-24 18:41:23,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2948853.3333333335, ans=0.125 2023-11-24 18:41:23,318 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.12 vs. limit=10.0 2023-11-24 18:41:27,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2948853.3333333335, ans=0.125 2023-11-24 18:41:33,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.255e+01 8.857e+01 9.505e+01 1.052e+02 1.346e+02, threshold=1.901e+02, percent-clipped=0.0 2023-11-24 18:41:38,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2948920.0, ans=0.125 2023-11-24 18:41:43,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2948920.0, ans=0.1 2023-11-24 18:41:47,128 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442350 2023-11-24 18:41:47,211 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2948986.6666666665, ans=0.125 2023-11-24 18:41:48,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2948986.6666666665, ans=0.1 2023-11-24 18:41:53,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2948986.6666666665, ans=0.125 2023-11-24 18:41:54,131 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.32 vs. limit=22.5 2023-11-24 18:41:58,691 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.01 vs. limit=15.0 2023-11-24 18:41:59,895 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9500, loss[loss=0.06854, simple_loss=0.1004, pruned_loss=0.01237, audio_tagging_loss=0.005987, over 14596.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09232, pruned_loss=0.01316, audio_tagging_loss=0.008804, over 3047398.61 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:42:04,216 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.58 vs. limit=15.0 2023-11-24 18:42:21,279 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.74 vs. limit=6.0 2023-11-24 18:42:44,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2949253.3333333335, ans=0.125 2023-11-24 18:42:49,693 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442400 2023-11-24 18:43:02,623 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9550, loss[loss=0.06789, simple_loss=0.0924, pruned_loss=0.01224, audio_tagging_loss=0.009442, over 16146.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09244, pruned_loss=0.01305, audio_tagging_loss=0.008887, over 3054633.70 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:43:38,730 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.675e+01 9.207e+01 9.929e+01 1.581e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-24 18:43:41,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2949586.6666666665, ans=0.125 2023-11-24 18:43:43,735 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2949586.6666666665, ans=0.125 2023-11-24 18:43:52,571 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442450 2023-11-24 18:43:54,986 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2949653.3333333335, ans=0.0 2023-11-24 18:44:04,422 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9600, loss[loss=0.06261, simple_loss=0.08145, pruned_loss=0.01426, audio_tagging_loss=0.007629, over 14844.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09248, pruned_loss=0.01299, audio_tagging_loss=0.00887, over 3052985.39 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:44:09,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2949720.0, ans=0.5 2023-11-24 18:44:11,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2949720.0, ans=0.0 2023-11-24 18:44:16,360 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2949786.6666666665, ans=0.2 2023-11-24 18:44:19,200 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.16 vs. limit=12.0 2023-11-24 18:44:39,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2949853.3333333335, ans=0.2 2023-11-24 18:44:54,662 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442500 2023-11-24 18:45:07,142 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9650, loss[loss=0.06741, simple_loss=0.09818, pruned_loss=0.01285, audio_tagging_loss=0.005471, over 15260.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09166, pruned_loss=0.01273, audio_tagging_loss=0.008897, over 3047142.71 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:45:12,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2950053.3333333335, ans=0.125 2023-11-24 18:45:21,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2950120.0, ans=0.04949747468305833 2023-11-24 18:45:36,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2950186.6666666665, ans=0.125 2023-11-24 18:45:43,084 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.493e+01 8.425e+01 9.108e+01 9.681e+01 1.366e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-24 18:45:57,327 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442550 2023-11-24 18:45:57,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2950320.0, ans=0.125 2023-11-24 18:46:09,227 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9700, loss[loss=0.06496, simple_loss=0.08742, pruned_loss=0.01239, audio_tagging_loss=0.008856, over 14948.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09132, pruned_loss=0.01272, audio_tagging_loss=0.008923, over 3042016.64 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:46:18,459 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=2950386.6666666665, ans=0.5 2023-11-24 18:46:56,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2950586.6666666665, ans=0.125 2023-11-24 18:46:57,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2950586.6666666665, ans=0.1 2023-11-24 18:47:00,203 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442600 2023-11-24 18:47:01,452 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2950653.3333333335, ans=0.125 2023-11-24 18:47:12,416 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9750, loss[loss=0.06696, simple_loss=0.0939, pruned_loss=0.01179, audio_tagging_loss=0.008216, over 15045.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09236, pruned_loss=0.01278, audio_tagging_loss=0.008814, over 3045644.54 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:47:13,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2950720.0, ans=0.125 2023-11-24 18:47:20,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2950720.0, ans=0.125 2023-11-24 18:47:21,509 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0 2023-11-24 18:47:22,658 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:47:31,785 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.48 vs. limit=15.0 2023-11-24 18:47:48,960 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.947e+01 8.439e+01 9.144e+01 9.970e+01 1.220e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 18:48:02,047 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442650 2023-11-24 18:48:14,354 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9800, loss[loss=0.05149, simple_loss=0.06708, pruned_loss=0.00889, audio_tagging_loss=0.009058, over 15474.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09169, pruned_loss=0.01282, audio_tagging_loss=0.008704, over 3039631.66 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:48:15,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2951053.3333333335, ans=0.2 2023-11-24 18:48:23,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2951053.3333333335, ans=0.2 2023-11-24 18:48:39,224 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2951186.6666666665, ans=0.125 2023-11-24 18:48:40,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2951186.6666666665, ans=0.0 2023-11-24 18:48:49,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2951253.3333333335, ans=0.1 2023-11-24 18:48:51,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2951253.3333333335, ans=0.0 2023-11-24 18:49:02,881 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2951320.0, ans=0.125 2023-11-24 18:49:03,869 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442700 2023-11-24 18:49:04,510 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.66 vs. limit=12.0 2023-11-24 18:49:09,059 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:49:11,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2951320.0, ans=0.0 2023-11-24 18:49:16,070 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9850, loss[loss=0.0756, simple_loss=0.1069, pruned_loss=0.01218, audio_tagging_loss=0.009989, over 15220.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09213, pruned_loss=0.01299, audio_tagging_loss=0.008573, over 3040296.73 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:49:49,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2951520.0, ans=0.125 2023-11-24 18:49:53,631 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.535e+01 9.041e+01 9.752e+01 1.279e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 18:50:05,512 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442750 2023-11-24 18:50:17,732 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9900, loss[loss=0.05683, simple_loss=0.08011, pruned_loss=0.007654, audio_tagging_loss=0.009121, over 13891.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09193, pruned_loss=0.0129, audio_tagging_loss=0.008607, over 3041206.64 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:50:23,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2951720.0, ans=0.025 2023-11-24 18:50:32,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2951786.6666666665, ans=0.125 2023-11-24 18:50:38,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2951786.6666666665, ans=0.0 2023-11-24 18:50:46,060 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:50:50,340 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.52 vs. limit=10.0 2023-11-24 18:51:02,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2951920.0, ans=0.2 2023-11-24 18:51:07,641 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442800 2023-11-24 18:51:08,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2951986.6666666665, ans=0.125 2023-11-24 18:51:08,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2951986.6666666665, ans=0.125 2023-11-24 18:51:19,592 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 9950, loss[loss=0.05578, simple_loss=0.07112, pruned_loss=0.009456, audio_tagging_loss=0.01077, over 13183.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09241, pruned_loss=0.01301, audio_tagging_loss=0.008641, over 3041869.50 frames. ], batch size: 52, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:51:22,932 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2952053.3333333335, ans=0.125 2023-11-24 18:51:38,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2952120.0, ans=0.125 2023-11-24 18:51:57,277 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.463e+01 9.019e+01 9.662e+01 1.211e+02, threshold=1.804e+02, percent-clipped=0.0 2023-11-24 18:52:06,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2952253.3333333335, ans=0.0 2023-11-24 18:52:10,139 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442850 2023-11-24 18:52:23,028 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10000, loss[loss=0.06311, simple_loss=0.08394, pruned_loss=0.01287, audio_tagging_loss=0.008265, over 15793.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09148, pruned_loss=0.01291, audio_tagging_loss=0.00866, over 3044890.01 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 18:52:25,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2952386.6666666665, ans=0.0 2023-11-24 18:52:34,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2952453.3333333335, ans=0.125 2023-11-24 18:52:38,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2952453.3333333335, ans=0.2 2023-11-24 18:53:04,100 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.09 vs. limit=6.0 2023-11-24 18:53:07,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2952586.6666666665, ans=0.125 2023-11-24 18:53:11,972 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442900 2023-11-24 18:53:24,191 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10050, loss[loss=0.07097, simple_loss=0.09439, pruned_loss=0.01577, audio_tagging_loss=0.008009, over 15086.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.0918, pruned_loss=0.01309, audio_tagging_loss=0.008704, over 3046325.02 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:53:31,718 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.67 vs. limit=15.0 2023-11-24 18:53:49,119 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.82 vs. limit=22.5 2023-11-24 18:53:55,894 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.17 vs. limit=15.0 2023-11-24 18:53:58,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2952853.3333333335, ans=0.2 2023-11-24 18:54:02,841 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.473e+01 8.442e+01 9.039e+01 9.729e+01 1.134e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 18:54:03,251 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2952920.0, ans=0.125 2023-11-24 18:54:12,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2952986.6666666665, ans=0.1 2023-11-24 18:54:13,539 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 442950 2023-11-24 18:54:14,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2952986.6666666665, ans=0.1 2023-11-24 18:54:25,236 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10100, loss[loss=0.07274, simple_loss=0.0907, pruned_loss=0.01539, audio_tagging_loss=0.01201, over 15443.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09221, pruned_loss=0.01318, audio_tagging_loss=0.008661, over 3045050.54 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:54:38,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2953120.0, ans=0.125 2023-11-24 18:54:48,358 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2953120.0, ans=0.125 2023-11-24 18:54:54,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2953186.6666666665, ans=0.0 2023-11-24 18:55:00,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2953186.6666666665, ans=0.125 2023-11-24 18:55:01,871 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:55:14,625 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:55:15,318 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443000 2023-11-24 18:55:28,606 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10150, loss[loss=0.0558, simple_loss=0.07228, pruned_loss=0.008077, audio_tagging_loss=0.01159, over 15304.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09133, pruned_loss=0.01304, audio_tagging_loss=0.008775, over 3048488.56 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:55:44,260 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=22.5 2023-11-24 18:55:48,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2953453.3333333335, ans=0.125 2023-11-24 18:55:50,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2953453.3333333335, ans=0.125 2023-11-24 18:55:56,457 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:56:06,392 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.628e+01 9.087e+01 9.762e+01 1.339e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 18:56:18,462 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443050 2023-11-24 18:56:30,715 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10200, loss[loss=0.05429, simple_loss=0.07474, pruned_loss=0.009039, audio_tagging_loss=0.007885, over 15499.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09151, pruned_loss=0.01304, audio_tagging_loss=0.008775, over 3050186.41 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:56:52,744 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 18:57:07,529 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0 2023-11-24 18:57:09,428 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 18:57:12,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.93 vs. limit=15.0 2023-11-24 18:57:18,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2953920.0, ans=0.1 2023-11-24 18:57:20,407 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443100 2023-11-24 18:57:26,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2953986.6666666665, ans=0.125 2023-11-24 18:57:32,188 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10250, loss[loss=0.04814, simple_loss=0.06057, pruned_loss=0.007943, audio_tagging_loss=0.009914, over 14592.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09079, pruned_loss=0.01287, audio_tagging_loss=0.008874, over 3060211.74 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:57:35,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten.whitening_limit, batch_count=2954053.3333333335, ans=22.5 2023-11-24 18:58:09,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.13 vs. limit=10.0 2023-11-24 18:58:11,696 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.337e+01 8.466e+01 9.132e+01 9.893e+01 1.266e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 18:58:22,730 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443150 2023-11-24 18:58:35,794 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10300, loss[loss=0.06912, simple_loss=0.09419, pruned_loss=0.01196, audio_tagging_loss=0.01007, over 17811.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.09026, pruned_loss=0.01274, audio_tagging_loss=0.00899, over 3059721.95 frames. ], batch size: 67, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:58:43,054 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2954386.6666666665, ans=0.1 2023-11-24 18:58:48,304 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.26 vs. limit=15.0 2023-11-24 18:58:49,218 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2954453.3333333335, ans=0.0 2023-11-24 18:59:25,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443200 2023-11-24 18:59:30,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2954653.3333333335, ans=0.125 2023-11-24 18:59:39,370 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10350, loss[loss=0.06514, simple_loss=0.08137, pruned_loss=0.01278, audio_tagging_loss=0.01168, over 14589.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09084, pruned_loss=0.01273, audio_tagging_loss=0.009128, over 3061342.27 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 18:59:49,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2954720.0, ans=0.125 2023-11-24 18:59:49,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2954720.0, ans=0.1 2023-11-24 19:00:10,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2954853.3333333335, ans=0.5 2023-11-24 19:00:15,153 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2954920.0, ans=0.125 2023-11-24 19:00:16,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2954920.0, ans=0.1 2023-11-24 19:00:17,087 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.601e+01 8.606e+01 9.110e+01 9.960e+01 1.328e+02, threshold=1.822e+02, percent-clipped=0.0 2023-11-24 19:00:17,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2954920.0, ans=0.0 2023-11-24 19:00:19,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2954920.0, ans=0.2 2023-11-24 19:00:28,632 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443250 2023-11-24 19:00:40,319 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10400, loss[loss=0.07285, simple_loss=0.1019, pruned_loss=0.01391, audio_tagging_loss=0.00802, over 15071.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09032, pruned_loss=0.01273, audio_tagging_loss=0.009352, over 3064192.96 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 19:00:45,735 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.54 vs. limit=22.5 2023-11-24 19:00:51,887 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2955120.0, ans=0.0 2023-11-24 19:01:23,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2955253.3333333335, ans=0.125 2023-11-24 19:01:30,226 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443300 2023-11-24 19:01:35,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2955320.0, ans=6.0 2023-11-24 19:01:43,033 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10450, loss[loss=0.07147, simple_loss=0.09772, pruned_loss=0.01315, audio_tagging_loss=0.009459, over 15360.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09013, pruned_loss=0.01275, audio_tagging_loss=0.009294, over 3065440.38 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:01:59,138 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2955453.3333333335, ans=0.125 2023-11-24 19:02:04,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2955453.3333333335, ans=0.1 2023-11-24 19:02:14,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2955520.0, ans=0.0 2023-11-24 19:02:15,273 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2955520.0, ans=0.125 2023-11-24 19:02:15,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2955520.0, ans=0.125 2023-11-24 19:02:17,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2955520.0, ans=0.1 2023-11-24 19:02:22,865 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.090e+01 8.670e+01 9.376e+01 1.004e+02 1.489e+02, threshold=1.875e+02, percent-clipped=0.0 2023-11-24 19:02:33,005 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443350 2023-11-24 19:02:40,184 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:02:45,217 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10500, loss[loss=0.06215, simple_loss=0.09013, pruned_loss=0.007848, audio_tagging_loss=0.009237, over 14261.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08917, pruned_loss=0.01263, audio_tagging_loss=0.009257, over 3058980.38 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 8.0 2023-11-24 19:02:49,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2955720.0, ans=0.0 2023-11-24 19:02:54,443 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2955720.0, ans=0.0 2023-11-24 19:03:03,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2955786.6666666665, ans=0.125 2023-11-24 19:03:17,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2955853.3333333335, ans=0.0 2023-11-24 19:03:24,291 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.65 vs. limit=10.0 2023-11-24 19:03:35,037 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443400 2023-11-24 19:03:43,484 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2955986.6666666665, ans=0.125 2023-11-24 19:03:48,022 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10550, loss[loss=0.0684, simple_loss=0.09422, pruned_loss=0.01276, audio_tagging_loss=0.008535, over 14740.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.0902, pruned_loss=0.0128, audio_tagging_loss=0.009081, over 3063777.84 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 8.0 2023-11-24 19:03:57,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2956053.3333333335, ans=0.1 2023-11-24 19:04:05,563 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.85 vs. limit=22.5 2023-11-24 19:04:06,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2956120.0, ans=0.0 2023-11-24 19:04:07,643 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=9.86 vs. limit=22.5 2023-11-24 19:04:14,205 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2956186.6666666665, ans=0.0 2023-11-24 19:04:14,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2956186.6666666665, ans=0.1 2023-11-24 19:04:17,659 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2956186.6666666665, ans=0.125 2023-11-24 19:04:19,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2956186.6666666665, ans=0.0 2023-11-24 19:04:24,162 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2956253.3333333335, ans=0.0 2023-11-24 19:04:25,311 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2956253.3333333335, ans=0.125 2023-11-24 19:04:29,168 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.503e+01 9.037e+01 9.861e+01 1.136e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 19:04:35,389 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2956253.3333333335, ans=0.0 2023-11-24 19:04:37,548 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443450 2023-11-24 19:04:49,852 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10600, loss[loss=0.07084, simple_loss=0.09487, pruned_loss=0.01348, audio_tagging_loss=0.00993, over 15696.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09097, pruned_loss=0.0129, audio_tagging_loss=0.008905, over 3064002.12 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 8.0 2023-11-24 19:04:56,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2956386.6666666665, ans=0.05 2023-11-24 19:04:56,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2956386.6666666665, ans=0.125 2023-11-24 19:04:58,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2956386.6666666665, ans=0.0 2023-11-24 19:05:02,291 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:05:21,199 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2956520.0, ans=0.125 2023-11-24 19:05:39,465 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443500 2023-11-24 19:05:50,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2956720.0, ans=0.125 2023-11-24 19:05:51,795 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10650, loss[loss=0.06738, simple_loss=0.09214, pruned_loss=0.01492, audio_tagging_loss=0.006388, over 15857.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09101, pruned_loss=0.0128, audio_tagging_loss=0.008825, over 3064964.78 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 8.0 2023-11-24 19:05:53,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2956720.0, ans=10.0 2023-11-24 19:06:07,007 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.84 vs. limit=12.0 2023-11-24 19:06:13,821 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2956786.6666666665, ans=0.125 2023-11-24 19:06:29,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2956920.0, ans=0.2 2023-11-24 19:06:33,393 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.594e+01 8.793e+01 9.512e+01 1.031e+02 1.281e+02, threshold=1.902e+02, percent-clipped=0.0 2023-11-24 19:06:38,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2956920.0, ans=0.125 2023-11-24 19:06:42,711 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443550 2023-11-24 19:06:45,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2956986.6666666665, ans=0.125 2023-11-24 19:06:52,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2956986.6666666665, ans=0.05 2023-11-24 19:06:55,216 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10700, loss[loss=0.05376, simple_loss=0.06918, pruned_loss=0.009684, audio_tagging_loss=0.009489, over 14991.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09128, pruned_loss=0.01287, audio_tagging_loss=0.008788, over 3063546.52 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 8.0 2023-11-24 19:07:00,717 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.24 vs. limit=15.0 2023-11-24 19:07:21,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2957186.6666666665, ans=0.125 2023-11-24 19:07:38,702 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2957253.3333333335, ans=0.125 2023-11-24 19:07:43,320 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.90 vs. limit=15.0 2023-11-24 19:07:45,009 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443600 2023-11-24 19:07:57,786 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10750, loss[loss=0.07641, simple_loss=0.09317, pruned_loss=0.01618, audio_tagging_loss=0.01365, over 16314.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09128, pruned_loss=0.01281, audio_tagging_loss=0.00882, over 3063068.60 frames. ], batch size: 62, lr: 1.82e-03, grad_scale: 8.0 2023-11-24 19:08:01,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2957386.6666666665, ans=0.125 2023-11-24 19:08:23,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2957520.0, ans=0.1 2023-11-24 19:08:37,954 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 8.444e+01 9.135e+01 9.725e+01 3.439e+02, threshold=1.827e+02, percent-clipped=1.0 2023-11-24 19:08:46,984 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443650 2023-11-24 19:08:49,526 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2957653.3333333335, ans=0.2 2023-11-24 19:08:51,139 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2957653.3333333335, ans=0.125 2023-11-24 19:08:59,156 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10800, loss[loss=0.06462, simple_loss=0.08727, pruned_loss=0.009137, audio_tagging_loss=0.01185, over 16083.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.0911, pruned_loss=0.01287, audio_tagging_loss=0.008778, over 3054968.27 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:09:00,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2957720.0, ans=0.125 2023-11-24 19:09:24,365 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.21 vs. limit=15.0 2023-11-24 19:09:28,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2957853.3333333335, ans=0.125 2023-11-24 19:09:41,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2957920.0, ans=0.125 2023-11-24 19:09:46,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2957920.0, ans=0.0 2023-11-24 19:09:48,873 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443700 2023-11-24 19:10:01,071 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10850, loss[loss=0.05883, simple_loss=0.08246, pruned_loss=0.009858, audio_tagging_loss=0.007739, over 14799.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09121, pruned_loss=0.01293, audio_tagging_loss=0.008751, over 3050616.90 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:10:02,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2958053.3333333335, ans=0.125 2023-11-24 19:10:41,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2958253.3333333335, ans=0.0 2023-11-24 19:10:43,177 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.404e+01 9.026e+01 9.467e+01 1.240e+02, threshold=1.805e+02, percent-clipped=0.0 2023-11-24 19:10:44,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2958253.3333333335, ans=0.05 2023-11-24 19:10:45,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2958253.3333333335, ans=0.125 2023-11-24 19:10:51,551 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443750 2023-11-24 19:10:59,088 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 19:11:04,360 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10900, loss[loss=0.06453, simple_loss=0.09148, pruned_loss=0.01114, audio_tagging_loss=0.007659, over 16364.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.091, pruned_loss=0.01296, audio_tagging_loss=0.00879, over 3053400.71 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:11:09,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2958386.6666666665, ans=0.125 2023-11-24 19:11:10,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2958386.6666666665, ans=0.1 2023-11-24 19:11:19,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2958453.3333333335, ans=0.125 2023-11-24 19:11:36,571 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2958520.0, ans=0.0 2023-11-24 19:11:52,674 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.38 vs. limit=12.0 2023-11-24 19:11:54,374 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443800 2023-11-24 19:11:58,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2958653.3333333335, ans=0.1 2023-11-24 19:12:06,965 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 10950, loss[loss=0.06934, simple_loss=0.09136, pruned_loss=0.01464, audio_tagging_loss=0.009017, over 15655.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.09054, pruned_loss=0.01283, audio_tagging_loss=0.008858, over 3048384.41 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:12:11,188 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.87 vs. limit=15.0 2023-11-24 19:12:48,387 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.471e+01 9.149e+01 9.860e+01 1.271e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 19:12:56,193 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.25 vs. limit=10.0 2023-11-24 19:12:56,861 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443850 2023-11-24 19:13:08,961 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11000, loss[loss=0.07689, simple_loss=0.1109, pruned_loss=0.01401, audio_tagging_loss=0.007411, over 16550.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09161, pruned_loss=0.01289, audio_tagging_loss=0.008904, over 3043930.13 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:13:15,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2959053.3333333335, ans=0.125 2023-11-24 19:13:17,573 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 19:13:25,897 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.01 vs. limit=12.0 2023-11-24 19:13:33,741 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.92 vs. limit=15.0 2023-11-24 19:13:58,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443900 2023-11-24 19:14:09,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2959386.6666666665, ans=0.0 2023-11-24 19:14:10,683 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11050, loss[loss=0.06194, simple_loss=0.0805, pruned_loss=0.01141, audio_tagging_loss=0.01028, over 15225.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09164, pruned_loss=0.01278, audio_tagging_loss=0.00902, over 3050309.11 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:14:20,988 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0 2023-11-24 19:14:24,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2959453.3333333335, ans=0.0 2023-11-24 19:14:36,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2959520.0, ans=0.07 2023-11-24 19:14:52,688 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.475e+01 8.793e+01 9.326e+01 1.000e+02 1.252e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 19:14:54,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2959586.6666666665, ans=0.015 2023-11-24 19:15:01,787 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 443950 2023-11-24 19:15:14,596 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11100, loss[loss=0.04875, simple_loss=0.06401, pruned_loss=0.007784, audio_tagging_loss=0.008959, over 14698.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09141, pruned_loss=0.01276, audio_tagging_loss=0.009018, over 3055032.42 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:15:19,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2959720.0, ans=0.125 2023-11-24 19:15:40,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2959853.3333333335, ans=0.2 2023-11-24 19:15:41,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2959853.3333333335, ans=0.5 2023-11-24 19:15:41,620 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2959853.3333333335, ans=0.125 2023-11-24 19:15:55,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2959920.0, ans=0.0 2023-11-24 19:16:04,354 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444000 2023-11-24 19:16:11,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2959986.6666666665, ans=0.2 2023-11-24 19:16:14,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2959986.6666666665, ans=0.125 2023-11-24 19:16:21,567 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11150, loss[loss=0.05086, simple_loss=0.06519, pruned_loss=0.00596, audio_tagging_loss=0.0123, over 15107.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09124, pruned_loss=0.01276, audio_tagging_loss=0.009178, over 3053570.23 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:16:40,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2960120.0, ans=0.0 2023-11-24 19:16:40,695 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.17 vs. limit=15.0 2023-11-24 19:17:02,996 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.194e+01 8.502e+01 9.253e+01 1.001e+02 1.275e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 19:17:11,597 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444050 2023-11-24 19:17:22,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2960386.6666666665, ans=0.1 2023-11-24 19:17:23,209 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11200, loss[loss=0.05549, simple_loss=0.07103, pruned_loss=0.0106, audio_tagging_loss=0.009372, over 15501.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09106, pruned_loss=0.01285, audio_tagging_loss=0.009281, over 3059077.40 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 19:17:56,546 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2960520.0, ans=0.125 2023-11-24 19:18:13,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444100 2023-11-24 19:18:13,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2960653.3333333335, ans=0.125 2023-11-24 19:18:26,579 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11250, loss[loss=0.07133, simple_loss=0.09279, pruned_loss=0.01591, audio_tagging_loss=0.009022, over 14604.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09054, pruned_loss=0.01279, audio_tagging_loss=0.009239, over 3050190.59 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 19:18:36,921 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0 2023-11-24 19:18:46,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2960786.6666666665, ans=0.125 2023-11-24 19:18:47,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2960786.6666666665, ans=0.1 2023-11-24 19:19:06,249 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.64 vs. limit=15.0 2023-11-24 19:19:08,007 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.311e+01 8.458e+01 9.183e+01 1.013e+02 1.310e+02, threshold=1.837e+02, percent-clipped=0.0 2023-11-24 19:19:16,059 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444150 2023-11-24 19:19:21,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2960986.6666666665, ans=15.0 2023-11-24 19:19:28,408 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11300, loss[loss=0.06068, simple_loss=0.08406, pruned_loss=0.009552, audio_tagging_loss=0.009098, over 15924.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.0903, pruned_loss=0.01264, audio_tagging_loss=0.00909, over 3048264.78 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:19:29,037 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.01 vs. limit=15.0 2023-11-24 19:19:30,930 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2961053.3333333335, ans=0.125 2023-11-24 19:19:34,504 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:19:34,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2961053.3333333335, ans=0.125 2023-11-24 19:20:18,496 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444200 2023-11-24 19:20:27,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.98 vs. limit=15.0 2023-11-24 19:20:30,539 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11350, loss[loss=0.04975, simple_loss=0.06468, pruned_loss=0.007312, audio_tagging_loss=0.0101, over 15249.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09092, pruned_loss=0.01278, audio_tagging_loss=0.008915, over 3047304.71 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:21:12,886 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.844e+01 9.278e+01 1.039e+02 2.071e+02, threshold=1.856e+02, percent-clipped=1.0 2023-11-24 19:21:13,626 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.61 vs. limit=6.0 2023-11-24 19:21:20,345 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444250 2023-11-24 19:21:24,768 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2961653.3333333335, ans=0.125 2023-11-24 19:21:32,755 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11400, loss[loss=0.08159, simple_loss=0.1187, pruned_loss=0.01583, audio_tagging_loss=0.006406, over 15583.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09139, pruned_loss=0.01289, audio_tagging_loss=0.008813, over 3043444.19 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:21:42,650 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.99 vs. limit=15.0 2023-11-24 19:21:58,254 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2961853.3333333335, ans=0.125 2023-11-24 19:22:01,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.41 vs. limit=22.5 2023-11-24 19:22:23,248 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444300 2023-11-24 19:22:26,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2961986.6666666665, ans=0.2 2023-11-24 19:22:36,161 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11450, loss[loss=0.0773, simple_loss=0.09567, pruned_loss=0.01799, audio_tagging_loss=0.01148, over 15618.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09134, pruned_loss=0.01302, audio_tagging_loss=0.008819, over 3042409.63 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:23:08,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2962186.6666666665, ans=0.1 2023-11-24 19:23:12,401 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.82 vs. limit=12.0 2023-11-24 19:23:18,420 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.568e+01 8.582e+01 9.161e+01 1.011e+02 1.144e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 19:23:26,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444350 2023-11-24 19:23:38,037 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11500, loss[loss=0.08152, simple_loss=0.1062, pruned_loss=0.01884, audio_tagging_loss=0.009589, over 14616.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09143, pruned_loss=0.01318, audio_tagging_loss=0.00877, over 3042623.16 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:24:00,121 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.64 vs. limit=22.5 2023-11-24 19:24:00,186 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.31 vs. limit=15.0 2023-11-24 19:24:28,123 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444400 2023-11-24 19:24:40,763 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11550, loss[loss=0.0657, simple_loss=0.0817, pruned_loss=0.0148, audio_tagging_loss=0.01006, over 14370.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09129, pruned_loss=0.0132, audio_tagging_loss=0.008814, over 3037079.33 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:24:51,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2962720.0, ans=0.2 2023-11-24 19:24:55,322 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2962786.6666666665, ans=0.125 2023-11-24 19:24:56,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2962786.6666666665, ans=0.125 2023-11-24 19:25:06,831 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2962853.3333333335, ans=10.0 2023-11-24 19:25:13,817 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:25:17,126 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 19:25:23,486 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.651e+01 8.528e+01 9.073e+01 9.782e+01 1.310e+02, threshold=1.815e+02, percent-clipped=0.0 2023-11-24 19:25:31,220 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444450 2023-11-24 19:25:33,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2962986.6666666665, ans=0.0 2023-11-24 19:25:34,105 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.71 vs. limit=10.0 2023-11-24 19:25:43,433 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11600, loss[loss=0.08333, simple_loss=0.1034, pruned_loss=0.02325, audio_tagging_loss=0.0084, over 15429.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.091, pruned_loss=0.01315, audio_tagging_loss=0.008795, over 3032987.73 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 19:25:49,040 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2963053.3333333335, ans=0.0 2023-11-24 19:25:50,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2963053.3333333335, ans=0.2 2023-11-24 19:26:04,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2963120.0, ans=0.125 2023-11-24 19:26:13,188 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2963186.6666666665, ans=0.1 2023-11-24 19:26:27,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.97 vs. limit=15.0 2023-11-24 19:26:33,528 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444500 2023-11-24 19:26:45,182 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11650, loss[loss=0.06622, simple_loss=0.09468, pruned_loss=0.01211, audio_tagging_loss=0.006767, over 15554.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09163, pruned_loss=0.01327, audio_tagging_loss=0.008796, over 3031541.07 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 19:27:05,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2963453.3333333335, ans=0.125 2023-11-24 19:27:16,439 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.51 vs. limit=15.0 2023-11-24 19:27:20,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2963520.0, ans=0.125 2023-11-24 19:27:28,350 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.973e+01 8.230e+01 8.998e+01 9.783e+01 1.223e+02, threshold=1.800e+02, percent-clipped=0.0 2023-11-24 19:27:29,203 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.11 vs. limit=15.0 2023-11-24 19:27:34,545 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444550 2023-11-24 19:27:46,805 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11700, loss[loss=0.05971, simple_loss=0.07909, pruned_loss=0.013, audio_tagging_loss=0.007162, over 15965.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09174, pruned_loss=0.01321, audio_tagging_loss=0.008811, over 3030167.08 frames. ], batch size: 63, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:28:13,515 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2963853.3333333335, ans=0.0 2023-11-24 19:28:36,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444600 2023-11-24 19:28:47,480 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.44 vs. limit=15.0 2023-11-24 19:28:48,937 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.43 vs. limit=12.0 2023-11-24 19:28:49,226 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11750, loss[loss=0.06092, simple_loss=0.08818, pruned_loss=0.009869, audio_tagging_loss=0.006959, over 14525.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09153, pruned_loss=0.01314, audio_tagging_loss=0.008839, over 3033071.65 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:28:53,375 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=3.91 vs. limit=6.0 2023-11-24 19:29:05,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2964120.0, ans=0.0 2023-11-24 19:29:07,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2964120.0, ans=0.0 2023-11-24 19:29:32,999 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.583e+01 9.295e+01 1.006e+02 1.151e+02, threshold=1.859e+02, percent-clipped=0.0 2023-11-24 19:29:35,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2964253.3333333335, ans=0.0 2023-11-24 19:29:38,252 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.48 vs. limit=15.0 2023-11-24 19:29:38,416 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.06 vs. limit=15.0 2023-11-24 19:29:38,958 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444650 2023-11-24 19:29:52,068 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11800, loss[loss=0.07408, simple_loss=0.1092, pruned_loss=0.0119, audio_tagging_loss=0.007592, over 15170.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09154, pruned_loss=0.01314, audio_tagging_loss=0.008842, over 3037320.38 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:30:01,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2964386.6666666665, ans=0.125 2023-11-24 19:30:05,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2964453.3333333335, ans=10.0 2023-11-24 19:30:08,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2964453.3333333335, ans=0.0 2023-11-24 19:30:29,059 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2964586.6666666665, ans=0.125 2023-11-24 19:30:38,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2964586.6666666665, ans=0.2 2023-11-24 19:30:41,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2964653.3333333335, ans=0.0 2023-11-24 19:30:42,152 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444700 2023-11-24 19:30:54,568 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11850, loss[loss=0.06409, simple_loss=0.08873, pruned_loss=0.009834, audio_tagging_loss=0.009891, over 15164.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09174, pruned_loss=0.01317, audio_tagging_loss=0.008927, over 3035068.46 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:31:04,287 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.18 vs. limit=15.0 2023-11-24 19:31:37,862 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.569e+01 9.195e+01 9.795e+01 1.501e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 19:31:40,013 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:31:44,497 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444750 2023-11-24 19:31:47,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.32 vs. limit=15.0 2023-11-24 19:31:48,657 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0 2023-11-24 19:31:56,653 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11900, loss[loss=0.07188, simple_loss=0.09922, pruned_loss=0.01418, audio_tagging_loss=0.008085, over 16244.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09174, pruned_loss=0.01316, audio_tagging_loss=0.009032, over 3039977.90 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:32:05,651 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2965053.3333333335, ans=0.1 2023-11-24 19:32:08,418 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.01 vs. limit=10.0 2023-11-24 19:32:20,529 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2965186.6666666665, ans=0.125 2023-11-24 19:32:21,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2965186.6666666665, ans=0.125 2023-11-24 19:32:38,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2965253.3333333335, ans=0.0 2023-11-24 19:32:46,397 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444800 2023-11-24 19:32:59,210 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 11950, loss[loss=0.06548, simple_loss=0.08407, pruned_loss=0.01236, audio_tagging_loss=0.01109, over 15906.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.0913, pruned_loss=0.01308, audio_tagging_loss=0.009125, over 3043414.10 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 16.0 2023-11-24 19:33:02,169 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.52 vs. limit=22.5 2023-11-24 19:33:11,883 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2965453.3333333335, ans=0.2 2023-11-24 19:33:14,100 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2965453.3333333335, ans=0.0 2023-11-24 19:33:14,910 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2965453.3333333335, ans=0.0 2023-11-24 19:33:19,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2965453.3333333335, ans=0.125 2023-11-24 19:33:29,787 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2965520.0, ans=0.0 2023-11-24 19:33:30,766 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2965520.0, ans=0.1 2023-11-24 19:33:36,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2965586.6666666665, ans=0.125 2023-11-24 19:33:40,354 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.04 vs. limit=6.0 2023-11-24 19:33:41,980 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.648e+01 8.644e+01 9.247e+01 9.996e+01 1.290e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-24 19:33:47,790 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444850 2023-11-24 19:33:59,357 INFO [train_asr.py:1221] (3/4) Epoch 37, batch 12000, loss[loss=0.06712, simple_loss=0.08974, pruned_loss=0.0129, audio_tagging_loss=0.009357, over 16116.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09195, pruned_loss=0.01331, audio_tagging_loss=0.009152, over 3039558.85 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 32.0 2023-11-24 19:33:59,358 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 19:34:41,831 INFO [train_asr.py:1253] (3/4) Epoch 37, validation: loss=0.058, simple_loss=0.05081, pruned_loss=0.005169, audio_tagging_loss=0.02743, over 4681554.00 frames. 2023-11-24 19:34:41,832 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 19:34:42,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2965720.0, ans=0.0 2023-11-24 19:34:46,485 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2965720.0, ans=0.125 2023-11-24 19:34:50,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2965720.0, ans=0.05 2023-11-24 19:34:56,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2965786.6666666665, ans=0.125 2023-11-24 19:34:57,848 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2965786.6666666665, ans=0.125 2023-11-24 19:35:40,410 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 0, loss[loss=0.0921, simple_loss=0.1157, pruned_loss=0.01716, audio_tagging_loss=0.01711, over 15543.00 frames. ], tot_loss[loss=0.0921, simple_loss=0.1157, pruned_loss=0.01716, audio_tagging_loss=0.01711, over 15543.00 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:35:40,411 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 19:36:03,765 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9622, 3.1398, 2.8262, 3.1020, 3.3648, 2.7743, 3.3744, 2.5729], device='cuda:3') 2023-11-24 19:36:08,056 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3399, 5.0064, 4.6145, 5.1569], device='cuda:3') 2023-11-24 19:36:09,419 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.1822, 3.9695, 3.7513, 3.2264], device='cuda:3') 2023-11-24 19:36:10,078 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8000, 4.9493, 5.0541, 4.9044], device='cuda:3') 2023-11-24 19:36:12,095 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3277, 5.0122, 4.6438, 5.1405], device='cuda:3') 2023-11-24 19:36:12,949 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8043, 4.9516, 5.0575, 4.9040], device='cuda:3') 2023-11-24 19:36:16,564 INFO [train_asr.py:1253] (3/4) Epoch 38, validation: loss=0.05758, simple_loss=0.05072, pruned_loss=0.005057, audio_tagging_loss=0.02716, over 4681554.00 frames. 2023-11-24 19:36:16,565 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 19:36:28,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2965940.0, ans=0.125 2023-11-24 19:36:35,660 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.37 vs. limit=15.0 2023-11-24 19:36:37,517 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444900 2023-11-24 19:37:16,998 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.93 vs. limit=15.0 2023-11-24 19:37:18,351 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.82 vs. limit=22.5 2023-11-24 19:37:18,859 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 50, loss[loss=0.05507, simple_loss=0.06702, pruned_loss=0.004134, audio_tagging_loss=0.01743, over 14707.00 frames. ], tot_loss[loss=0.07707, simple_loss=0.09332, pruned_loss=0.01347, audio_tagging_loss=0.01694, over 689179.28 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:37:19,150 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:37:20,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2966206.6666666665, ans=0.0 2023-11-24 19:37:20,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2966206.6666666665, ans=0.2 2023-11-24 19:37:35,484 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 8.245e+01 9.441e+01 1.025e+02 1.118e+02 1.388e+02, threshold=2.050e+02, percent-clipped=0.0 2023-11-24 19:37:40,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 444950 2023-11-24 19:37:52,294 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2966340.0, ans=0.125 2023-11-24 19:38:16,144 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2966473.3333333335, ans=0.2 2023-11-24 19:38:17,176 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2966473.3333333335, ans=0.0 2023-11-24 19:38:21,598 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 100, loss[loss=0.06623, simple_loss=0.09232, pruned_loss=0.008657, audio_tagging_loss=0.01142, over 14032.00 frames. ], tot_loss[loss=0.07556, simple_loss=0.09279, pruned_loss=0.01321, audio_tagging_loss=0.01595, over 1205861.70 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:38:37,686 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.33 vs. limit=15.0 2023-11-24 19:38:43,055 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445000 2023-11-24 19:38:47,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2966673.3333333335, ans=0.1 2023-11-24 19:38:47,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2966673.3333333335, ans=0.125 2023-11-24 19:38:50,737 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2966673.3333333335, ans=0.0 2023-11-24 19:39:11,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2966806.6666666665, ans=0.0 2023-11-24 19:39:13,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2966806.6666666665, ans=0.0 2023-11-24 19:39:24,352 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 150, loss[loss=0.06205, simple_loss=0.07623, pruned_loss=0.01073, audio_tagging_loss=0.0132, over 15613.00 frames. ], tot_loss[loss=0.07296, simple_loss=0.09109, pruned_loss=0.01279, audio_tagging_loss=0.01462, over 1610165.82 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:39:30,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.11 vs. limit=22.5 2023-11-24 19:39:32,959 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2966873.3333333335, ans=0.0 2023-11-24 19:39:37,677 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2966940.0, ans=0.0 2023-11-24 19:39:39,715 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 9.070e+01 9.567e+01 1.041e+02 1.259e+02, threshold=1.913e+02, percent-clipped=0.0 2023-11-24 19:39:43,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.11 vs. limit=15.0 2023-11-24 19:39:44,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445050 2023-11-24 19:40:14,804 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.80 vs. limit=15.0 2023-11-24 19:40:26,151 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 200, loss[loss=0.06899, simple_loss=0.08548, pruned_loss=0.01266, audio_tagging_loss=0.01359, over 14934.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09128, pruned_loss=0.01285, audio_tagging_loss=0.01299, over 1931888.63 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:40:45,711 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2967273.3333333335, ans=0.09899494936611666 2023-11-24 19:40:48,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445100 2023-11-24 19:40:49,745 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2967273.3333333335, ans=0.125 2023-11-24 19:40:51,025 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2967340.0, ans=0.0 2023-11-24 19:40:57,474 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.29 vs. limit=15.0 2023-11-24 19:41:28,686 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 250, loss[loss=0.05547, simple_loss=0.07412, pruned_loss=0.01036, audio_tagging_loss=0.008047, over 15323.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09054, pruned_loss=0.01294, audio_tagging_loss=0.01175, over 2177994.17 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:41:34,258 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=2967540.0, ans=0.02 2023-11-24 19:41:45,192 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.968e+01 8.613e+01 9.234e+01 1.003e+02 1.300e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 19:41:45,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2967606.6666666665, ans=0.1 2023-11-24 19:41:47,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2967606.6666666665, ans=0.2 2023-11-24 19:41:50,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445150 2023-11-24 19:41:57,162 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.93 vs. limit=15.0 2023-11-24 19:42:08,144 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.81 vs. limit=8.0 2023-11-24 19:42:15,342 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2967740.0, ans=0.025 2023-11-24 19:42:21,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2967806.6666666665, ans=0.04949747468305833 2023-11-24 19:42:31,603 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 300, loss[loss=0.06744, simple_loss=0.09697, pruned_loss=0.009718, audio_tagging_loss=0.009237, over 16007.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09019, pruned_loss=0.0129, audio_tagging_loss=0.01096, over 2374125.62 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:42:36,042 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2967873.3333333335, ans=0.0 2023-11-24 19:42:52,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445200 2023-11-24 19:43:06,830 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2968006.6666666665, ans=0.0 2023-11-24 19:43:08,067 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:43:23,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.22 vs. limit=22.5 2023-11-24 19:43:34,422 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 350, loss[loss=0.07676, simple_loss=0.1057, pruned_loss=0.01635, audio_tagging_loss=0.007568, over 16101.00 frames. ], tot_loss[loss=0.069, simple_loss=0.0914, pruned_loss=0.01306, audio_tagging_loss=0.01024, over 2526552.58 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:43:49,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2968273.3333333335, ans=0.125 2023-11-24 19:43:51,715 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.795e+01 8.774e+01 9.424e+01 1.013e+02 1.303e+02, threshold=1.885e+02, percent-clipped=0.0 2023-11-24 19:43:55,381 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445250 2023-11-24 19:43:58,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2968340.0, ans=0.05 2023-11-24 19:44:07,702 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.08 vs. limit=15.0 2023-11-24 19:44:35,929 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2968540.0, ans=0.0 2023-11-24 19:44:36,729 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 400, loss[loss=0.05386, simple_loss=0.07078, pruned_loss=0.008297, audio_tagging_loss=0.01018, over 16222.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09074, pruned_loss=0.01294, audio_tagging_loss=0.009885, over 2644659.33 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:44:58,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445300 2023-11-24 19:45:02,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2968673.3333333335, ans=0.1 2023-11-24 19:45:05,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2968673.3333333335, ans=0.2 2023-11-24 19:45:05,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2968673.3333333335, ans=0.125 2023-11-24 19:45:10,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2968673.3333333335, ans=0.0 2023-11-24 19:45:16,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2968740.0, ans=0.125 2023-11-24 19:45:36,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=2968806.6666666665, ans=0.02 2023-11-24 19:45:38,902 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2968873.3333333335, ans=0.125 2023-11-24 19:45:39,747 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 450, loss[loss=0.0654, simple_loss=0.09571, pruned_loss=0.01117, audio_tagging_loss=0.006373, over 15911.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09123, pruned_loss=0.01288, audio_tagging_loss=0.009544, over 2739615.25 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:45:49,094 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.73 vs. limit=15.0 2023-11-24 19:45:55,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2968940.0, ans=0.125 2023-11-24 19:45:56,758 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.512e+01 9.246e+01 9.935e+01 1.248e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-24 19:46:00,394 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445350 2023-11-24 19:46:12,086 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.43 vs. limit=22.5 2023-11-24 19:46:41,817 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 500, loss[loss=0.05783, simple_loss=0.0714, pruned_loss=0.01315, audio_tagging_loss=0.008987, over 14504.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09121, pruned_loss=0.013, audio_tagging_loss=0.009462, over 2809202.51 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:46:48,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2969206.6666666665, ans=0.1 2023-11-24 19:46:54,421 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2969273.3333333335, ans=0.95 2023-11-24 19:46:57,731 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=15.0 2023-11-24 19:47:03,046 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445400 2023-11-24 19:47:10,159 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2969340.0, ans=0.2 2023-11-24 19:47:39,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2969473.3333333335, ans=0.125 2023-11-24 19:47:42,267 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2969473.3333333335, ans=0.125 2023-11-24 19:47:44,342 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 550, loss[loss=0.04988, simple_loss=0.06222, pruned_loss=0.00974, audio_tagging_loss=0.009033, over 15710.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09013, pruned_loss=0.0127, audio_tagging_loss=0.009256, over 2860307.58 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:47:46,508 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2969540.0, ans=0.2 2023-11-24 19:48:02,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2969606.6666666665, ans=0.125 2023-11-24 19:48:04,006 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 8.415e+01 9.096e+01 9.896e+01 1.420e+02, threshold=1.819e+02, percent-clipped=0.0 2023-11-24 19:48:06,504 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445450 2023-11-24 19:48:12,083 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2969673.3333333335, ans=0.125 2023-11-24 19:48:20,821 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.85 vs. limit=15.0 2023-11-24 19:48:38,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2969806.6666666665, ans=0.125 2023-11-24 19:48:47,451 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 600, loss[loss=0.07583, simple_loss=0.1043, pruned_loss=0.01462, audio_tagging_loss=0.009078, over 15443.00 frames. ], tot_loss[loss=0.06641, simple_loss=0.08909, pruned_loss=0.01257, audio_tagging_loss=0.009287, over 2897080.56 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:49:05,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2969940.0, ans=0.1 2023-11-24 19:49:08,918 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445500 2023-11-24 19:49:26,402 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2970073.3333333335, ans=0.125 2023-11-24 19:49:45,474 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2970140.0, ans=0.125 2023-11-24 19:49:49,972 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 650, loss[loss=0.07408, simple_loss=0.1052, pruned_loss=0.01012, audio_tagging_loss=0.01136, over 15140.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.0895, pruned_loss=0.01266, audio_tagging_loss=0.009255, over 2936812.13 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:50:01,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2970273.3333333335, ans=0.0 2023-11-24 19:50:04,807 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 19:50:07,963 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.559e+01 9.252e+01 1.019e+02 1.352e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 19:50:10,982 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445550 2023-11-24 19:50:11,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2970273.3333333335, ans=0.125 2023-11-24 19:50:14,015 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.50 vs. limit=22.5 2023-11-24 19:50:42,978 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2970473.3333333335, ans=0.05 2023-11-24 19:50:51,219 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 700, loss[loss=0.04441, simple_loss=0.05196, pruned_loss=0.00835, audio_tagging_loss=0.01008, over 14925.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09086, pruned_loss=0.01291, audio_tagging_loss=0.009086, over 2964275.73 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:51:13,048 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445600 2023-11-24 19:51:19,255 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.68 vs. limit=15.0 2023-11-24 19:51:54,371 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2970873.3333333335, ans=0.125 2023-11-24 19:51:55,299 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 750, loss[loss=0.06056, simple_loss=0.07628, pruned_loss=0.01426, audio_tagging_loss=0.008168, over 13804.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09055, pruned_loss=0.01269, audio_tagging_loss=0.009039, over 2990805.17 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:51:59,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2970873.3333333335, ans=0.125 2023-11-24 19:52:01,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2970873.3333333335, ans=0.1 2023-11-24 19:52:10,593 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0 2023-11-24 19:52:14,361 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.538e+01 8.682e+01 9.156e+01 1.004e+02 1.177e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-24 19:52:16,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445650 2023-11-24 19:52:18,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2970940.0, ans=0.0 2023-11-24 19:52:19,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.28 vs. limit=15.0 2023-11-24 19:52:23,420 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.74 vs. limit=10.0 2023-11-24 19:52:36,175 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.62 vs. limit=15.0 2023-11-24 19:52:42,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2971073.3333333335, ans=0.5 2023-11-24 19:52:49,945 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.92 vs. limit=6.0 2023-11-24 19:52:51,031 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.14 vs. limit=6.0 2023-11-24 19:52:58,208 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 800, loss[loss=0.05664, simple_loss=0.07885, pruned_loss=0.006829, audio_tagging_loss=0.01039, over 15347.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09059, pruned_loss=0.01272, audio_tagging_loss=0.009082, over 2998740.78 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:53:07,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2971206.6666666665, ans=0.125 2023-11-24 19:53:19,227 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445700 2023-11-24 19:53:33,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2971340.0, ans=0.125 2023-11-24 19:53:33,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2971340.0, ans=0.0 2023-11-24 19:53:43,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2971406.6666666665, ans=0.0 2023-11-24 19:53:49,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2971473.3333333335, ans=0.0 2023-11-24 19:54:00,943 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 850, loss[loss=0.0621, simple_loss=0.08288, pruned_loss=0.009124, audio_tagging_loss=0.01154, over 15659.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09095, pruned_loss=0.01283, audio_tagging_loss=0.009075, over 3010193.78 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 19:54:19,782 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.517e+01 9.234e+01 9.613e+01 1.155e+02, threshold=1.847e+02, percent-clipped=0.0 2023-11-24 19:54:22,325 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445750 2023-11-24 19:54:28,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2971673.3333333335, ans=0.125 2023-11-24 19:54:30,539 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.18 vs. limit=15.0 2023-11-24 19:54:38,240 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2971740.0, ans=0.1 2023-11-24 19:54:44,538 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.62 vs. limit=6.0 2023-11-24 19:54:50,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2971806.6666666665, ans=0.0 2023-11-24 19:55:00,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=13.38 vs. limit=15.0 2023-11-24 19:55:03,976 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 900, loss[loss=0.08542, simple_loss=0.1286, pruned_loss=0.01562, audio_tagging_loss=0.00549, over 15899.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09138, pruned_loss=0.0129, audio_tagging_loss=0.009115, over 3017238.93 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:55:04,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2971873.3333333335, ans=0.1 2023-11-24 19:55:12,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2971873.3333333335, ans=0.0 2023-11-24 19:55:21,970 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2971940.0, ans=0.125 2023-11-24 19:55:22,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2971940.0, ans=0.125 2023-11-24 19:55:25,474 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445800 2023-11-24 19:55:25,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2971940.0, ans=0.125 2023-11-24 19:55:43,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2972073.3333333335, ans=0.1 2023-11-24 19:56:03,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2972140.0, ans=0.015 2023-11-24 19:56:07,362 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 950, loss[loss=0.08301, simple_loss=0.1077, pruned_loss=0.01771, audio_tagging_loss=0.01144, over 15173.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09191, pruned_loss=0.01301, audio_tagging_loss=0.009021, over 3025492.34 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:56:21,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2972273.3333333335, ans=0.125 2023-11-24 19:56:26,842 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.519e+01 9.243e+01 9.717e+01 1.374e+02, threshold=1.849e+02, percent-clipped=0.0 2023-11-24 19:56:28,173 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445850 2023-11-24 19:57:09,560 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1000, loss[loss=0.06591, simple_loss=0.08188, pruned_loss=0.01324, audio_tagging_loss=0.01173, over 14767.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09212, pruned_loss=0.01311, audio_tagging_loss=0.008845, over 3026713.49 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:57:14,907 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.88 vs. limit=6.0 2023-11-24 19:57:22,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2972606.6666666665, ans=0.1 2023-11-24 19:57:30,783 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445900 2023-11-24 19:57:35,511 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 19:57:49,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2972740.0, ans=0.125 2023-11-24 19:57:58,328 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2972806.6666666665, ans=0.2 2023-11-24 19:58:11,861 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1050, loss[loss=0.07537, simple_loss=0.11, pruned_loss=0.01523, audio_tagging_loss=0.005153, over 15605.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09128, pruned_loss=0.01295, audio_tagging_loss=0.008791, over 3031085.83 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:58:16,590 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.91 vs. limit=22.5 2023-11-24 19:58:28,716 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2972940.0, ans=0.0 2023-11-24 19:58:32,041 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.733e+01 9.359e+01 1.012e+02 1.444e+02, threshold=1.872e+02, percent-clipped=0.0 2023-11-24 19:58:33,417 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 445950 2023-11-24 19:58:39,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2973006.6666666665, ans=0.125 2023-11-24 19:58:52,821 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.04 vs. limit=15.0 2023-11-24 19:59:14,426 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1100, loss[loss=0.07913, simple_loss=0.105, pruned_loss=0.01737, audio_tagging_loss=0.009276, over 15934.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09066, pruned_loss=0.01298, audio_tagging_loss=0.008709, over 3033384.42 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 19:59:17,468 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 19:59:19,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2973206.6666666665, ans=10.0 2023-11-24 19:59:21,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2973206.6666666665, ans=0.125 2023-11-24 19:59:33,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2973273.3333333335, ans=0.1 2023-11-24 19:59:35,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446000 2023-11-24 19:59:36,505 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=15.0 2023-11-24 19:59:41,151 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2973340.0, ans=0.125 2023-11-24 19:59:44,339 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.38 vs. limit=6.0 2023-11-24 20:00:16,981 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1150, loss[loss=0.05919, simple_loss=0.08392, pruned_loss=0.01093, audio_tagging_loss=0.006307, over 15708.00 frames. ], tot_loss[loss=0.06618, simple_loss=0.08936, pruned_loss=0.01275, audio_tagging_loss=0.008754, over 3041640.29 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:00:37,598 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.661e+01 8.560e+01 9.055e+01 9.595e+01 1.732e+02, threshold=1.811e+02, percent-clipped=0.0 2023-11-24 20:00:38,894 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446050 2023-11-24 20:00:43,788 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2973673.3333333335, ans=0.125 2023-11-24 20:01:18,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2973873.3333333335, ans=0.125 2023-11-24 20:01:19,755 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1200, loss[loss=0.06784, simple_loss=0.09265, pruned_loss=0.01106, audio_tagging_loss=0.01045, over 15756.00 frames. ], tot_loss[loss=0.06619, simple_loss=0.08939, pruned_loss=0.01274, audio_tagging_loss=0.008758, over 3040604.37 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:01:33,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2973940.0, ans=0.2 2023-11-24 20:01:36,384 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.65 vs. limit=10.0 2023-11-24 20:01:37,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2973940.0, ans=0.125 2023-11-24 20:01:41,055 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446100 2023-11-24 20:01:44,931 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2974006.6666666665, ans=0.2 2023-11-24 20:01:47,329 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 20:01:48,806 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.08 vs. limit=6.0 2023-11-24 20:01:52,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2974006.6666666665, ans=0.125 2023-11-24 20:01:53,398 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2974006.6666666665, ans=0.125 2023-11-24 20:02:06,621 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.97 vs. limit=15.0 2023-11-24 20:02:07,648 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2974073.3333333335, ans=0.0 2023-11-24 20:02:09,893 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2974140.0, ans=0.1 2023-11-24 20:02:22,289 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1250, loss[loss=0.07421, simple_loss=0.1075, pruned_loss=0.01271, audio_tagging_loss=0.007735, over 16399.00 frames. ], tot_loss[loss=0.06595, simple_loss=0.08921, pruned_loss=0.01261, audio_tagging_loss=0.008733, over 3037831.55 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:02:42,693 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.510e+01 9.127e+01 9.845e+01 1.206e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-24 20:02:42,858 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446150 2023-11-24 20:02:48,350 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0 2023-11-24 20:03:24,243 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1300, loss[loss=0.05521, simple_loss=0.06793, pruned_loss=0.009484, audio_tagging_loss=0.01176, over 15945.00 frames. ], tot_loss[loss=0.06605, simple_loss=0.08953, pruned_loss=0.01264, audio_tagging_loss=0.008642, over 3044960.53 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:03:25,667 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2974540.0, ans=0.2 2023-11-24 20:03:35,104 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2974606.6666666665, ans=0.0 2023-11-24 20:03:46,025 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446200 2023-11-24 20:03:49,328 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0 2023-11-24 20:03:55,938 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2974673.3333333335, ans=10.0 2023-11-24 20:04:02,495 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2974740.0, ans=0.1 2023-11-24 20:04:03,582 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2974740.0, ans=0.09899494936611666 2023-11-24 20:04:08,722 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.99 vs. limit=12.0 2023-11-24 20:04:26,374 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1350, loss[loss=0.06003, simple_loss=0.07346, pruned_loss=0.01195, audio_tagging_loss=0.01135, over 15315.00 frames. ], tot_loss[loss=0.06644, simple_loss=0.08981, pruned_loss=0.01279, audio_tagging_loss=0.008751, over 3050516.84 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:04:33,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2974873.3333333335, ans=0.125 2023-11-24 20:04:46,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2974940.0, ans=0.2 2023-11-24 20:04:48,067 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.451e+01 8.554e+01 9.036e+01 9.639e+01 2.297e+02, threshold=1.807e+02, percent-clipped=1.0 2023-11-24 20:04:48,209 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446250 2023-11-24 20:04:57,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2975006.6666666665, ans=0.0 2023-11-24 20:05:10,678 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 20:05:16,433 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.29 vs. limit=15.0 2023-11-24 20:05:22,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2975140.0, ans=0.125 2023-11-24 20:05:24,818 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.30 vs. limit=10.0 2023-11-24 20:05:29,116 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1400, loss[loss=0.05256, simple_loss=0.06704, pruned_loss=0.00956, audio_tagging_loss=0.009482, over 14924.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.09047, pruned_loss=0.01281, audio_tagging_loss=0.008789, over 3052784.85 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:05:29,372 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2975206.6666666665, ans=0.125 2023-11-24 20:05:38,576 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2975206.6666666665, ans=0.1 2023-11-24 20:05:49,961 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446300 2023-11-24 20:05:58,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2975340.0, ans=15.0 2023-11-24 20:06:05,501 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2975406.6666666665, ans=0.125 2023-11-24 20:06:31,102 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1450, loss[loss=0.05588, simple_loss=0.07298, pruned_loss=0.007964, audio_tagging_loss=0.01143, over 15702.00 frames. ], tot_loss[loss=0.06642, simple_loss=0.08966, pruned_loss=0.01274, audio_tagging_loss=0.008852, over 3056122.80 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:06:37,296 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2975540.0, ans=0.1 2023-11-24 20:06:51,821 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.634e+01 9.284e+01 1.040e+02 1.664e+02, threshold=1.857e+02, percent-clipped=0.0 2023-11-24 20:06:51,979 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446350 2023-11-24 20:06:55,697 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2975673.3333333335, ans=0.0 2023-11-24 20:06:56,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2975673.3333333335, ans=0.0 2023-11-24 20:07:00,673 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.00 vs. limit=22.5 2023-11-24 20:07:03,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2975673.3333333335, ans=0.0 2023-11-24 20:07:06,812 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.53 vs. limit=15.0 2023-11-24 20:07:08,760 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2975740.0, ans=0.125 2023-11-24 20:07:33,035 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1500, loss[loss=0.09751, simple_loss=0.1353, pruned_loss=0.01917, audio_tagging_loss=0.01067, over 14593.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09109, pruned_loss=0.01297, audio_tagging_loss=0.008817, over 3057295.03 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:07:37,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2975873.3333333335, ans=0.0 2023-11-24 20:07:39,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.21 vs. limit=15.0 2023-11-24 20:07:54,862 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446400 2023-11-24 20:08:07,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2976006.6666666665, ans=0.125 2023-11-24 20:08:07,498 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2976006.6666666665, ans=0.1 2023-11-24 20:08:13,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2976073.3333333335, ans=0.125 2023-11-24 20:08:18,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2976073.3333333335, ans=0.125 2023-11-24 20:08:19,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2976073.3333333335, ans=0.125 2023-11-24 20:08:19,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2976073.3333333335, ans=0.0 2023-11-24 20:08:23,700 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2976140.0, ans=0.2 2023-11-24 20:08:31,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2976140.0, ans=0.2 2023-11-24 20:08:36,382 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1550, loss[loss=0.05909, simple_loss=0.07435, pruned_loss=0.01134, audio_tagging_loss=0.01058, over 15387.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09075, pruned_loss=0.01299, audio_tagging_loss=0.008902, over 3052835.52 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:08:57,012 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446450 2023-11-24 20:08:58,029 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.670e+01 9.483e+01 1.014e+02 1.933e+02, threshold=1.897e+02, percent-clipped=2.0 2023-11-24 20:09:37,980 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1600, loss[loss=0.05937, simple_loss=0.08428, pruned_loss=0.008529, audio_tagging_loss=0.008697, over 14239.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.091, pruned_loss=0.01304, audio_tagging_loss=0.008942, over 3047499.44 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:09:52,269 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2976606.6666666665, ans=0.125 2023-11-24 20:09:58,773 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446500 2023-11-24 20:10:02,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2976673.3333333335, ans=0.0 2023-11-24 20:10:14,506 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.55 vs. limit=15.0 2023-11-24 20:10:21,834 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2976740.0, ans=0.0 2023-11-24 20:10:27,968 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2976806.6666666665, ans=0.0 2023-11-24 20:10:28,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2976806.6666666665, ans=0.0 2023-11-24 20:10:30,215 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2976806.6666666665, ans=0.125 2023-11-24 20:10:39,344 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1650, loss[loss=0.05447, simple_loss=0.0704, pruned_loss=0.008756, audio_tagging_loss=0.01052, over 14373.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.0911, pruned_loss=0.01292, audio_tagging_loss=0.008998, over 3043933.38 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:10:42,107 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2976873.3333333335, ans=0.125 2023-11-24 20:11:01,279 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446550 2023-11-24 20:11:02,255 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.314e+01 8.733e+01 9.209e+01 9.885e+01 1.194e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-24 20:11:05,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2977006.6666666665, ans=0.125 2023-11-24 20:11:22,088 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2977073.3333333335, ans=0.125 2023-11-24 20:11:39,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2977140.0, ans=0.125 2023-11-24 20:11:41,955 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1700, loss[loss=0.06269, simple_loss=0.08339, pruned_loss=0.009417, audio_tagging_loss=0.01158, over 15941.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09099, pruned_loss=0.0129, audio_tagging_loss=0.00905, over 3042503.31 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:11:50,338 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.72 vs. limit=15.0 2023-11-24 20:11:56,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2977273.3333333335, ans=0.0 2023-11-24 20:12:02,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2977273.3333333335, ans=0.0 2023-11-24 20:12:03,401 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446600 2023-11-24 20:12:04,726 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 20:12:45,066 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1750, loss[loss=0.06389, simple_loss=0.08158, pruned_loss=0.01687, audio_tagging_loss=0.006229, over 15953.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.0896, pruned_loss=0.01286, audio_tagging_loss=0.009058, over 3039392.49 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:12:56,082 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2977606.6666666665, ans=0.2 2023-11-24 20:12:59,522 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2977606.6666666665, ans=0.2 2023-11-24 20:13:05,918 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446650 2023-11-24 20:13:07,379 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2977606.6666666665, ans=0.125 2023-11-24 20:13:08,151 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.455e+01 9.155e+01 9.735e+01 1.313e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-24 20:13:13,779 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2977673.3333333335, ans=0.0 2023-11-24 20:13:30,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2977740.0, ans=0.125 2023-11-24 20:13:31,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2977740.0, ans=0.05 2023-11-24 20:13:42,169 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2977806.6666666665, ans=0.2 2023-11-24 20:13:46,506 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1800, loss[loss=0.05133, simple_loss=0.07021, pruned_loss=0.007942, audio_tagging_loss=0.008282, over 15234.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.08985, pruned_loss=0.01287, audio_tagging_loss=0.008994, over 3044279.77 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:13:49,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2977873.3333333335, ans=0.125 2023-11-24 20:14:01,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2977940.0, ans=0.0 2023-11-24 20:14:05,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2977940.0, ans=0.125 2023-11-24 20:14:07,957 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446700 2023-11-24 20:14:20,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=15.0 2023-11-24 20:14:35,698 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.77 vs. limit=10.0 2023-11-24 20:14:39,157 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=15.0 2023-11-24 20:14:42,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2978140.0, ans=0.125 2023-11-24 20:14:49,178 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1850, loss[loss=0.075, simple_loss=0.09872, pruned_loss=0.01617, audio_tagging_loss=0.009461, over 14584.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09036, pruned_loss=0.01304, audio_tagging_loss=0.008864, over 3044245.62 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:14:55,364 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2978206.6666666665, ans=0.125 2023-11-24 20:14:57,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.whiten.whitening_limit, batch_count=2978206.6666666665, ans=15.0 2023-11-24 20:15:05,835 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2978273.3333333335, ans=0.0 2023-11-24 20:15:10,452 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446750 2023-11-24 20:15:12,637 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.609e+01 9.248e+01 1.012e+02 1.189e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 20:15:26,128 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2978406.6666666665, ans=0.125 2023-11-24 20:15:34,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2978406.6666666665, ans=0.125 2023-11-24 20:15:36,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2978406.6666666665, ans=0.2 2023-11-24 20:15:51,162 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1900, loss[loss=0.06843, simple_loss=0.09097, pruned_loss=0.01292, audio_tagging_loss=0.01003, over 14441.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09071, pruned_loss=0.01298, audio_tagging_loss=0.008848, over 3036415.24 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:15:52,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2978540.0, ans=0.0 2023-11-24 20:15:58,798 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.46 vs. limit=22.5 2023-11-24 20:16:11,326 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446800 2023-11-24 20:16:12,957 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.51 vs. limit=15.0 2023-11-24 20:16:43,905 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.70 vs. limit=15.0 2023-11-24 20:16:49,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2978806.6666666665, ans=0.125 2023-11-24 20:16:50,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2978806.6666666665, ans=0.07 2023-11-24 20:16:52,748 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 1950, loss[loss=0.06983, simple_loss=0.1048, pruned_loss=0.01163, audio_tagging_loss=0.005805, over 15054.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09002, pruned_loss=0.01293, audio_tagging_loss=0.00887, over 3031025.62 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 4.0 2023-11-24 20:16:52,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2978873.3333333335, ans=0.1 2023-11-24 20:17:00,093 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2978873.3333333335, ans=0.125 2023-11-24 20:17:01,133 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2978873.3333333335, ans=0.1 2023-11-24 20:17:13,905 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446850 2023-11-24 20:17:17,337 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.207e+01 8.609e+01 9.375e+01 1.004e+02 1.642e+02, threshold=1.875e+02, percent-clipped=0.0 2023-11-24 20:17:33,840 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.78 vs. limit=15.0 2023-11-24 20:17:39,033 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2979073.3333333335, ans=0.125 2023-11-24 20:17:55,540 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2000, loss[loss=0.07445, simple_loss=0.1004, pruned_loss=0.01488, audio_tagging_loss=0.009347, over 15686.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09009, pruned_loss=0.01305, audio_tagging_loss=0.008883, over 3031054.96 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:18:04,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2979206.6666666665, ans=0.1 2023-11-24 20:18:06,784 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2979273.3333333335, ans=0.2 2023-11-24 20:18:17,327 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446900 2023-11-24 20:18:31,818 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2979406.6666666665, ans=0.1 2023-11-24 20:18:40,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.47 vs. limit=8.0 2023-11-24 20:18:41,225 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.43 vs. limit=15.0 2023-11-24 20:18:57,602 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2050, loss[loss=0.06393, simple_loss=0.08898, pruned_loss=0.01005, audio_tagging_loss=0.009387, over 14317.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09016, pruned_loss=0.01298, audio_tagging_loss=0.00888, over 3025758.27 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:19:19,234 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 446950 2023-11-24 20:19:22,627 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.128e+01 8.521e+01 9.047e+01 9.602e+01 1.413e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-24 20:19:33,698 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2979673.3333333335, ans=0.0 2023-11-24 20:19:34,783 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2979740.0, ans=0.1 2023-11-24 20:19:53,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2979806.6666666665, ans=0.125 2023-11-24 20:20:00,616 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2100, loss[loss=0.06151, simple_loss=0.08648, pruned_loss=0.01085, audio_tagging_loss=0.00742, over 16680.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09024, pruned_loss=0.01304, audio_tagging_loss=0.008824, over 3029060.93 frames. ], batch size: 62, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:20:15,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2979940.0, ans=0.2 2023-11-24 20:20:22,408 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447000 2023-11-24 20:20:22,699 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2979940.0, ans=0.1 2023-11-24 20:21:03,520 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2150, loss[loss=0.08667, simple_loss=0.1161, pruned_loss=0.02405, audio_tagging_loss=0.004582, over 14590.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.08995, pruned_loss=0.01301, audio_tagging_loss=0.00881, over 3028114.07 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:21:24,848 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447050 2023-11-24 20:21:27,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2980340.0, ans=0.04949747468305833 2023-11-24 20:21:28,342 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.688e+01 9.386e+01 1.033e+02 1.815e+02, threshold=1.877e+02, percent-clipped=1.0 2023-11-24 20:21:36,971 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.83 vs. limit=22.5 2023-11-24 20:21:39,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2980406.6666666665, ans=0.0 2023-11-24 20:21:40,956 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 20:21:55,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2980473.3333333335, ans=0.1 2023-11-24 20:22:03,592 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2980473.3333333335, ans=10.0 2023-11-24 20:22:05,667 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2200, loss[loss=0.07143, simple_loss=0.09929, pruned_loss=0.01223, audio_tagging_loss=0.009555, over 16426.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09052, pruned_loss=0.01299, audio_tagging_loss=0.008812, over 3029867.91 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:22:26,955 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447100 2023-11-24 20:22:30,912 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.39 vs. limit=12.0 2023-11-24 20:22:42,690 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2980740.0, ans=0.125 2023-11-24 20:22:58,611 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2980806.6666666665, ans=0.2 2023-11-24 20:23:07,825 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2250, loss[loss=0.05351, simple_loss=0.06511, pruned_loss=0.008545, audio_tagging_loss=0.01241, over 14406.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09025, pruned_loss=0.01294, audio_tagging_loss=0.008827, over 3032688.36 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:23:18,868 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.94 vs. limit=10.0 2023-11-24 20:23:27,652 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2980940.0, ans=0.1 2023-11-24 20:23:29,841 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447150 2023-11-24 20:23:31,158 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2980940.0, ans=0.125 2023-11-24 20:23:33,355 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.657e+01 9.252e+01 1.018e+02 2.312e+02, threshold=1.850e+02, percent-clipped=2.0 2023-11-24 20:23:34,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2981006.6666666665, ans=0.0 2023-11-24 20:23:48,966 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.33 vs. limit=6.0 2023-11-24 20:23:59,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2981140.0, ans=0.125 2023-11-24 20:23:59,417 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2981140.0, ans=0.125 2023-11-24 20:24:06,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.69 vs. limit=15.0 2023-11-24 20:24:11,235 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2300, loss[loss=0.06199, simple_loss=0.08639, pruned_loss=0.01166, audio_tagging_loss=0.007137, over 14676.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09019, pruned_loss=0.01285, audio_tagging_loss=0.008965, over 3037860.12 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:24:32,685 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447200 2023-11-24 20:24:37,017 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.44 vs. limit=15.0 2023-11-24 20:24:45,019 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2981340.0, ans=0.125 2023-11-24 20:24:54,034 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2981406.6666666665, ans=0.125 2023-11-24 20:24:55,502 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.64 vs. limit=15.0 2023-11-24 20:24:56,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2981406.6666666665, ans=0.1 2023-11-24 20:25:06,739 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 20:25:14,006 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2350, loss[loss=0.07917, simple_loss=0.1106, pruned_loss=0.01493, audio_tagging_loss=0.00894, over 15546.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09062, pruned_loss=0.01309, audio_tagging_loss=0.00907, over 3039026.19 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 8.0 2023-11-24 20:25:19,047 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2981540.0, ans=0.0 2023-11-24 20:25:30,678 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2981606.6666666665, ans=0.0 2023-11-24 20:25:33,192 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.64 vs. limit=10.0 2023-11-24 20:25:34,991 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447250 2023-11-24 20:25:39,018 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.625e+01 9.128e+01 9.764e+01 1.329e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 20:25:49,350 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2981673.3333333335, ans=0.125 2023-11-24 20:26:01,775 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2981740.0, ans=0.0 2023-11-24 20:26:16,229 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2400, loss[loss=0.07249, simple_loss=0.09616, pruned_loss=0.01625, audio_tagging_loss=0.008159, over 15839.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09083, pruned_loss=0.01315, audio_tagging_loss=0.009034, over 3035726.46 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:26:24,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2981873.3333333335, ans=0.125 2023-11-24 20:26:38,249 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447300 2023-11-24 20:26:39,541 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2981940.0, ans=0.0 2023-11-24 20:26:39,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2981940.0, ans=0.2 2023-11-24 20:26:47,890 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 20:26:56,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2982073.3333333335, ans=0.035 2023-11-24 20:27:18,568 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2450, loss[loss=0.04608, simple_loss=0.05222, pruned_loss=0.006339, audio_tagging_loss=0.01363, over 16880.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09045, pruned_loss=0.0131, audio_tagging_loss=0.009111, over 3030007.88 frames. ], batch size: 68, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:27:34,409 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2982273.3333333335, ans=0.0 2023-11-24 20:27:35,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.85 vs. limit=10.0 2023-11-24 20:27:40,054 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447350 2023-11-24 20:27:44,107 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.597e+01 9.419e+01 1.015e+02 1.292e+02, threshold=1.884e+02, percent-clipped=0.0 2023-11-24 20:27:58,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2982406.6666666665, ans=0.125 2023-11-24 20:28:03,450 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2982406.6666666665, ans=0.125 2023-11-24 20:28:21,321 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2500, loss[loss=0.05196, simple_loss=0.06256, pruned_loss=0.008359, audio_tagging_loss=0.01232, over 14750.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.08981, pruned_loss=0.01285, audio_tagging_loss=0.009205, over 3035906.32 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:28:21,507 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2982540.0, ans=0.2 2023-11-24 20:28:22,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2982540.0, ans=0.1 2023-11-24 20:28:22,761 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2982540.0, ans=0.125 2023-11-24 20:28:24,578 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.71 vs. limit=10.0 2023-11-24 20:28:40,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.52 vs. limit=22.5 2023-11-24 20:28:42,246 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447400 2023-11-24 20:28:45,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2982673.3333333335, ans=0.09899494936611666 2023-11-24 20:28:47,594 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2982673.3333333335, ans=0.125 2023-11-24 20:29:14,220 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2982806.6666666665, ans=0.125 2023-11-24 20:29:23,918 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2550, loss[loss=0.09778, simple_loss=0.1367, pruned_loss=0.02293, audio_tagging_loss=0.00651, over 15881.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.08988, pruned_loss=0.01286, audio_tagging_loss=0.009135, over 3035082.01 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:29:44,536 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447450 2023-11-24 20:29:48,492 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.648e+01 8.553e+01 9.176e+01 1.003e+02 1.865e+02, threshold=1.835e+02, percent-clipped=0.0 2023-11-24 20:29:56,170 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2983006.6666666665, ans=0.125 2023-11-24 20:30:25,782 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2600, loss[loss=0.05859, simple_loss=0.07605, pruned_loss=0.01232, audio_tagging_loss=0.008242, over 14004.00 frames. ], tot_loss[loss=0.0661, simple_loss=0.08901, pruned_loss=0.01266, audio_tagging_loss=0.008934, over 3032052.51 frames. ], batch size: 52, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:30:34,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2983206.6666666665, ans=0.1 2023-11-24 20:30:48,125 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447500 2023-11-24 20:31:04,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2983406.6666666665, ans=0.0 2023-11-24 20:31:06,759 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2983406.6666666665, ans=0.125 2023-11-24 20:31:29,465 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2650, loss[loss=0.04589, simple_loss=0.05597, pruned_loss=0.006862, audio_tagging_loss=0.01105, over 14339.00 frames. ], tot_loss[loss=0.06562, simple_loss=0.08817, pruned_loss=0.01259, audio_tagging_loss=0.008944, over 3034444.85 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:31:50,184 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447550 2023-11-24 20:31:53,544 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.521e+01 8.637e+01 9.375e+01 1.000e+02 1.273e+02, threshold=1.875e+02, percent-clipped=0.0 2023-11-24 20:32:01,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2983673.3333333335, ans=0.125 2023-11-24 20:32:01,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2983673.3333333335, ans=0.1 2023-11-24 20:32:18,514 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2983806.6666666665, ans=0.125 2023-11-24 20:32:21,342 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.92 vs. limit=15.0 2023-11-24 20:32:24,562 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2983806.6666666665, ans=0.1 2023-11-24 20:32:30,772 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2700, loss[loss=0.07325, simple_loss=0.1036, pruned_loss=0.01421, audio_tagging_loss=0.007249, over 15509.00 frames. ], tot_loss[loss=0.06643, simple_loss=0.08956, pruned_loss=0.01283, audio_tagging_loss=0.008814, over 3049207.06 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:32:48,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2983940.0, ans=0.0 2023-11-24 20:32:52,245 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447600 2023-11-24 20:32:58,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2984006.6666666665, ans=0.125 2023-11-24 20:33:03,587 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2984006.6666666665, ans=0.125 2023-11-24 20:33:03,719 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2984006.6666666665, ans=0.0 2023-11-24 20:33:33,447 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2750, loss[loss=0.06117, simple_loss=0.07786, pruned_loss=0.0147, audio_tagging_loss=0.007552, over 14937.00 frames. ], tot_loss[loss=0.06651, simple_loss=0.08964, pruned_loss=0.01288, audio_tagging_loss=0.008812, over 3048856.20 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:33:37,184 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2984206.6666666665, ans=0.1 2023-11-24 20:33:55,339 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447650 2023-11-24 20:33:59,989 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.580e+01 9.104e+01 1.006e+02 1.298e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 20:34:27,044 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 20:34:30,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2984473.3333333335, ans=0.125 2023-11-24 20:34:30,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2984473.3333333335, ans=0.125 2023-11-24 20:34:35,509 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.92 vs. limit=22.5 2023-11-24 20:34:35,946 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2800, loss[loss=0.06847, simple_loss=0.09188, pruned_loss=0.0155, audio_tagging_loss=0.007032, over 15894.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.08996, pruned_loss=0.01288, audio_tagging_loss=0.008884, over 3049388.70 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:34:40,347 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2984540.0, ans=0.1 2023-11-24 20:34:56,473 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2984606.6666666665, ans=0.0 2023-11-24 20:34:57,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447700 2023-11-24 20:34:59,993 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2984673.3333333335, ans=0.125 2023-11-24 20:35:05,975 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2984673.3333333335, ans=0.0 2023-11-24 20:35:08,919 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.49 vs. limit=10.0 2023-11-24 20:35:21,221 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2984740.0, ans=0.125 2023-11-24 20:35:22,572 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2984740.0, ans=0.125 2023-11-24 20:35:38,993 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2850, loss[loss=0.0732, simple_loss=0.1101, pruned_loss=0.01033, audio_tagging_loss=0.007809, over 15719.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09038, pruned_loss=0.01288, audio_tagging_loss=0.008801, over 3046263.89 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:35:48,085 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.21 vs. limit=15.0 2023-11-24 20:35:52,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2984940.0, ans=0.0 2023-11-24 20:35:59,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447750 2023-11-24 20:36:05,248 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.522e+01 9.152e+01 9.707e+01 1.376e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 20:36:05,680 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2985006.6666666665, ans=0.0 2023-11-24 20:36:14,581 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2985006.6666666665, ans=0.125 2023-11-24 20:36:33,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2985140.0, ans=0.04949747468305833 2023-11-24 20:36:41,875 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2900, loss[loss=0.06445, simple_loss=0.07331, pruned_loss=0.01366, audio_tagging_loss=0.01413, over 14645.00 frames. ], tot_loss[loss=0.06665, simple_loss=0.09, pruned_loss=0.01275, audio_tagging_loss=0.008893, over 3049720.72 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:37:01,142 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2985273.3333333335, ans=0.0 2023-11-24 20:37:02,203 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2985273.3333333335, ans=0.0 2023-11-24 20:37:03,292 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447800 2023-11-24 20:37:25,077 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2985406.6666666665, ans=0.125 2023-11-24 20:37:25,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2985406.6666666665, ans=0.125 2023-11-24 20:37:42,405 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 20:37:45,075 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 2950, loss[loss=0.08989, simple_loss=0.1158, pruned_loss=0.02082, audio_tagging_loss=0.01115, over 14284.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09128, pruned_loss=0.01298, audio_tagging_loss=0.008856, over 3053070.19 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:37:45,736 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=2985540.0, ans=15.0 2023-11-24 20:37:56,601 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2985606.6666666665, ans=0.07 2023-11-24 20:38:06,282 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447850 2023-11-24 20:38:10,883 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.599e+01 9.301e+01 9.933e+01 1.313e+02, threshold=1.860e+02, percent-clipped=0.0 2023-11-24 20:38:17,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2985673.3333333335, ans=0.05 2023-11-24 20:38:33,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2985806.6666666665, ans=0.1 2023-11-24 20:38:40,727 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2985806.6666666665, ans=0.125 2023-11-24 20:38:46,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5 2023-11-24 20:38:47,535 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3000, loss[loss=0.06513, simple_loss=0.08859, pruned_loss=0.01054, audio_tagging_loss=0.0103, over 15085.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09055, pruned_loss=0.01288, audio_tagging_loss=0.008943, over 3046697.01 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:38:47,536 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 20:39:19,400 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.3025, 4.3080, 4.5050, 4.4794], device='cuda:3') 2023-11-24 20:39:31,504 INFO [train_asr.py:1253] (3/4) Epoch 38, validation: loss=0.05738, simple_loss=0.0507, pruned_loss=0.005077, audio_tagging_loss=0.02696, over 4681554.00 frames. 2023-11-24 20:39:31,505 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 20:39:31,749 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2985873.3333333335, ans=0.07 2023-11-24 20:39:42,330 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2985873.3333333335, ans=0.125 2023-11-24 20:39:44,658 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2985940.0, ans=0.0 2023-11-24 20:39:52,645 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447900 2023-11-24 20:40:11,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2986073.3333333335, ans=0.125 2023-11-24 20:40:19,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2986073.3333333335, ans=0.125 2023-11-24 20:40:34,328 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3050, loss[loss=0.05978, simple_loss=0.07624, pruned_loss=0.01125, audio_tagging_loss=0.01042, over 16142.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09037, pruned_loss=0.01282, audio_tagging_loss=0.009049, over 3044754.93 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:40:55,641 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 447950 2023-11-24 20:40:55,871 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2986273.3333333335, ans=0.125 2023-11-24 20:40:57,395 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.00 vs. limit=15.0 2023-11-24 20:41:00,358 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.683e+01 8.702e+01 9.218e+01 9.877e+01 1.257e+02, threshold=1.844e+02, percent-clipped=0.0 2023-11-24 20:41:10,493 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 20:41:24,317 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2986473.3333333335, ans=0.0 2023-11-24 20:41:34,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2986473.3333333335, ans=0.0 2023-11-24 20:41:37,185 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3100, loss[loss=0.0836, simple_loss=0.1261, pruned_loss=0.01351, audio_tagging_loss=0.007033, over 15740.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.0905, pruned_loss=0.01285, audio_tagging_loss=0.009041, over 3043954.13 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:41:52,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2986606.6666666665, ans=0.5 2023-11-24 20:41:58,114 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448000 2023-11-24 20:41:59,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2986606.6666666665, ans=0.0 2023-11-24 20:41:59,539 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2986606.6666666665, ans=0.125 2023-11-24 20:42:07,459 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.01 vs. limit=15.0 2023-11-24 20:42:37,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2986806.6666666665, ans=0.125 2023-11-24 20:42:40,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2986806.6666666665, ans=0.0 2023-11-24 20:42:40,617 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2986806.6666666665, ans=0.125 2023-11-24 20:42:42,700 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3150, loss[loss=0.06337, simple_loss=0.0906, pruned_loss=0.00985, audio_tagging_loss=0.008222, over 14111.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09131, pruned_loss=0.0129, audio_tagging_loss=0.00903, over 3049731.32 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:42:57,562 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=22.5 2023-11-24 20:43:04,330 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448050 2023-11-24 20:43:08,909 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.191e+01 8.654e+01 9.120e+01 9.857e+01 1.344e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-24 20:43:09,156 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2987006.6666666665, ans=0.125 2023-11-24 20:43:13,303 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2987006.6666666665, ans=0.0 2023-11-24 20:43:13,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2987006.6666666665, ans=0.0 2023-11-24 20:43:33,138 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.04 vs. limit=15.0 2023-11-24 20:43:45,949 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3200, loss[loss=0.06548, simple_loss=0.09566, pruned_loss=0.01103, audio_tagging_loss=0.006613, over 15581.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09091, pruned_loss=0.01271, audio_tagging_loss=0.00898, over 3054841.44 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:43:52,508 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.46 vs. limit=12.0 2023-11-24 20:44:07,212 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448100 2023-11-24 20:44:35,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2987473.3333333335, ans=0.1 2023-11-24 20:44:47,845 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3250, loss[loss=0.06844, simple_loss=0.09012, pruned_loss=0.01422, audio_tagging_loss=0.009169, over 15181.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09074, pruned_loss=0.01262, audio_tagging_loss=0.009016, over 3050085.29 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:45:08,968 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448150 2023-11-24 20:45:10,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2987606.6666666665, ans=0.125 2023-11-24 20:45:15,408 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.226e+01 8.565e+01 9.106e+01 9.950e+01 1.221e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 20:45:26,973 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2987740.0, ans=0.2 2023-11-24 20:45:28,003 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 20:45:50,295 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3300, loss[loss=0.0576, simple_loss=0.08104, pruned_loss=0.007953, audio_tagging_loss=0.009124, over 16075.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09041, pruned_loss=0.01274, audio_tagging_loss=0.009124, over 3053745.16 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:45:51,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2987873.3333333335, ans=0.0 2023-11-24 20:45:55,661 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.10 vs. limit=15.0 2023-11-24 20:45:58,388 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2987873.3333333335, ans=0.0 2023-11-24 20:46:11,541 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448200 2023-11-24 20:46:23,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2988006.6666666665, ans=0.125 2023-11-24 20:46:35,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2988073.3333333335, ans=0.125 2023-11-24 20:46:50,584 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2988140.0, ans=0.125 2023-11-24 20:46:53,179 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3350, loss[loss=0.05515, simple_loss=0.06712, pruned_loss=0.01279, audio_tagging_loss=0.008798, over 13843.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09079, pruned_loss=0.01279, audio_tagging_loss=0.009116, over 3053117.00 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:47:02,908 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2988206.6666666665, ans=0.09899494936611666 2023-11-24 20:47:07,861 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.47 vs. limit=10.0 2023-11-24 20:47:13,913 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448250 2023-11-24 20:47:16,497 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2988340.0, ans=0.125 2023-11-24 20:47:19,734 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.779e+01 9.397e+01 1.017e+02 1.316e+02, threshold=1.879e+02, percent-clipped=0.0 2023-11-24 20:47:35,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2988406.6666666665, ans=0.125 2023-11-24 20:47:53,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2988540.0, ans=0.1 2023-11-24 20:47:54,743 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3400, loss[loss=0.06758, simple_loss=0.09579, pruned_loss=0.01093, audio_tagging_loss=0.008748, over 15731.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.0903, pruned_loss=0.01279, audio_tagging_loss=0.008966, over 3056708.25 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:48:00,373 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2988540.0, ans=0.2 2023-11-24 20:48:16,227 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448300 2023-11-24 20:48:32,268 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.20 vs. limit=12.0 2023-11-24 20:48:39,469 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.01 vs. limit=15.0 2023-11-24 20:48:49,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2988806.6666666665, ans=0.125 2023-11-24 20:48:57,293 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3450, loss[loss=0.05637, simple_loss=0.07279, pruned_loss=0.01028, audio_tagging_loss=0.009691, over 14586.00 frames. ], tot_loss[loss=0.06648, simple_loss=0.08972, pruned_loss=0.01272, audio_tagging_loss=0.008904, over 3049330.16 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:49:19,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448350 2023-11-24 20:49:19,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2988940.0, ans=0.1 2023-11-24 20:49:25,635 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.675e+01 9.257e+01 9.906e+01 1.659e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 20:49:28,666 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2989006.6666666665, ans=0.1 2023-11-24 20:49:33,936 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2989006.6666666665, ans=0.2 2023-11-24 20:50:01,080 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3500, loss[loss=0.09439, simple_loss=0.1316, pruned_loss=0.021, audio_tagging_loss=0.007599, over 15629.00 frames. ], tot_loss[loss=0.06638, simple_loss=0.08989, pruned_loss=0.01272, audio_tagging_loss=0.008715, over 3049788.37 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:50:04,166 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2989206.6666666665, ans=0.125 2023-11-24 20:50:17,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2989273.3333333335, ans=0.2 2023-11-24 20:50:22,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448400 2023-11-24 20:50:32,137 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 20:50:36,064 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2989340.0, ans=0.125 2023-11-24 20:50:46,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2989406.6666666665, ans=0.04949747468305833 2023-11-24 20:51:02,976 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2989540.0, ans=0.125 2023-11-24 20:51:03,971 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3550, loss[loss=0.07058, simple_loss=0.09635, pruned_loss=0.0145, audio_tagging_loss=0.007908, over 15064.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.09022, pruned_loss=0.0128, audio_tagging_loss=0.008724, over 3053357.33 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0 2023-11-24 20:51:04,227 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2989540.0, ans=0.0 2023-11-24 20:51:25,106 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448450 2023-11-24 20:51:32,067 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.602e+01 9.062e+01 9.844e+01 1.252e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 20:51:32,880 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.66 vs. limit=15.0 2023-11-24 20:51:34,755 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2989673.3333333335, ans=0.0 2023-11-24 20:51:39,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.65 vs. limit=15.0 2023-11-24 20:51:40,171 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2989673.3333333335, ans=0.0 2023-11-24 20:51:48,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2989740.0, ans=0.125 2023-11-24 20:51:55,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2989806.6666666665, ans=0.125 2023-11-24 20:52:06,926 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3600, loss[loss=0.04883, simple_loss=0.06808, pruned_loss=0.005844, audio_tagging_loss=0.008946, over 14747.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.09026, pruned_loss=0.01275, audio_tagging_loss=0.008763, over 3051046.68 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:52:13,510 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.62 vs. limit=15.0 2023-11-24 20:52:14,336 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2989873.3333333335, ans=0.0 2023-11-24 20:52:22,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2989940.0, ans=0.1 2023-11-24 20:52:29,120 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448500 2023-11-24 20:52:36,472 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2990006.6666666665, ans=0.04949747468305833 2023-11-24 20:52:49,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2990073.3333333335, ans=0.1 2023-11-24 20:53:05,406 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2990140.0, ans=0.125 2023-11-24 20:53:07,598 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2990140.0, ans=0.1 2023-11-24 20:53:09,703 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3650, loss[loss=0.06586, simple_loss=0.09206, pruned_loss=0.01046, audio_tagging_loss=0.009369, over 16033.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09146, pruned_loss=0.01297, audio_tagging_loss=0.008689, over 3053644.23 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:53:16,412 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2990206.6666666665, ans=0.125 2023-11-24 20:53:21,341 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2990273.3333333335, ans=0.0 2023-11-24 20:53:30,723 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448550 2023-11-24 20:53:36,941 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.710e+01 8.766e+01 9.382e+01 1.003e+02 1.165e+02, threshold=1.876e+02, percent-clipped=0.0 2023-11-24 20:53:56,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.56 vs. limit=10.0 2023-11-24 20:54:11,708 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3700, loss[loss=0.07498, simple_loss=0.09467, pruned_loss=0.01531, audio_tagging_loss=0.01234, over 14789.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09199, pruned_loss=0.01296, audio_tagging_loss=0.008624, over 3058721.55 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:54:14,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2990540.0, ans=0.0 2023-11-24 20:54:17,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.22 vs. limit=12.0 2023-11-24 20:54:28,402 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0 2023-11-24 20:54:29,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2990606.6666666665, ans=0.1 2023-11-24 20:54:32,479 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448600 2023-11-24 20:54:43,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2990673.3333333335, ans=0.07 2023-11-24 20:54:49,175 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2990740.0, ans=0.125 2023-11-24 20:55:00,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2990806.6666666665, ans=0.125 2023-11-24 20:55:14,089 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3750, loss[loss=0.07158, simple_loss=0.08992, pruned_loss=0.01489, audio_tagging_loss=0.01173, over 15839.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09149, pruned_loss=0.01282, audio_tagging_loss=0.008732, over 3054951.16 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:55:14,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2990873.3333333335, ans=0.125 2023-11-24 20:55:35,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448650 2023-11-24 20:55:35,798 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2990940.0, ans=0.125 2023-11-24 20:55:41,382 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.492e+01 9.238e+01 9.913e+01 1.162e+02, threshold=1.848e+02, percent-clipped=0.0 2023-11-24 20:55:54,935 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 20:55:56,890 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.09 vs. limit=15.0 2023-11-24 20:56:15,260 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3800, loss[loss=0.09039, simple_loss=0.128, pruned_loss=0.01941, audio_tagging_loss=0.006995, over 16880.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09195, pruned_loss=0.01291, audio_tagging_loss=0.008718, over 3054949.67 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:56:28,938 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.84 vs. limit=15.0 2023-11-24 20:56:36,680 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448700 2023-11-24 20:56:46,378 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.54 vs. limit=10.0 2023-11-24 20:56:54,037 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2991406.6666666665, ans=0.125 2023-11-24 20:57:17,970 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3850, loss[loss=0.04893, simple_loss=0.06257, pruned_loss=0.006278, audio_tagging_loss=0.01137, over 14561.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09194, pruned_loss=0.01305, audio_tagging_loss=0.008869, over 3057164.23 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:57:18,230 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2991540.0, ans=0.125 2023-11-24 20:57:38,535 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448750 2023-11-24 20:57:42,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2991673.3333333335, ans=0.1 2023-11-24 20:57:43,505 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2991673.3333333335, ans=0.0 2023-11-24 20:57:44,193 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.441e+01 8.712e+01 9.172e+01 9.948e+01 1.223e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-24 20:58:04,289 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.21 vs. limit=6.0 2023-11-24 20:58:19,597 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3900, loss[loss=0.07925, simple_loss=0.1111, pruned_loss=0.01311, audio_tagging_loss=0.01058, over 15154.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09226, pruned_loss=0.01305, audio_tagging_loss=0.008948, over 3051584.59 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0 2023-11-24 20:58:28,222 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2991873.3333333335, ans=0.125 2023-11-24 20:58:40,337 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448800 2023-11-24 20:58:59,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2992073.3333333335, ans=0.09899494936611666 2023-11-24 20:59:00,825 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2992073.3333333335, ans=0.125 2023-11-24 20:59:10,758 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2992140.0, ans=0.0 2023-11-24 20:59:15,729 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.88 vs. limit=22.5 2023-11-24 20:59:18,846 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2992140.0, ans=0.1 2023-11-24 20:59:20,994 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 3950, loss[loss=0.07598, simple_loss=0.1098, pruned_loss=0.01622, audio_tagging_loss=0.004836, over 15772.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09151, pruned_loss=0.01299, audio_tagging_loss=0.008967, over 3055283.09 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 20:59:21,359 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2992206.6666666665, ans=0.0 2023-11-24 20:59:24,115 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2992206.6666666665, ans=0.125 2023-11-24 20:59:35,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2992273.3333333335, ans=0.0 2023-11-24 20:59:37,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2992273.3333333335, ans=0.125 2023-11-24 20:59:37,400 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.38 vs. limit=12.0 2023-11-24 20:59:42,960 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448850 2023-11-24 20:59:46,660 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2992340.0, ans=0.1 2023-11-24 20:59:47,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2992340.0, ans=0.0 2023-11-24 20:59:48,705 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.553e+01 9.326e+01 9.949e+01 1.288e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 21:00:02,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2992406.6666666665, ans=0.125 2023-11-24 21:00:11,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2992473.3333333335, ans=0.0 2023-11-24 21:00:14,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2992473.3333333335, ans=0.125 2023-11-24 21:00:18,280 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.59 vs. limit=15.0 2023-11-24 21:00:23,987 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4000, loss[loss=0.08181, simple_loss=0.111, pruned_loss=0.01949, audio_tagging_loss=0.006835, over 16418.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09131, pruned_loss=0.01302, audio_tagging_loss=0.008946, over 3057095.55 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:00:26,942 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0 2023-11-24 21:00:27,975 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 21:00:35,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2992606.6666666665, ans=0.125 2023-11-24 21:00:44,788 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448900 2023-11-24 21:00:46,772 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0 2023-11-24 21:01:09,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2992740.0, ans=0.1 2023-11-24 21:01:25,068 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2992873.3333333335, ans=0.0 2023-11-24 21:01:25,960 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4050, loss[loss=0.07372, simple_loss=0.1082, pruned_loss=0.01447, audio_tagging_loss=0.00515, over 15130.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09131, pruned_loss=0.01297, audio_tagging_loss=0.008927, over 3058725.07 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:01:26,233 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2992873.3333333335, ans=0.125 2023-11-24 21:01:27,220 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 21:01:47,016 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 448950 2023-11-24 21:01:54,344 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.560e+01 8.711e+01 9.505e+01 1.006e+02 1.290e+02, threshold=1.901e+02, percent-clipped=0.0 2023-11-24 21:02:02,775 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0 2023-11-24 21:02:09,686 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2993073.3333333335, ans=0.09899494936611666 2023-11-24 21:02:24,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2993140.0, ans=0.125 2023-11-24 21:02:27,591 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4100, loss[loss=0.04139, simple_loss=0.05251, pruned_loss=0.003924, audio_tagging_loss=0.01121, over 15034.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09187, pruned_loss=0.01303, audio_tagging_loss=0.009019, over 3056348.06 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:02:50,057 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449000 2023-11-24 21:02:56,726 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2993340.0, ans=0.125 2023-11-24 21:03:31,083 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4150, loss[loss=0.05803, simple_loss=0.0802, pruned_loss=0.008428, audio_tagging_loss=0.009498, over 15861.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.0924, pruned_loss=0.01309, audio_tagging_loss=0.008855, over 3054028.57 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:03:41,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2993540.0, ans=0.0 2023-11-24 21:03:49,489 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.99 vs. limit=12.0 2023-11-24 21:03:49,573 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.38 vs. limit=22.5 2023-11-24 21:03:52,555 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449050 2023-11-24 21:03:55,550 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.07 vs. limit=15.0 2023-11-24 21:03:59,448 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.744e+01 8.629e+01 9.149e+01 9.710e+01 1.194e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 21:04:00,042 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.67 vs. limit=15.0 2023-11-24 21:04:02,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2993673.3333333335, ans=0.0 2023-11-24 21:04:10,184 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.03 vs. limit=15.0 2023-11-24 21:04:15,006 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 21:04:22,582 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.88 vs. limit=15.0 2023-11-24 21:04:33,283 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4200, loss[loss=0.05036, simple_loss=0.05977, pruned_loss=0.009451, audio_tagging_loss=0.01102, over 14318.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09243, pruned_loss=0.01318, audio_tagging_loss=0.008818, over 3046775.30 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:04:50,930 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0 2023-11-24 21:04:53,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449100 2023-11-24 21:04:57,529 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.31 vs. limit=15.0 2023-11-24 21:04:59,436 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2994006.6666666665, ans=0.125 2023-11-24 21:05:13,056 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2994073.3333333335, ans=0.125 2023-11-24 21:05:13,691 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=7.75 vs. limit=15.0 2023-11-24 21:05:24,462 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.49 vs. limit=12.0 2023-11-24 21:05:35,512 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4250, loss[loss=0.08044, simple_loss=0.1089, pruned_loss=0.01841, audio_tagging_loss=0.00756, over 16009.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09217, pruned_loss=0.01305, audio_tagging_loss=0.008734, over 3042432.06 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:05:45,945 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2994206.6666666665, ans=0.07 2023-11-24 21:05:56,954 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449150 2023-11-24 21:06:05,032 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.615e+01 8.581e+01 9.086e+01 9.848e+01 1.238e+02, threshold=1.817e+02, percent-clipped=0.0 2023-11-24 21:06:07,264 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.07 vs. limit=22.5 2023-11-24 21:06:17,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2994406.6666666665, ans=0.0 2023-11-24 21:06:34,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2994473.3333333335, ans=0.125 2023-11-24 21:06:37,935 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4300, loss[loss=0.05851, simple_loss=0.08901, pruned_loss=0.005676, audio_tagging_loss=0.008331, over 15791.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09248, pruned_loss=0.01307, audio_tagging_loss=0.008651, over 3041749.71 frames. ], batch size: 63, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:06:38,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2994540.0, ans=0.05 2023-11-24 21:06:39,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2994540.0, ans=0.1 2023-11-24 21:06:47,683 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2994540.0, ans=0.125 2023-11-24 21:06:59,456 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449200 2023-11-24 21:07:40,219 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4350, loss[loss=0.06204, simple_loss=0.08498, pruned_loss=0.01286, audio_tagging_loss=0.006702, over 14532.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09155, pruned_loss=0.01301, audio_tagging_loss=0.00866, over 3036454.93 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:07:53,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2994940.0, ans=0.2 2023-11-24 21:08:01,549 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449250 2023-11-24 21:08:10,474 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.676e+01 9.413e+01 1.023e+02 1.276e+02, threshold=1.883e+02, percent-clipped=0.0 2023-11-24 21:08:15,285 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0 2023-11-24 21:08:23,018 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2995073.3333333335, ans=0.0 2023-11-24 21:08:31,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2995140.0, ans=0.0 2023-11-24 21:08:42,714 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.21 vs. limit=5.0 2023-11-24 21:08:42,979 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4400, loss[loss=0.04009, simple_loss=0.05005, pruned_loss=0.00336, audio_tagging_loss=0.01171, over 15814.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09162, pruned_loss=0.01304, audio_tagging_loss=0.008623, over 3042891.58 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:09:04,472 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449300 2023-11-24 21:09:12,816 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2995340.0, ans=0.0 2023-11-24 21:09:21,624 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2995406.6666666665, ans=0.95 2023-11-24 21:09:21,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2995406.6666666665, ans=0.0 2023-11-24 21:09:45,356 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4450, loss[loss=0.05211, simple_loss=0.05826, pruned_loss=0.0116, audio_tagging_loss=0.01139, over 15460.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.09072, pruned_loss=0.01287, audio_tagging_loss=0.00867, over 3046476.18 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:10:06,696 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449350 2023-11-24 21:10:13,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2995673.3333333335, ans=0.125 2023-11-24 21:10:16,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.51 vs. limit=12.0 2023-11-24 21:10:16,946 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.434e+01 8.708e+01 9.388e+01 1.016e+02 1.957e+02, threshold=1.878e+02, percent-clipped=1.0 2023-11-24 21:10:28,500 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2995740.0, ans=0.1 2023-11-24 21:10:29,200 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.39 vs. limit=15.0 2023-11-24 21:10:47,834 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4500, loss[loss=0.08283, simple_loss=0.1127, pruned_loss=0.01943, audio_tagging_loss=0.007037, over 15146.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09063, pruned_loss=0.01286, audio_tagging_loss=0.008721, over 3051718.03 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:11:09,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449400 2023-11-24 21:11:10,964 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2995940.0, ans=0.125 2023-11-24 21:11:18,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2996006.6666666665, ans=0.125 2023-11-24 21:11:20,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2996006.6666666665, ans=0.125 2023-11-24 21:11:42,890 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2996140.0, ans=0.125 2023-11-24 21:11:51,469 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4550, loss[loss=0.0736, simple_loss=0.1028, pruned_loss=0.01485, audio_tagging_loss=0.007352, over 15586.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.09049, pruned_loss=0.01289, audio_tagging_loss=0.008756, over 3055426.78 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:11:55,092 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2996206.6666666665, ans=0.0 2023-11-24 21:11:57,610 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2996206.6666666665, ans=0.04949747468305833 2023-11-24 21:11:57,984 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.93 vs. limit=10.0 2023-11-24 21:12:12,993 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449450 2023-11-24 21:12:22,245 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.502e+01 8.617e+01 8.939e+01 9.828e+01 1.230e+02, threshold=1.788e+02, percent-clipped=0.0 2023-11-24 21:12:35,860 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 21:12:36,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2996406.6666666665, ans=0.125 2023-11-24 21:12:38,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2996406.6666666665, ans=0.1 2023-11-24 21:12:42,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2996473.3333333335, ans=0.2 2023-11-24 21:12:53,665 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4600, loss[loss=0.06692, simple_loss=0.0821, pruned_loss=0.01538, audio_tagging_loss=0.0105, over 14980.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09034, pruned_loss=0.01294, audio_tagging_loss=0.008856, over 3052893.68 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:13:03,961 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2996540.0, ans=0.125 2023-11-24 21:13:07,831 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.20 vs. limit=12.0 2023-11-24 21:13:12,611 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=3.95 vs. limit=6.0 2023-11-24 21:13:14,392 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449500 2023-11-24 21:13:14,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2996606.6666666665, ans=0.025 2023-11-24 21:13:19,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2996673.3333333335, ans=0.0 2023-11-24 21:13:20,955 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2996673.3333333335, ans=0.1 2023-11-24 21:13:22,076 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2996673.3333333335, ans=0.0 2023-11-24 21:13:28,137 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2996673.3333333335, ans=0.1 2023-11-24 21:13:55,727 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4650, loss[loss=0.07312, simple_loss=0.09909, pruned_loss=0.01325, audio_tagging_loss=0.01032, over 14898.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09075, pruned_loss=0.01307, audio_tagging_loss=0.008924, over 3046955.42 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:14:08,656 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2996940.0, ans=0.125 2023-11-24 21:14:15,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2996940.0, ans=0.025 2023-11-24 21:14:16,907 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449550 2023-11-24 21:14:16,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2996940.0, ans=0.2 2023-11-24 21:14:25,628 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2997006.6666666665, ans=0.1 2023-11-24 21:14:27,680 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.430e+01 9.300e+01 1.028e+02 1.616e+02, threshold=1.860e+02, percent-clipped=0.0 2023-11-24 21:14:54,404 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.76 vs. limit=15.0 2023-11-24 21:14:58,452 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4700, loss[loss=0.05342, simple_loss=0.0652, pruned_loss=0.01055, audio_tagging_loss=0.01027, over 15116.00 frames. ], tot_loss[loss=0.067, simple_loss=0.0901, pruned_loss=0.01295, audio_tagging_loss=0.008998, over 3043383.78 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:15:20,738 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449600 2023-11-24 21:15:22,008 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2997273.3333333335, ans=0.0 2023-11-24 21:15:54,274 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2997473.3333333335, ans=0.0 2023-11-24 21:16:02,264 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4750, loss[loss=0.06654, simple_loss=0.0911, pruned_loss=0.012, audio_tagging_loss=0.008985, over 15917.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.08985, pruned_loss=0.01279, audio_tagging_loss=0.009014, over 3037855.37 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:16:02,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2997540.0, ans=0.0 2023-11-24 21:16:11,304 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2997540.0, ans=0.1 2023-11-24 21:16:11,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2997540.0, ans=0.125 2023-11-24 21:16:14,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.65 vs. limit=6.0 2023-11-24 21:16:23,100 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449650 2023-11-24 21:16:33,254 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.234e+01 8.730e+01 9.403e+01 9.994e+01 1.287e+02, threshold=1.881e+02, percent-clipped=0.0 2023-11-24 21:16:45,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2997740.0, ans=0.0 2023-11-24 21:17:04,667 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4800, loss[loss=0.05555, simple_loss=0.07077, pruned_loss=0.01075, audio_tagging_loss=0.009418, over 15870.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09072, pruned_loss=0.01297, audio_tagging_loss=0.009097, over 3044405.93 frames. ], batch size: 62, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:17:06,297 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.19 vs. limit=15.0 2023-11-24 21:17:07,636 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.whiten.whitening_limit, batch_count=2997873.3333333335, ans=12.0 2023-11-24 21:17:16,447 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2997940.0, ans=0.125 2023-11-24 21:17:21,136 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2997940.0, ans=0.0 2023-11-24 21:17:25,550 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449700 2023-11-24 21:17:27,050 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2997940.0, ans=0.2 2023-11-24 21:17:31,605 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2998006.6666666665, ans=0.125 2023-11-24 21:17:35,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2998006.6666666665, ans=0.125 2023-11-24 21:17:47,837 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.84 vs. limit=15.0 2023-11-24 21:18:00,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2998140.0, ans=0.0 2023-11-24 21:18:06,833 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4850, loss[loss=0.05419, simple_loss=0.06953, pruned_loss=0.00876, audio_tagging_loss=0.01066, over 16083.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09053, pruned_loss=0.01288, audio_tagging_loss=0.009215, over 3041176.78 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:18:26,780 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.64 vs. limit=15.0 2023-11-24 21:18:28,653 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449750 2023-11-24 21:18:38,539 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.567e+01 9.194e+01 1.005e+02 1.604e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 21:18:42,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2998340.0, ans=0.125 2023-11-24 21:18:47,247 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2998406.6666666665, ans=0.125 2023-11-24 21:19:09,500 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4900, loss[loss=0.07816, simple_loss=0.1033, pruned_loss=0.01881, audio_tagging_loss=0.007702, over 15467.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.0901, pruned_loss=0.01282, audio_tagging_loss=0.009151, over 3039614.55 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:19:27,108 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2998606.6666666665, ans=0.1 2023-11-24 21:19:31,519 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449800 2023-11-24 21:19:34,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2998673.3333333335, ans=0.2 2023-11-24 21:19:35,710 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2998673.3333333335, ans=0.125 2023-11-24 21:19:37,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2998673.3333333335, ans=0.125 2023-11-24 21:19:39,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2998673.3333333335, ans=0.125 2023-11-24 21:19:44,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2998673.3333333335, ans=0.2 2023-11-24 21:20:06,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2998806.6666666665, ans=0.125 2023-11-24 21:20:10,868 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2998806.6666666665, ans=0.0 2023-11-24 21:20:13,068 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 4950, loss[loss=0.0508, simple_loss=0.06189, pruned_loss=0.009344, audio_tagging_loss=0.01051, over 15350.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09119, pruned_loss=0.01294, audio_tagging_loss=0.008963, over 3035275.76 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:20:13,339 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2998873.3333333335, ans=10.0 2023-11-24 21:20:15,635 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2998873.3333333335, ans=0.125 2023-11-24 21:20:22,997 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.34 vs. limit=15.0 2023-11-24 21:20:33,965 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449850 2023-11-24 21:20:34,241 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2998940.0, ans=0.0 2023-11-24 21:20:35,783 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.72 vs. limit=10.0 2023-11-24 21:20:42,504 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2999006.6666666665, ans=0.025 2023-11-24 21:20:43,264 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.508e+01 9.344e+01 9.867e+01 1.192e+02, threshold=1.869e+02, percent-clipped=0.0 2023-11-24 21:21:04,212 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2999140.0, ans=0.1 2023-11-24 21:21:12,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2999140.0, ans=0.125 2023-11-24 21:21:15,142 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5000, loss[loss=0.07719, simple_loss=0.1032, pruned_loss=0.01569, audio_tagging_loss=0.009884, over 15029.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09063, pruned_loss=0.01278, audio_tagging_loss=0.008794, over 3038979.98 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:21:21,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2999206.6666666665, ans=0.125 2023-11-24 21:21:22,729 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2999206.6666666665, ans=0.0 2023-11-24 21:21:36,331 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449900 2023-11-24 21:21:36,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2999273.3333333335, ans=0.125 2023-11-24 21:21:48,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2999340.0, ans=0.025 2023-11-24 21:21:55,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2999406.6666666665, ans=0.0 2023-11-24 21:21:58,954 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2999406.6666666665, ans=0.1 2023-11-24 21:22:03,191 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2999406.6666666665, ans=0.04949747468305833 2023-11-24 21:22:04,518 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.76 vs. limit=15.0 2023-11-24 21:22:10,119 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2999473.3333333335, ans=0.5 2023-11-24 21:22:16,955 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5050, loss[loss=0.05629, simple_loss=0.06829, pruned_loss=0.0115, audio_tagging_loss=0.01065, over 15087.00 frames. ], tot_loss[loss=0.06614, simple_loss=0.08976, pruned_loss=0.01247, audio_tagging_loss=0.008795, over 3042584.36 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:22:33,362 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.51 vs. limit=15.0 2023-11-24 21:22:38,481 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 449950 2023-11-24 21:22:41,187 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2999673.3333333335, ans=0.0 2023-11-24 21:22:48,396 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.754e+01 9.208e+01 9.764e+01 1.294e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-24 21:23:19,851 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5100, loss[loss=0.07774, simple_loss=0.102, pruned_loss=0.01722, audio_tagging_loss=0.009513, over 13588.00 frames. ], tot_loss[loss=0.06566, simple_loss=0.08882, pruned_loss=0.01234, audio_tagging_loss=0.008908, over 3037509.21 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:23:22,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2999873.3333333335, ans=0.0 2023-11-24 21:23:38,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2999940.0, ans=0.0 2023-11-24 21:23:40,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450000 2023-11-24 21:23:59,616 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3000073.3333333335, ans=0.125 2023-11-24 21:23:59,672 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=3000073.3333333335, ans=0.125 2023-11-24 21:24:22,060 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5150, loss[loss=0.08017, simple_loss=0.1201, pruned_loss=0.01336, audio_tagging_loss=0.006767, over 16320.00 frames. ], tot_loss[loss=0.0658, simple_loss=0.08913, pruned_loss=0.01246, audio_tagging_loss=0.008781, over 3039484.75 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:24:24,143 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3000206.6666666665, ans=0.1 2023-11-24 21:24:43,485 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450050 2023-11-24 21:24:53,309 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.721e+01 9.361e+01 9.842e+01 1.528e+02, threshold=1.872e+02, percent-clipped=0.0 2023-11-24 21:25:24,409 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5200, loss[loss=0.0793, simple_loss=0.1172, pruned_loss=0.0155, audio_tagging_loss=0.005219, over 15392.00 frames. ], tot_loss[loss=0.06594, simple_loss=0.0894, pruned_loss=0.01251, audio_tagging_loss=0.008725, over 3037811.26 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:25:25,951 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=3000540.0, ans=0.04949747468305833 2023-11-24 21:25:29,448 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3000540.0, ans=0.1 2023-11-24 21:25:44,757 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=3000606.6666666665, ans=0.2 2023-11-24 21:25:45,791 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450100 2023-11-24 21:26:06,295 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3000740.0, ans=0.0 2023-11-24 21:26:18,873 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3000806.6666666665, ans=0.125 2023-11-24 21:26:22,538 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=3000806.6666666665, ans=0.125 2023-11-24 21:26:25,198 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.10 vs. limit=15.0 2023-11-24 21:26:27,390 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5250, loss[loss=0.07501, simple_loss=0.0965, pruned_loss=0.01754, audio_tagging_loss=0.009215, over 15395.00 frames. ], tot_loss[loss=0.06618, simple_loss=0.08976, pruned_loss=0.01261, audio_tagging_loss=0.008687, over 3043562.11 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:26:27,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3000873.3333333335, ans=0.125 2023-11-24 21:26:33,433 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=3000873.3333333335, ans=0.125 2023-11-24 21:26:33,449 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=3000873.3333333335, ans=0.04949747468305833 2023-11-24 21:26:48,582 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450150 2023-11-24 21:26:55,406 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.84 vs. limit=15.0 2023-11-24 21:26:59,278 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.195e+01 8.390e+01 9.326e+01 9.807e+01 1.099e+02, threshold=1.865e+02, percent-clipped=0.0 2023-11-24 21:27:18,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=3001140.0, ans=0.125 2023-11-24 21:27:29,181 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5300, loss[loss=0.07307, simple_loss=0.08707, pruned_loss=0.01811, audio_tagging_loss=0.01142, over 15100.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.09006, pruned_loss=0.01286, audio_tagging_loss=0.008732, over 3042579.29 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:27:30,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=3001206.6666666665, ans=0.0 2023-11-24 21:27:44,101 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=3001273.3333333335, ans=0.2 2023-11-24 21:27:50,016 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450200 2023-11-24 21:27:51,255 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3001273.3333333335, ans=0.1 2023-11-24 21:28:00,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=3001340.0, ans=0.2 2023-11-24 21:28:01,807 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=3001340.0, ans=0.025 2023-11-24 21:28:06,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=3001406.6666666665, ans=0.2 2023-11-24 21:28:29,079 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=3001473.3333333335, ans=0.125 2023-11-24 21:28:31,200 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5350, loss[loss=0.07362, simple_loss=0.09798, pruned_loss=0.016, audio_tagging_loss=0.00862, over 14392.00 frames. ], tot_loss[loss=0.06634, simple_loss=0.08987, pruned_loss=0.01272, audio_tagging_loss=0.00869, over 3045487.42 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:28:50,331 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3001606.6666666665, ans=0.0 2023-11-24 21:28:51,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=3001606.6666666665, ans=0.07 2023-11-24 21:28:52,502 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450250 2023-11-24 21:29:03,594 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.563e+01 9.107e+01 9.848e+01 1.290e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 21:29:12,766 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 21:29:14,000 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3001740.0, ans=0.125 2023-11-24 21:29:16,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3001740.0, ans=0.0 2023-11-24 21:29:24,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3001806.6666666665, ans=0.0 2023-11-24 21:29:33,374 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5400, loss[loss=0.0621, simple_loss=0.08625, pruned_loss=0.01164, audio_tagging_loss=0.007342, over 15458.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08982, pruned_loss=0.01264, audio_tagging_loss=0.008758, over 3049704.45 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:29:35,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3001873.3333333335, ans=0.0 2023-11-24 21:29:35,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3001873.3333333335, ans=0.125 2023-11-24 21:29:35,637 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.52 vs. limit=10.0 2023-11-24 21:29:36,793 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=15.0 2023-11-24 21:29:41,210 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=3001873.3333333335, ans=0.0 2023-11-24 21:29:49,385 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=3001940.0, ans=0.2 2023-11-24 21:29:51,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=3001940.0, ans=0.2 2023-11-24 21:29:55,202 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450300 2023-11-24 21:30:26,899 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=3002140.0, ans=10.0 2023-11-24 21:30:32,149 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.01 vs. limit=15.0 2023-11-24 21:30:34,890 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5450, loss[loss=0.04661, simple_loss=0.0574, pruned_loss=0.007255, audio_tagging_loss=0.01066, over 15118.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09024, pruned_loss=0.01288, audio_tagging_loss=0.008819, over 3045693.18 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:30:43,885 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.48 vs. limit=12.0 2023-11-24 21:30:56,253 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450350 2023-11-24 21:30:59,867 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3002340.0, ans=0.0 2023-11-24 21:31:00,613 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3002340.0, ans=0.125 2023-11-24 21:31:06,496 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3002340.0, ans=0.0 2023-11-24 21:31:07,239 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.563e+01 9.171e+01 9.760e+01 1.153e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-24 21:31:07,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3002340.0, ans=0.0 2023-11-24 21:31:36,964 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5500, loss[loss=0.06996, simple_loss=0.1039, pruned_loss=0.01109, audio_tagging_loss=0.006906, over 15424.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.0912, pruned_loss=0.01309, audio_tagging_loss=0.008866, over 3055099.65 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:31:39,763 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3002540.0, ans=0.0 2023-11-24 21:31:47,245 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3002540.0, ans=0.2 2023-11-24 21:31:57,995 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450400 2023-11-24 21:32:03,811 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.80 vs. limit=22.5 2023-11-24 21:32:05,785 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3002673.3333333335, ans=0.125 2023-11-24 21:32:26,544 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=3002806.6666666665, ans=0.0 2023-11-24 21:32:32,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3002806.6666666665, ans=0.125 2023-11-24 21:32:38,617 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5550, loss[loss=0.06068, simple_loss=0.08169, pruned_loss=0.01021, audio_tagging_loss=0.009624, over 14236.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09221, pruned_loss=0.01315, audio_tagging_loss=0.009024, over 3061017.63 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 21:32:48,484 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.35 vs. limit=15.0 2023-11-24 21:32:50,337 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3002940.0, ans=0.125 2023-11-24 21:32:54,588 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.14 vs. limit=10.0 2023-11-24 21:32:59,872 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450450 2023-11-24 21:33:09,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=3003006.6666666665, ans=0.125 2023-11-24 21:33:12,770 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.457e+01 9.165e+01 9.990e+01 1.185e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 21:33:33,015 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3003140.0, ans=0.125 2023-11-24 21:33:40,978 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5600, loss[loss=0.07502, simple_loss=0.1011, pruned_loss=0.01677, audio_tagging_loss=0.007693, over 15339.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09176, pruned_loss=0.01309, audio_tagging_loss=0.009094, over 3061066.64 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:33:55,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=3003273.3333333335, ans=0.0 2023-11-24 21:34:02,674 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450500 2023-11-24 21:34:04,407 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.94 vs. limit=15.0 2023-11-24 21:34:16,364 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 21:34:20,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3003406.6666666665, ans=0.125 2023-11-24 21:34:23,713 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 21:34:26,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=3003406.6666666665, ans=0.2 2023-11-24 21:34:28,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=3003406.6666666665, ans=0.2 2023-11-24 21:34:44,288 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5650, loss[loss=0.0755, simple_loss=0.107, pruned_loss=0.01435, audio_tagging_loss=0.007664, over 15247.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09091, pruned_loss=0.01293, audio_tagging_loss=0.009128, over 3061754.16 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:35:01,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=3003606.6666666665, ans=0.2 2023-11-24 21:35:03,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=3003606.6666666665, ans=0.2 2023-11-24 21:35:05,661 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450550 2023-11-24 21:35:08,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3003673.3333333335, ans=0.0 2023-11-24 21:35:11,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3003673.3333333335, ans=0.0 2023-11-24 21:35:14,275 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=3003673.3333333335, ans=0.0 2023-11-24 21:35:17,494 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.220e+01 8.565e+01 9.205e+01 1.008e+02 1.241e+02, threshold=1.841e+02, percent-clipped=0.0 2023-11-24 21:35:28,217 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=12.31 vs. limit=15.0 2023-11-24 21:35:40,183 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3003806.6666666665, ans=0.0 2023-11-24 21:35:46,050 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.51 vs. limit=15.0 2023-11-24 21:35:46,680 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5700, loss[loss=0.07627, simple_loss=0.1001, pruned_loss=0.01937, audio_tagging_loss=0.006834, over 14706.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09078, pruned_loss=0.01306, audio_tagging_loss=0.00912, over 3060354.97 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:35:48,484 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.31 vs. limit=12.0 2023-11-24 21:36:07,415 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.91 vs. limit=10.0 2023-11-24 21:36:08,118 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450600 2023-11-24 21:36:20,320 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.07 vs. limit=15.0 2023-11-24 21:36:34,112 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=3004073.3333333335, ans=0.125 2023-11-24 21:36:38,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3004140.0, ans=0.125 2023-11-24 21:36:49,709 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5750, loss[loss=0.1116, simple_loss=0.1612, pruned_loss=0.02648, audio_tagging_loss=0.004536, over 15072.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09087, pruned_loss=0.01298, audio_tagging_loss=0.008978, over 3055905.80 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:37:11,193 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450650 2023-11-24 21:37:15,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3004340.0, ans=0.125 2023-11-24 21:37:18,482 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=3004340.0, ans=0.0 2023-11-24 21:37:20,149 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3004340.0, ans=0.1 2023-11-24 21:37:21,470 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=3004340.0, ans=0.125 2023-11-24 21:37:21,679 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.66 vs. limit=12.0 2023-11-24 21:37:23,499 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.366e+01 9.124e+01 1.023e+02 1.214e+02, threshold=1.825e+02, percent-clipped=0.0 2023-11-24 21:37:37,676 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=15.0 2023-11-24 21:37:52,012 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5800, loss[loss=0.06097, simple_loss=0.07645, pruned_loss=0.01136, audio_tagging_loss=0.01138, over 16258.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09021, pruned_loss=0.01297, audio_tagging_loss=0.008918, over 3052854.09 frames. ], batch size: 63, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:37:58,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3004540.0, ans=0.1 2023-11-24 21:38:06,773 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=3004606.6666666665, ans=15.0 2023-11-24 21:38:13,119 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450700 2023-11-24 21:38:23,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=3004673.3333333335, ans=0.0 2023-11-24 21:38:27,218 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=15.0 2023-11-24 21:38:54,023 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5850, loss[loss=0.05683, simple_loss=0.06574, pruned_loss=0.01124, audio_tagging_loss=0.01272, over 15033.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.0912, pruned_loss=0.01306, audio_tagging_loss=0.008857, over 3048737.61 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:39:10,333 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3004940.0, ans=0.1 2023-11-24 21:39:12,567 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=3004940.0, ans=0.0 2023-11-24 21:39:13,734 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3004940.0, ans=0.0 2023-11-24 21:39:14,681 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450750 2023-11-24 21:39:17,243 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3005006.6666666665, ans=0.125 2023-11-24 21:39:26,994 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.612e+01 8.737e+01 9.217e+01 9.924e+01 1.196e+02, threshold=1.843e+02, percent-clipped=0.0 2023-11-24 21:39:44,814 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3005140.0, ans=0.125 2023-11-24 21:39:51,355 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3005140.0, ans=0.125 2023-11-24 21:39:52,917 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.97 vs. limit=10.0 2023-11-24 21:39:54,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3005206.6666666665, ans=0.125 2023-11-24 21:39:55,825 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5900, loss[loss=0.06077, simple_loss=0.09149, pruned_loss=0.007179, audio_tagging_loss=0.007848, over 15683.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09228, pruned_loss=0.01323, audio_tagging_loss=0.008778, over 3051611.18 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:39:58,483 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3005206.6666666665, ans=0.0 2023-11-24 21:40:04,689 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.88 vs. limit=15.0 2023-11-24 21:40:06,111 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.66 vs. limit=15.0 2023-11-24 21:40:11,919 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=3005273.3333333335, ans=0.2 2023-11-24 21:40:14,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=3005273.3333333335, ans=0.5 2023-11-24 21:40:16,492 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450800 2023-11-24 21:40:44,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3005473.3333333335, ans=0.125 2023-11-24 21:40:57,906 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 5950, loss[loss=0.06912, simple_loss=0.08881, pruned_loss=0.01485, audio_tagging_loss=0.009865, over 13829.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09189, pruned_loss=0.01315, audio_tagging_loss=0.008706, over 3052974.15 frames. ], batch size: 52, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:41:02,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=3005540.0, ans=0.05 2023-11-24 21:41:15,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3005606.6666666665, ans=0.1 2023-11-24 21:41:19,142 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450850 2023-11-24 21:41:20,471 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3005606.6666666665, ans=0.0 2023-11-24 21:41:31,422 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.501e+01 8.441e+01 9.152e+01 9.875e+01 1.330e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 21:41:52,833 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0 2023-11-24 21:41:59,105 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6000, loss[loss=0.08265, simple_loss=0.116, pruned_loss=0.01536, audio_tagging_loss=0.009319, over 15455.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09195, pruned_loss=0.01312, audio_tagging_loss=0.008617, over 3047976.06 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:41:59,105 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 21:42:36,711 INFO [zipformer.py:1873] (3/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8119, 4.9874, 5.0347, 4.8790], device='cuda:3') 2023-11-24 21:42:40,941 INFO [train_asr.py:1253] (3/4) Epoch 38, validation: loss=0.05788, simple_loss=0.05074, pruned_loss=0.005119, audio_tagging_loss=0.02739, over 4681554.00 frames. 2023-11-24 21:42:40,942 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 21:43:01,682 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450900 2023-11-24 21:43:01,774 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=3005940.0, ans=0.125 2023-11-24 21:43:05,222 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 21:43:05,366 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=3006006.6666666665, ans=0.2 2023-11-24 21:43:06,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3006006.6666666665, ans=0.125 2023-11-24 21:43:16,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3006073.3333333335, ans=0.125 2023-11-24 21:43:23,364 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 21:43:24,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3006073.3333333335, ans=0.125 2023-11-24 21:43:42,802 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6050, loss[loss=0.09471, simple_loss=0.12, pruned_loss=0.02677, audio_tagging_loss=0.007926, over 14753.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09241, pruned_loss=0.01326, audio_tagging_loss=0.008642, over 3046896.09 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:44:04,052 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 450950 2023-11-24 21:44:17,480 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.516e+01 8.966e+01 9.859e+01 1.258e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 21:44:23,925 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3006406.6666666665, ans=0.125 2023-11-24 21:44:28,267 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.74 vs. limit=10.0 2023-11-24 21:44:44,319 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6100, loss[loss=0.06063, simple_loss=0.07035, pruned_loss=0.01117, audio_tagging_loss=0.01428, over 13993.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.0913, pruned_loss=0.01303, audio_tagging_loss=0.008738, over 3042518.89 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:44:56,000 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.75 vs. limit=15.0 2023-11-24 21:45:00,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=3006606.6666666665, ans=0.125 2023-11-24 21:45:02,815 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3006606.6666666665, ans=0.125 2023-11-24 21:45:06,042 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451000 2023-11-24 21:45:47,790 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6150, loss[loss=0.07069, simple_loss=0.09253, pruned_loss=0.01532, audio_tagging_loss=0.00911, over 16284.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09065, pruned_loss=0.01306, audio_tagging_loss=0.008772, over 3044098.80 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:46:08,493 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451050 2023-11-24 21:46:21,899 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.214e+01 8.834e+01 9.340e+01 1.011e+02 1.357e+02, threshold=1.868e+02, percent-clipped=0.0 2023-11-24 21:46:33,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0 2023-11-24 21:46:35,471 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.28 vs. limit=10.0 2023-11-24 21:46:49,584 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6200, loss[loss=0.05785, simple_loss=0.08433, pruned_loss=0.006886, audio_tagging_loss=0.008802, over 14922.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09073, pruned_loss=0.0131, audio_tagging_loss=0.008855, over 3044018.65 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:47:07,048 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3007273.3333333335, ans=0.1 2023-11-24 21:47:10,332 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451100 2023-11-24 21:47:23,129 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.88 vs. limit=22.5 2023-11-24 21:47:36,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=3007406.6666666665, ans=10.0 2023-11-24 21:47:47,553 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3007473.3333333335, ans=0.125 2023-11-24 21:47:50,826 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6250, loss[loss=0.06296, simple_loss=0.08174, pruned_loss=0.0113, audio_tagging_loss=0.01079, over 14771.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.08995, pruned_loss=0.01307, audio_tagging_loss=0.008999, over 3037706.15 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:48:01,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=3007540.0, ans=0.125 2023-11-24 21:48:02,722 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0 2023-11-24 21:48:12,977 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451150 2023-11-24 21:48:14,206 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=3007606.6666666665, ans=0.125 2023-11-24 21:48:26,314 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 8.537e+01 9.157e+01 9.825e+01 1.470e+02, threshold=1.831e+02, percent-clipped=0.0 2023-11-24 21:48:27,740 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3007740.0, ans=0.1 2023-11-24 21:48:48,345 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=3007806.6666666665, ans=0.0 2023-11-24 21:48:52,351 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3007873.3333333335, ans=0.125 2023-11-24 21:48:53,255 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6300, loss[loss=0.07545, simple_loss=0.09886, pruned_loss=0.01731, audio_tagging_loss=0.008716, over 15577.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.08942, pruned_loss=0.01298, audio_tagging_loss=0.009053, over 3034756.94 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:48:56,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3007873.3333333335, ans=0.1 2023-11-24 21:49:01,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3007873.3333333335, ans=0.1 2023-11-24 21:49:05,175 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.60 vs. limit=15.0 2023-11-24 21:49:11,454 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=3007940.0, ans=0.2 2023-11-24 21:49:14,670 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451200 2023-11-24 21:49:21,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3008006.6666666665, ans=0.125 2023-11-24 21:49:30,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3008073.3333333335, ans=0.125 2023-11-24 21:49:31,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3008073.3333333335, ans=0.1 2023-11-24 21:49:31,157 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=3008073.3333333335, ans=0.125 2023-11-24 21:49:42,197 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3008140.0, ans=0.0 2023-11-24 21:49:44,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3008140.0, ans=0.1 2023-11-24 21:49:54,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3008206.6666666665, ans=0.125 2023-11-24 21:49:56,229 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6350, loss[loss=0.07469, simple_loss=0.1038, pruned_loss=0.01459, audio_tagging_loss=0.008215, over 14565.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09002, pruned_loss=0.01297, audio_tagging_loss=0.009028, over 3034397.35 frames. ], batch size: 52, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 21:50:10,249 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.33 vs. limit=15.0 2023-11-24 21:50:17,222 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451250 2023-11-24 21:50:21,053 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3008340.0, ans=0.125 2023-11-24 21:50:22,262 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3008340.0, ans=0.0 2023-11-24 21:50:23,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3008340.0, ans=0.125 2023-11-24 21:50:30,742 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.435e+01 8.876e+01 9.772e+01 1.151e+02, threshold=1.775e+02, percent-clipped=0.0 2023-11-24 21:50:33,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3008406.6666666665, ans=0.125 2023-11-24 21:50:40,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3008406.6666666665, ans=0.0 2023-11-24 21:50:57,815 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6400, loss[loss=0.06077, simple_loss=0.08655, pruned_loss=0.009562, audio_tagging_loss=0.007929, over 16266.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09027, pruned_loss=0.01276, audio_tagging_loss=0.009013, over 3041410.08 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:50:58,031 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3008540.0, ans=0.125 2023-11-24 21:51:01,073 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.85 vs. limit=15.0 2023-11-24 21:51:09,413 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3008606.6666666665, ans=0.0 2023-11-24 21:51:19,010 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451300 2023-11-24 21:51:21,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3008606.6666666665, ans=0.125 2023-11-24 21:51:28,016 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=3008673.3333333335, ans=0.0 2023-11-24 21:51:41,568 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.99 vs. limit=15.0 2023-11-24 21:51:50,292 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=3008806.6666666665, ans=0.125 2023-11-24 21:51:52,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3008806.6666666665, ans=0.0 2023-11-24 21:51:53,091 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=8.35 vs. limit=8.0 2023-11-24 21:51:58,516 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3008873.3333333335, ans=0.1 2023-11-24 21:51:59,918 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6450, loss[loss=0.05668, simple_loss=0.07472, pruned_loss=0.009883, audio_tagging_loss=0.009439, over 15213.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.09026, pruned_loss=0.01265, audio_tagging_loss=0.009076, over 3037411.64 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:52:05,130 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=15.67 vs. limit=15.0 2023-11-24 21:52:17,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3008940.0, ans=0.1 2023-11-24 21:52:21,360 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451350 2023-11-24 21:52:33,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3009006.6666666665, ans=0.125 2023-11-24 21:52:34,133 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.612e+01 9.313e+01 1.021e+02 1.233e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 21:53:01,739 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6500, loss[loss=0.04804, simple_loss=0.05713, pruned_loss=0.006485, audio_tagging_loss=0.01298, over 14716.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09096, pruned_loss=0.01266, audio_tagging_loss=0.008935, over 3037632.73 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:53:17,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3009273.3333333335, ans=0.1 2023-11-24 21:53:22,264 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.92 vs. limit=6.0 2023-11-24 21:53:22,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451400 2023-11-24 21:53:51,345 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 21:53:52,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3009473.3333333335, ans=0.0 2023-11-24 21:53:55,606 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 21:54:04,596 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6550, loss[loss=0.06079, simple_loss=0.09142, pruned_loss=0.009146, audio_tagging_loss=0.005936, over 15602.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09123, pruned_loss=0.01273, audio_tagging_loss=0.00889, over 3048924.34 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:54:16,030 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=3009606.6666666665, ans=0.07 2023-11-24 21:54:22,692 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3009606.6666666665, ans=0.0 2023-11-24 21:54:25,941 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451450 2023-11-24 21:54:27,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3009606.6666666665, ans=0.0 2023-11-24 21:54:39,837 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.039e+01 8.549e+01 9.293e+01 1.004e+02 1.709e+02, threshold=1.859e+02, percent-clipped=0.0 2023-11-24 21:54:47,259 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 21:55:06,536 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6600, loss[loss=0.07927, simple_loss=0.11, pruned_loss=0.01443, audio_tagging_loss=0.009833, over 14648.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09143, pruned_loss=0.01287, audio_tagging_loss=0.008781, over 3050479.84 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:55:20,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3009940.0, ans=0.0 2023-11-24 21:55:23,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3009940.0, ans=0.1 2023-11-24 21:55:28,511 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451500 2023-11-24 21:55:34,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=3010006.6666666665, ans=0.0 2023-11-24 21:55:39,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3010006.6666666665, ans=0.0 2023-11-24 21:55:47,209 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=22.5 2023-11-24 21:55:50,460 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=3010073.3333333335, ans=0.0 2023-11-24 21:55:55,664 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.63 vs. limit=15.0 2023-11-24 21:56:08,547 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6650, loss[loss=0.07525, simple_loss=0.1011, pruned_loss=0.0139, audio_tagging_loss=0.01082, over 15346.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09138, pruned_loss=0.01286, audio_tagging_loss=0.008754, over 3050280.99 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:56:28,769 INFO [scaling.py:1022] (3/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.64 vs. limit=5.0 2023-11-24 21:56:30,299 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451550 2023-11-24 21:56:36,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3010340.0, ans=0.125 2023-11-24 21:56:44,238 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.352e+01 8.506e+01 9.140e+01 9.928e+01 1.246e+02, threshold=1.828e+02, percent-clipped=0.0 2023-11-24 21:56:45,894 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=3010406.6666666665, ans=0.2 2023-11-24 21:56:46,003 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=3010406.6666666665, ans=15.0 2023-11-24 21:56:48,504 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.42 vs. limit=15.0 2023-11-24 21:56:50,577 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3010406.6666666665, ans=0.125 2023-11-24 21:56:54,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=3010406.6666666665, ans=0.2 2023-11-24 21:57:11,184 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6700, loss[loss=0.07069, simple_loss=0.09912, pruned_loss=0.01416, audio_tagging_loss=0.006972, over 14026.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.09055, pruned_loss=0.01272, audio_tagging_loss=0.00872, over 3046494.05 frames. ], batch size: 51, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:57:22,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3010606.6666666665, ans=0.1 2023-11-24 21:57:22,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3010606.6666666665, ans=0.125 2023-11-24 21:57:27,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3010606.6666666665, ans=0.125 2023-11-24 21:57:28,563 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3010606.6666666665, ans=0.125 2023-11-24 21:57:32,713 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451600 2023-11-24 21:57:41,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=3010673.3333333335, ans=0.125 2023-11-24 21:57:52,780 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=3010740.0, ans=15.0 2023-11-24 21:57:58,738 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.58 vs. limit=12.0 2023-11-24 21:58:07,926 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.96 vs. limit=12.0 2023-11-24 21:58:12,756 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=3010873.3333333335, ans=0.025 2023-11-24 21:58:13,691 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6750, loss[loss=0.05559, simple_loss=0.06996, pruned_loss=0.01004, audio_tagging_loss=0.01056, over 14626.00 frames. ], tot_loss[loss=0.06633, simple_loss=0.08991, pruned_loss=0.01265, audio_tagging_loss=0.008732, over 3041562.03 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:58:22,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=3010873.3333333335, ans=0.2 2023-11-24 21:58:26,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3010940.0, ans=0.125 2023-11-24 21:58:30,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3010940.0, ans=0.125 2023-11-24 21:58:34,505 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451650 2023-11-24 21:58:34,802 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3010940.0, ans=0.125 2023-11-24 21:58:44,520 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.26 vs. limit=15.0 2023-11-24 21:58:49,689 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.221e+01 8.416e+01 8.949e+01 9.766e+01 1.528e+02, threshold=1.790e+02, percent-clipped=0.0 2023-11-24 21:59:15,545 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6800, loss[loss=0.04444, simple_loss=0.05607, pruned_loss=0.007549, audio_tagging_loss=0.008857, over 16964.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.0901, pruned_loss=0.01288, audio_tagging_loss=0.008655, over 3044583.24 frames. ], batch size: 67, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 21:59:19,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3011206.6666666665, ans=0.125 2023-11-24 21:59:35,665 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3011273.3333333335, ans=0.125 2023-11-24 21:59:36,663 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451700 2023-11-24 22:00:18,169 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6850, loss[loss=0.05314, simple_loss=0.0641, pruned_loss=0.01115, audio_tagging_loss=0.009943, over 14654.00 frames. ], tot_loss[loss=0.06624, simple_loss=0.08965, pruned_loss=0.01275, audio_tagging_loss=0.008667, over 3046025.95 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:00:23,013 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3011540.0, ans=0.0 2023-11-24 22:00:27,038 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.53 vs. limit=10.0 2023-11-24 22:00:35,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=3011606.6666666665, ans=0.025 2023-11-24 22:00:35,807 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.32 vs. limit=15.0 2023-11-24 22:00:39,364 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451750 2023-11-24 22:00:44,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.44 vs. limit=15.0 2023-11-24 22:00:50,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=3011673.3333333335, ans=0.125 2023-11-24 22:00:54,774 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.792e+01 9.209e+01 9.855e+01 1.264e+02, threshold=1.842e+02, percent-clipped=0.0 2023-11-24 22:00:57,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3011740.0, ans=0.125 2023-11-24 22:00:57,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3011740.0, ans=0.125 2023-11-24 22:00:59,478 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.36 vs. limit=15.0 2023-11-24 22:01:03,804 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=3011740.0, ans=0.04949747468305833 2023-11-24 22:01:03,812 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3011740.0, ans=0.125 2023-11-24 22:01:12,005 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.51 vs. limit=15.0 2023-11-24 22:01:19,656 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6900, loss[loss=0.05847, simple_loss=0.07441, pruned_loss=0.009855, audio_tagging_loss=0.01141, over 15029.00 frames. ], tot_loss[loss=0.06643, simple_loss=0.08986, pruned_loss=0.01273, audio_tagging_loss=0.008772, over 3048729.09 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:01:21,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3011873.3333333335, ans=0.125 2023-11-24 22:01:22,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3011873.3333333335, ans=0.125 2023-11-24 22:01:24,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=3011873.3333333335, ans=0.125 2023-11-24 22:01:41,160 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451800 2023-11-24 22:01:54,696 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=3012006.6666666665, ans=0.95 2023-11-24 22:01:57,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=3012073.3333333335, ans=0.0 2023-11-24 22:02:00,611 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:02:04,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3012073.3333333335, ans=0.125 2023-11-24 22:02:06,115 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 22:02:19,760 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:02:22,854 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 6950, loss[loss=0.06648, simple_loss=0.08376, pruned_loss=0.01341, audio_tagging_loss=0.01118, over 14191.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.09036, pruned_loss=0.01282, audio_tagging_loss=0.00876, over 3044941.23 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:02:30,326 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3012206.6666666665, ans=0.125 2023-11-24 22:02:44,287 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451850 2023-11-24 22:02:59,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3012406.6666666665, ans=0.125 2023-11-24 22:03:00,155 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.498e+01 8.404e+01 9.015e+01 9.725e+01 1.456e+02, threshold=1.803e+02, percent-clipped=0.0 2023-11-24 22:03:24,969 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7000, loss[loss=0.0775, simple_loss=0.1013, pruned_loss=0.01944, audio_tagging_loss=0.007411, over 15001.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09044, pruned_loss=0.01282, audio_tagging_loss=0.008794, over 3046863.71 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:03:25,123 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=3012540.0, ans=0.125 2023-11-24 22:03:33,982 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=3012540.0, ans=0.0 2023-11-24 22:03:41,283 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3012606.6666666665, ans=0.0 2023-11-24 22:03:46,407 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451900 2023-11-24 22:03:47,115 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.21 vs. limit=15.0 2023-11-24 22:04:07,536 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.19 vs. limit=15.0 2023-11-24 22:04:25,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3012806.6666666665, ans=0.1 2023-11-24 22:04:27,262 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7050, loss[loss=0.05663, simple_loss=0.07249, pruned_loss=0.009825, audio_tagging_loss=0.01056, over 13805.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09034, pruned_loss=0.01278, audio_tagging_loss=0.008875, over 3040856.85 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:04:37,525 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.78 vs. limit=15.0 2023-11-24 22:04:48,498 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 451950 2023-11-24 22:04:52,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.12 vs. limit=15.0 2023-11-24 22:04:57,425 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3013006.6666666665, ans=0.125 2023-11-24 22:05:02,606 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.48 vs. limit=15.0 2023-11-24 22:05:04,199 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.687e+01 9.290e+01 1.031e+02 1.279e+02, threshold=1.858e+02, percent-clipped=0.0 2023-11-24 22:05:10,004 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=3013073.3333333335, ans=0.2 2023-11-24 22:05:13,291 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3013073.3333333335, ans=0.125 2023-11-24 22:05:29,449 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7100, loss[loss=0.05808, simple_loss=0.0765, pruned_loss=0.01169, audio_tagging_loss=0.008137, over 15171.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.09005, pruned_loss=0.01268, audio_tagging_loss=0.008894, over 3034569.54 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:05:48,038 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3013273.3333333335, ans=0.125 2023-11-24 22:05:50,281 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452000 2023-11-24 22:06:17,695 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=3013406.6666666665, ans=0.2 2023-11-24 22:06:17,720 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=3013406.6666666665, ans=0.0 2023-11-24 22:06:33,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=3013473.3333333335, ans=0.2 2023-11-24 22:06:35,706 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7150, loss[loss=0.07565, simple_loss=0.09966, pruned_loss=0.01622, audio_tagging_loss=0.009607, over 15226.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09047, pruned_loss=0.01272, audio_tagging_loss=0.008863, over 3036884.51 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:06:48,290 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3013606.6666666665, ans=0.125 2023-11-24 22:06:56,910 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452050 2023-11-24 22:06:59,557 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=3013673.3333333335, ans=0.0 2023-11-24 22:07:09,782 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:07:12,066 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:07:12,928 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.770e+01 9.367e+01 1.013e+02 1.404e+02, threshold=1.873e+02, percent-clipped=0.0 2023-11-24 22:07:15,746 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3013740.0, ans=0.1 2023-11-24 22:07:29,256 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=3013806.6666666665, ans=0.125 2023-11-24 22:07:37,809 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7200, loss[loss=0.06176, simple_loss=0.08241, pruned_loss=0.01206, audio_tagging_loss=0.008492, over 15488.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.09004, pruned_loss=0.01277, audio_tagging_loss=0.008984, over 3035110.97 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 22:07:58,906 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452100 2023-11-24 22:08:04,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=3014006.6666666665, ans=0.0 2023-11-24 22:08:10,830 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.32 vs. limit=15.0 2023-11-24 22:08:15,299 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3014073.3333333335, ans=0.125 2023-11-24 22:08:38,666 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.77 vs. limit=15.0 2023-11-24 22:08:40,397 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7250, loss[loss=0.06922, simple_loss=0.106, pruned_loss=0.009237, audio_tagging_loss=0.006965, over 16050.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.09028, pruned_loss=0.01273, audio_tagging_loss=0.009084, over 3045359.32 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 22:08:40,633 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3014206.6666666665, ans=0.125 2023-11-24 22:08:51,361 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=3014273.3333333335, ans=0.2 2023-11-24 22:08:53,809 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=3014273.3333333335, ans=0.0 2023-11-24 22:09:01,068 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452150 2023-11-24 22:09:01,217 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3014273.3333333335, ans=0.125 2023-11-24 22:09:18,539 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.621e+01 8.553e+01 9.150e+01 9.601e+01 1.170e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 22:09:37,569 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.70 vs. limit=15.0 2023-11-24 22:09:38,687 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=3014473.3333333335, ans=0.2 2023-11-24 22:09:41,864 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7300, loss[loss=0.07867, simple_loss=0.1052, pruned_loss=0.01645, audio_tagging_loss=0.009621, over 15866.00 frames. ], tot_loss[loss=0.06629, simple_loss=0.08946, pruned_loss=0.01254, audio_tagging_loss=0.009025, over 3048642.66 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:09:42,099 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:10:02,579 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452200 2023-11-24 22:10:07,394 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=3014673.3333333335, ans=0.125 2023-11-24 22:10:17,466 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3014673.3333333335, ans=0.1 2023-11-24 22:10:28,002 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3014740.0, ans=0.125 2023-11-24 22:10:28,081 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3014740.0, ans=0.125 2023-11-24 22:10:43,883 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7350, loss[loss=0.06824, simple_loss=0.09001, pruned_loss=0.01399, audio_tagging_loss=0.009244, over 16001.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08976, pruned_loss=0.0126, audio_tagging_loss=0.008827, over 3054132.01 frames. ], batch size: 63, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:10:45,305 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=3014873.3333333335, ans=0.125 2023-11-24 22:11:00,944 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3014940.0, ans=0.125 2023-11-24 22:11:05,660 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452250 2023-11-24 22:11:16,623 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3015006.6666666665, ans=0.125 2023-11-24 22:11:22,652 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.188e+01 8.406e+01 8.894e+01 9.896e+01 1.273e+02, threshold=1.779e+02, percent-clipped=0.0 2023-11-24 22:11:34,861 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=3015140.0, ans=0.0 2023-11-24 22:11:46,721 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7400, loss[loss=0.06933, simple_loss=0.09701, pruned_loss=0.01362, audio_tagging_loss=0.007205, over 14671.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.09055, pruned_loss=0.0127, audio_tagging_loss=0.00876, over 3057287.44 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:12:02,577 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.24 vs. limit=15.0 2023-11-24 22:12:07,431 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452300 2023-11-24 22:12:44,082 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0 2023-11-24 22:12:48,021 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7450, loss[loss=0.06711, simple_loss=0.09469, pruned_loss=0.009967, audio_tagging_loss=0.009805, over 15592.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.09026, pruned_loss=0.01274, audio_tagging_loss=0.00879, over 3055059.67 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:12:50,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3015540.0, ans=0.125 2023-11-24 22:13:00,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3015606.6666666665, ans=0.125 2023-11-24 22:13:04,277 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=3015606.6666666665, ans=0.2 2023-11-24 22:13:08,739 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452350 2023-11-24 22:13:08,864 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=3015606.6666666665, ans=0.2 2023-11-24 22:13:28,037 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.262e+01 8.690e+01 9.275e+01 1.001e+02 1.240e+02, threshold=1.855e+02, percent-clipped=0.0 2023-11-24 22:13:42,165 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.16 vs. limit=15.0 2023-11-24 22:13:44,097 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=3015806.6666666665, ans=0.0 2023-11-24 22:13:48,971 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=3015873.3333333335, ans=0.0 2023-11-24 22:13:49,773 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7500, loss[loss=0.07833, simple_loss=0.1001, pruned_loss=0.01642, audio_tagging_loss=0.01187, over 14955.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.09048, pruned_loss=0.01277, audio_tagging_loss=0.008848, over 3055258.51 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:13:49,950 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3015873.3333333335, ans=0.125 2023-11-24 22:13:53,618 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3015873.3333333335, ans=0.125 2023-11-24 22:13:55,352 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0 2023-11-24 22:14:11,884 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452400 2023-11-24 22:14:16,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.42 vs. limit=22.5 2023-11-24 22:14:34,446 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.09 vs. limit=15.0 2023-11-24 22:14:52,041 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7550, loss[loss=0.05236, simple_loss=0.0774, pruned_loss=0.00575, audio_tagging_loss=0.00791, over 15944.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.09037, pruned_loss=0.01278, audio_tagging_loss=0.008783, over 3057885.83 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:15:07,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=3016273.3333333335, ans=0.0 2023-11-24 22:15:14,480 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452450 2023-11-24 22:15:18,315 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3016340.0, ans=0.125 2023-11-24 22:15:22,847 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=3016340.0, ans=0.125 2023-11-24 22:15:26,632 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.73 vs. limit=15.0 2023-11-24 22:15:29,856 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3016406.6666666665, ans=0.125 2023-11-24 22:15:32,166 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.402e+01 8.333e+01 9.038e+01 9.761e+01 1.363e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 22:15:38,184 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.66 vs. limit=22.5 2023-11-24 22:15:50,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=3016473.3333333335, ans=0.125 2023-11-24 22:15:52,160 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3016473.3333333335, ans=0.125 2023-11-24 22:15:55,621 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7600, loss[loss=0.05932, simple_loss=0.07019, pruned_loss=0.01418, audio_tagging_loss=0.01005, over 15678.00 frames. ], tot_loss[loss=0.06627, simple_loss=0.08955, pruned_loss=0.01274, audio_tagging_loss=0.008758, over 3053080.29 frames. ], batch size: 62, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:15:57,213 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=3016540.0, ans=0.025 2023-11-24 22:16:00,464 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3016540.0, ans=0.125 2023-11-24 22:16:12,012 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3016606.6666666665, ans=0.0 2023-11-24 22:16:13,526 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.29 vs. limit=15.0 2023-11-24 22:16:16,333 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452500 2023-11-24 22:16:30,895 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=3016673.3333333335, ans=0.2 2023-11-24 22:16:32,126 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=3016740.0, ans=0.125 2023-11-24 22:16:36,803 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=3016740.0, ans=0.125 2023-11-24 22:16:57,864 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7650, loss[loss=0.05231, simple_loss=0.06792, pruned_loss=0.009603, audio_tagging_loss=0.008748, over 13574.00 frames. ], tot_loss[loss=0.06574, simple_loss=0.08904, pruned_loss=0.01249, audio_tagging_loss=0.008729, over 3051237.51 frames. ], batch size: 52, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:16:58,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3016873.3333333335, ans=0.0 2023-11-24 22:17:19,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452550 2023-11-24 22:17:37,944 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.417e+01 9.176e+01 1.003e+02 1.926e+02, threshold=1.835e+02, percent-clipped=1.0 2023-11-24 22:18:00,073 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7700, loss[loss=0.05492, simple_loss=0.07408, pruned_loss=0.01182, audio_tagging_loss=0.006058, over 15314.00 frames. ], tot_loss[loss=0.066, simple_loss=0.08952, pruned_loss=0.01252, audio_tagging_loss=0.008721, over 3044955.77 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:18:04,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3017206.6666666665, ans=0.125 2023-11-24 22:18:08,172 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=3017206.6666666665, ans=0.0 2023-11-24 22:18:09,349 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=3017206.6666666665, ans=0.125 2023-11-24 22:18:13,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3017273.3333333335, ans=0.0 2023-11-24 22:18:17,005 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3017273.3333333335, ans=0.1 2023-11-24 22:18:18,043 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=3017273.3333333335, ans=0.0 2023-11-24 22:18:21,486 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452600 2023-11-24 22:18:27,625 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.03 vs. limit=15.0 2023-11-24 22:18:34,453 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3017340.0, ans=0.125 2023-11-24 22:18:39,113 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3017406.6666666665, ans=0.1 2023-11-24 22:19:02,804 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7750, loss[loss=0.06812, simple_loss=0.09037, pruned_loss=0.01373, audio_tagging_loss=0.009205, over 15138.00 frames. ], tot_loss[loss=0.06596, simple_loss=0.08953, pruned_loss=0.01243, audio_tagging_loss=0.008766, over 3042665.35 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:19:07,266 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3017540.0, ans=0.125 2023-11-24 22:19:22,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3017606.6666666665, ans=0.0 2023-11-24 22:19:23,996 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452650 2023-11-24 22:19:27,800 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3017673.3333333335, ans=0.0 2023-11-24 22:19:42,995 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.572e+01 9.238e+01 9.882e+01 1.332e+02, threshold=1.848e+02, percent-clipped=0.0 2023-11-24 22:19:47,974 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=3017740.0, ans=0.0 2023-11-24 22:20:05,363 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7800, loss[loss=0.06136, simple_loss=0.08322, pruned_loss=0.01199, audio_tagging_loss=0.007765, over 13926.00 frames. ], tot_loss[loss=0.06639, simple_loss=0.09008, pruned_loss=0.01256, audio_tagging_loss=0.008792, over 3034144.63 frames. ], batch size: 52, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:20:15,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.94 vs. limit=15.0 2023-11-24 22:20:24,146 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=3017940.0, ans=0.07 2023-11-24 22:20:26,856 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452700 2023-11-24 22:20:28,173 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=3017940.0, ans=0.2 2023-11-24 22:20:48,125 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=3018073.3333333335, ans=0.05 2023-11-24 22:21:06,945 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7850, loss[loss=0.0673, simple_loss=0.09774, pruned_loss=0.01216, audio_tagging_loss=0.00627, over 14761.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09126, pruned_loss=0.01286, audio_tagging_loss=0.008859, over 3035509.63 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:21:28,264 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452750 2023-11-24 22:21:31,952 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=3018340.0, ans=0.125 2023-11-24 22:21:46,949 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.360e+01 8.800e+01 9.383e+01 9.995e+01 1.679e+02, threshold=1.877e+02, percent-clipped=0.0 2023-11-24 22:21:52,236 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3018406.6666666665, ans=0.125 2023-11-24 22:21:55,446 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=3018406.6666666665, ans=15.0 2023-11-24 22:22:07,928 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3018473.3333333335, ans=0.125 2023-11-24 22:22:09,987 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7900, loss[loss=0.05017, simple_loss=0.06987, pruned_loss=0.007134, audio_tagging_loss=0.008105, over 15939.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09105, pruned_loss=0.0129, audio_tagging_loss=0.008906, over 3045786.00 frames. ], batch size: 64, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:22:11,440 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3018540.0, ans=0.125 2023-11-24 22:22:22,933 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3018606.6666666665, ans=0.125 2023-11-24 22:22:25,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3018606.6666666665, ans=0.0 2023-11-24 22:22:27,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3018606.6666666665, ans=0.1 2023-11-24 22:22:30,876 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452800 2023-11-24 22:22:50,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3018740.0, ans=0.125 2023-11-24 22:22:53,523 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.15 vs. limit=15.0 2023-11-24 22:23:07,193 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3018806.6666666665, ans=0.125 2023-11-24 22:23:08,318 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=3018806.6666666665, ans=0.0 2023-11-24 22:23:12,335 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 7950, loss[loss=0.06056, simple_loss=0.07976, pruned_loss=0.009973, audio_tagging_loss=0.01071, over 14306.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09074, pruned_loss=0.01285, audio_tagging_loss=0.008954, over 3043649.43 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:23:25,694 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 22:23:33,464 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452850 2023-11-24 22:23:48,225 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=3019073.3333333335, ans=0.2 2023-11-24 22:23:52,665 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.512e+01 8.496e+01 9.034e+01 9.610e+01 1.973e+02, threshold=1.807e+02, percent-clipped=1.0 2023-11-24 22:24:01,251 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:24:13,979 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8000, loss[loss=0.07819, simple_loss=0.1111, pruned_loss=0.01465, audio_tagging_loss=0.007962, over 14958.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09044, pruned_loss=0.01283, audio_tagging_loss=0.00911, over 3048751.85 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:24:25,028 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=3019206.6666666665, ans=0.2 2023-11-24 22:24:35,519 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452900 2023-11-24 22:24:40,670 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3019340.0, ans=0.125 2023-11-24 22:24:49,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3019340.0, ans=0.125 2023-11-24 22:25:02,833 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3019473.3333333335, ans=0.0 2023-11-24 22:25:12,305 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:25:16,643 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8050, loss[loss=0.0846, simple_loss=0.1192, pruned_loss=0.01838, audio_tagging_loss=0.006635, over 15143.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.08992, pruned_loss=0.01272, audio_tagging_loss=0.009175, over 3041964.57 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:25:16,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=3019540.0, ans=0.07 2023-11-24 22:25:30,257 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=3019606.6666666665, ans=0.5 2023-11-24 22:25:38,410 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 452950 2023-11-24 22:25:41,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=3019673.3333333335, ans=0.0 2023-11-24 22:25:49,705 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.07 vs. limit=15.0 2023-11-24 22:25:53,555 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=3019740.0, ans=0.2 2023-11-24 22:25:59,476 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.598e+01 8.520e+01 9.232e+01 9.839e+01 1.227e+02, threshold=1.846e+02, percent-clipped=0.0 2023-11-24 22:26:11,327 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.08 vs. limit=15.0 2023-11-24 22:26:18,641 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8100, loss[loss=0.06952, simple_loss=0.09703, pruned_loss=0.01129, audio_tagging_loss=0.009715, over 15183.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09064, pruned_loss=0.01284, audio_tagging_loss=0.009015, over 3038828.77 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:26:40,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453000 2023-11-24 22:27:00,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3020073.3333333335, ans=0.0 2023-11-24 22:27:21,698 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8150, loss[loss=0.06497, simple_loss=0.09105, pruned_loss=0.01317, audio_tagging_loss=0.006271, over 14407.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09058, pruned_loss=0.01283, audio_tagging_loss=0.008906, over 3038114.18 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:27:30,242 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=6.89 vs. limit=15.0 2023-11-24 22:27:43,157 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453050 2023-11-24 22:27:50,576 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.19 vs. limit=15.0 2023-11-24 22:27:56,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3020340.0, ans=0.1 2023-11-24 22:27:59,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.07 vs. limit=22.5 2023-11-24 22:28:04,263 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.627e+01 9.105e+01 9.732e+01 1.211e+02, threshold=1.821e+02, percent-clipped=0.0 2023-11-24 22:28:11,556 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=3020473.3333333335, ans=10.0 2023-11-24 22:28:23,169 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 22:28:24,323 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8200, loss[loss=0.0803, simple_loss=0.107, pruned_loss=0.01859, audio_tagging_loss=0.008237, over 14952.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09089, pruned_loss=0.01299, audio_tagging_loss=0.008886, over 3045689.25 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:28:24,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3020540.0, ans=0.125 2023-11-24 22:28:26,412 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.62 vs. limit=12.0 2023-11-24 22:28:36,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3020606.6666666665, ans=0.125 2023-11-24 22:28:37,922 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3020606.6666666665, ans=0.1 2023-11-24 22:28:38,294 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0 2023-11-24 22:28:45,380 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453100 2023-11-24 22:28:45,478 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3020606.6666666665, ans=0.0 2023-11-24 22:29:26,348 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8250, loss[loss=0.08597, simple_loss=0.1245, pruned_loss=0.01663, audio_tagging_loss=0.007108, over 15875.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.0902, pruned_loss=0.01285, audio_tagging_loss=0.008915, over 3046012.91 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:29:47,860 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453150 2023-11-24 22:30:08,937 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.191e+01 8.522e+01 9.040e+01 9.801e+01 1.361e+02, threshold=1.808e+02, percent-clipped=0.0 2023-11-24 22:30:26,676 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=3021140.0, ans=0.2 2023-11-24 22:30:28,683 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8300, loss[loss=0.07474, simple_loss=0.1049, pruned_loss=0.01396, audio_tagging_loss=0.008336, over 14423.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.0909, pruned_loss=0.01294, audio_tagging_loss=0.008954, over 3054311.38 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:30:43,851 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3021273.3333333335, ans=0.125 2023-11-24 22:30:48,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3021273.3333333335, ans=0.1 2023-11-24 22:30:50,904 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453200 2023-11-24 22:30:51,036 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3021273.3333333335, ans=0.125 2023-11-24 22:30:59,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=3021340.0, ans=0.0 2023-11-24 22:31:17,992 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3021473.3333333335, ans=0.0 2023-11-24 22:31:19,065 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=3021473.3333333335, ans=0.035 2023-11-24 22:31:21,001 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=3021473.3333333335, ans=0.09899494936611666 2023-11-24 22:31:32,471 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8350, loss[loss=0.06317, simple_loss=0.08012, pruned_loss=0.01333, audio_tagging_loss=0.009777, over 15075.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09061, pruned_loss=0.01276, audio_tagging_loss=0.008904, over 3054582.18 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0 2023-11-24 22:31:32,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3021540.0, ans=0.125 2023-11-24 22:31:32,738 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3021540.0, ans=0.125 2023-11-24 22:31:52,705 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453250 2023-11-24 22:31:57,094 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3021673.3333333335, ans=0.125 2023-11-24 22:32:09,575 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=3021740.0, ans=0.125 2023-11-24 22:32:12,709 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.11 vs. limit=15.0 2023-11-24 22:32:14,493 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.506e+01 8.493e+01 9.150e+01 9.803e+01 1.531e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 22:32:29,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3021806.6666666665, ans=0.125 2023-11-24 22:32:30,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=3021806.6666666665, ans=0.2 2023-11-24 22:32:34,032 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8400, loss[loss=0.07727, simple_loss=0.1025, pruned_loss=0.01789, audio_tagging_loss=0.008118, over 15173.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08974, pruned_loss=0.01258, audio_tagging_loss=0.008863, over 3045975.32 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:32:54,793 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453300 2023-11-24 22:32:57,857 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=3022006.6666666665, ans=0.035 2023-11-24 22:33:01,888 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=3022006.6666666665, ans=0.0 2023-11-24 22:33:08,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3022006.6666666665, ans=0.125 2023-11-24 22:33:09,879 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=3022006.6666666665, ans=0.125 2023-11-24 22:33:19,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3022073.3333333335, ans=0.0 2023-11-24 22:33:36,195 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8450, loss[loss=0.07467, simple_loss=0.1062, pruned_loss=0.01573, audio_tagging_loss=0.00582, over 14271.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08974, pruned_loss=0.0127, audio_tagging_loss=0.008879, over 3047208.29 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:33:38,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3022206.6666666665, ans=0.1 2023-11-24 22:33:42,499 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3022206.6666666665, ans=0.125 2023-11-24 22:33:57,129 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3022273.3333333335, ans=0.1 2023-11-24 22:33:58,056 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453350 2023-11-24 22:34:07,828 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3022340.0, ans=0.125 2023-11-24 22:34:18,674 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.351e+01 8.708e+01 9.231e+01 1.022e+02 1.410e+02, threshold=1.846e+02, percent-clipped=0.0 2023-11-24 22:34:25,915 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=3022473.3333333335, ans=0.035 2023-11-24 22:34:30,732 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=3022473.3333333335, ans=0.0 2023-11-24 22:34:33,778 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=3022473.3333333335, ans=10.0 2023-11-24 22:34:38,772 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8500, loss[loss=0.07665, simple_loss=0.1108, pruned_loss=0.01178, audio_tagging_loss=0.009463, over 15539.00 frames. ], tot_loss[loss=0.06638, simple_loss=0.08974, pruned_loss=0.01266, audio_tagging_loss=0.00885, over 3050392.72 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:34:59,580 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453400 2023-11-24 22:35:06,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=3022673.3333333335, ans=0.125 2023-11-24 22:35:07,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3022673.3333333335, ans=0.125 2023-11-24 22:35:32,533 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0 2023-11-24 22:35:41,410 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8550, loss[loss=0.08205, simple_loss=0.1054, pruned_loss=0.01815, audio_tagging_loss=0.01119, over 15662.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.09031, pruned_loss=0.01276, audio_tagging_loss=0.008951, over 3051881.12 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:35:50,029 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3022873.3333333335, ans=0.125 2023-11-24 22:35:52,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3022940.0, ans=0.0 2023-11-24 22:35:56,531 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=3022940.0, ans=0.125 2023-11-24 22:36:02,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453450 2023-11-24 22:36:23,754 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.604e+01 8.608e+01 9.249e+01 9.944e+01 1.269e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 22:36:43,372 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8600, loss[loss=0.05348, simple_loss=0.08036, pruned_loss=0.004852, audio_tagging_loss=0.00845, over 14124.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09086, pruned_loss=0.01271, audio_tagging_loss=0.008974, over 3056065.64 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:36:57,599 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3023273.3333333335, ans=0.125 2023-11-24 22:37:05,760 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453500 2023-11-24 22:37:22,668 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3023406.6666666665, ans=0.125 2023-11-24 22:37:31,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=3023406.6666666665, ans=0.2 2023-11-24 22:37:45,786 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8650, loss[loss=0.05368, simple_loss=0.07082, pruned_loss=0.008222, audio_tagging_loss=0.01004, over 14899.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09131, pruned_loss=0.01276, audio_tagging_loss=0.009037, over 3053166.63 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:37:56,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3023540.0, ans=0.0 2023-11-24 22:38:07,182 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453550 2023-11-24 22:38:21,382 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0 2023-11-24 22:38:27,884 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.604e+01 9.306e+01 1.022e+02 1.267e+02, threshold=1.861e+02, percent-clipped=0.0 2023-11-24 22:38:28,234 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=3023740.0, ans=0.125 2023-11-24 22:38:32,287 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3023740.0, ans=0.125 2023-11-24 22:38:37,588 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=3023806.6666666665, ans=0.95 2023-11-24 22:38:48,891 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8700, loss[loss=0.05995, simple_loss=0.06714, pruned_loss=0.01455, audio_tagging_loss=0.01183, over 16857.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09153, pruned_loss=0.01282, audio_tagging_loss=0.008967, over 3051264.83 frames. ], batch size: 66, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:38:55,087 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:39:09,863 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453600 2023-11-24 22:39:24,625 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3024006.6666666665, ans=0.0 2023-11-24 22:39:26,204 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.36 vs. limit=22.5 2023-11-24 22:39:51,315 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8750, loss[loss=0.08486, simple_loss=0.1181, pruned_loss=0.01773, audio_tagging_loss=0.008061, over 15762.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.0916, pruned_loss=0.01267, audio_tagging_loss=0.009074, over 3050491.60 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:40:02,506 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=3024273.3333333335, ans=0.5 2023-11-24 22:40:12,868 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453650 2023-11-24 22:40:22,537 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0 2023-11-24 22:40:33,323 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.979e+01 9.554e+01 1.051e+02 1.529e+02, threshold=1.911e+02, percent-clipped=0.0 2023-11-24 22:40:34,913 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=3024406.6666666665, ans=0.2 2023-11-24 22:40:50,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=3024473.3333333335, ans=0.0 2023-11-24 22:40:52,685 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8800, loss[loss=0.05839, simple_loss=0.07614, pruned_loss=0.009133, audio_tagging_loss=0.01118, over 15408.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09237, pruned_loss=0.01299, audio_tagging_loss=0.009058, over 3047546.32 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 22:41:00,134 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3024540.0, ans=0.1 2023-11-24 22:41:02,494 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=3024540.0, ans=0.125 2023-11-24 22:41:14,648 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453700 2023-11-24 22:41:15,997 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=3024606.6666666665, ans=0.0 2023-11-24 22:41:37,447 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.00 vs. limit=22.5 2023-11-24 22:41:50,202 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.78 vs. limit=22.5 2023-11-24 22:41:52,845 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.51 vs. limit=15.0 2023-11-24 22:41:56,237 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8850, loss[loss=0.07263, simple_loss=0.09933, pruned_loss=0.01337, audio_tagging_loss=0.009596, over 15456.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09216, pruned_loss=0.01283, audio_tagging_loss=0.009028, over 3050210.60 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 22:42:05,733 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 22:42:14,624 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:42:16,885 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453750 2023-11-24 22:42:19,889 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.50 vs. limit=15.0 2023-11-24 22:42:22,862 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3025006.6666666665, ans=0.0 2023-11-24 22:42:33,390 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=3025073.3333333335, ans=10.0 2023-11-24 22:42:37,812 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.566e+01 8.570e+01 9.101e+01 9.826e+01 1.259e+02, threshold=1.820e+02, percent-clipped=0.0 2023-11-24 22:42:42,912 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3025073.3333333335, ans=0.125 2023-11-24 22:42:43,055 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3025073.3333333335, ans=0.125 2023-11-24 22:42:44,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3025140.0, ans=0.1 2023-11-24 22:42:51,949 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=3025140.0, ans=0.09899494936611666 2023-11-24 22:42:54,067 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=3025140.0, ans=0.125 2023-11-24 22:42:57,358 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8900, loss[loss=0.06368, simple_loss=0.09063, pruned_loss=0.009842, audio_tagging_loss=0.008521, over 15314.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09175, pruned_loss=0.01261, audio_tagging_loss=0.008949, over 3051552.45 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 22:43:02,104 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.91 vs. limit=22.5 2023-11-24 22:43:18,689 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453800 2023-11-24 22:43:43,972 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=3025406.6666666665, ans=0.0 2023-11-24 22:43:59,732 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 8950, loss[loss=0.07012, simple_loss=0.09688, pruned_loss=0.0142, audio_tagging_loss=0.007481, over 16019.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09105, pruned_loss=0.0126, audio_tagging_loss=0.008878, over 3047463.83 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0 2023-11-24 22:44:21,889 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453850 2023-11-24 22:44:22,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=3025606.6666666665, ans=0.1 2023-11-24 22:44:27,907 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=3025673.3333333335, ans=0.0 2023-11-24 22:44:32,569 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=3025673.3333333335, ans=0.2 2023-11-24 22:44:35,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=3025673.3333333335, ans=0.2 2023-11-24 22:44:39,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=3025740.0, ans=0.0 2023-11-24 22:44:43,569 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.683e+01 8.565e+01 9.194e+01 9.956e+01 1.408e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 22:45:02,694 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9000, loss[loss=0.04919, simple_loss=0.06077, pruned_loss=0.009193, audio_tagging_loss=0.009613, over 14833.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09149, pruned_loss=0.01272, audio_tagging_loss=0.008805, over 3050192.47 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0 2023-11-24 22:45:02,695 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 22:45:46,033 INFO [train_asr.py:1253] (3/4) Epoch 38, validation: loss=0.05855, simple_loss=0.05069, pruned_loss=0.005085, audio_tagging_loss=0.02812, over 4681554.00 frames. 2023-11-24 22:45:46,034 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 22:46:07,174 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453900 2023-11-24 22:46:32,284 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3026073.3333333335, ans=0.0 2023-11-24 22:46:44,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3026140.0, ans=0.125 2023-11-24 22:46:47,999 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9050, loss[loss=0.06087, simple_loss=0.08006, pruned_loss=0.01227, audio_tagging_loss=0.008576, over 15296.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09149, pruned_loss=0.01284, audio_tagging_loss=0.008794, over 3052007.90 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:46:49,954 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.39 vs. limit=6.0 2023-11-24 22:46:50,525 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=3026206.6666666665, ans=0.2 2023-11-24 22:46:58,343 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=3026206.6666666665, ans=0.2 2023-11-24 22:46:59,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3026273.3333333335, ans=0.125 2023-11-24 22:47:08,728 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 453950 2023-11-24 22:47:13,091 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3026340.0, ans=0.1 2023-11-24 22:47:15,376 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3026340.0, ans=0.0 2023-11-24 22:47:16,579 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=3026340.0, ans=0.125 2023-11-24 22:47:30,941 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.553e+01 8.549e+01 9.034e+01 9.764e+01 1.451e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 22:47:34,285 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3026406.6666666665, ans=0.125 2023-11-24 22:47:43,182 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=3026473.3333333335, ans=0.07 2023-11-24 22:47:44,551 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3026473.3333333335, ans=0.125 2023-11-24 22:47:45,621 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=3026473.3333333335, ans=0.0 2023-11-24 22:47:50,110 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9100, loss[loss=0.04489, simple_loss=0.05759, pruned_loss=0.007733, audio_tagging_loss=0.008361, over 15603.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09097, pruned_loss=0.01275, audio_tagging_loss=0.008691, over 3049460.25 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:48:09,629 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3026606.6666666665, ans=0.0 2023-11-24 22:48:11,699 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454000 2023-11-24 22:48:14,901 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.37 vs. limit=15.0 2023-11-24 22:48:18,214 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3026673.3333333335, ans=0.125 2023-11-24 22:48:22,583 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:48:22,865 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.71 vs. limit=15.0 2023-11-24 22:48:51,854 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=3026873.3333333335, ans=0.0 2023-11-24 22:48:53,412 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9150, loss[loss=0.06742, simple_loss=0.0923, pruned_loss=0.01239, audio_tagging_loss=0.008879, over 14219.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09133, pruned_loss=0.01281, audio_tagging_loss=0.008611, over 3052772.71 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:48:54,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=3026873.3333333335, ans=0.125 2023-11-24 22:49:03,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3026873.3333333335, ans=0.1 2023-11-24 22:49:14,968 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454050 2023-11-24 22:49:17,513 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3027006.6666666665, ans=0.0 2023-11-24 22:49:36,303 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.421e+01 9.061e+01 9.734e+01 1.251e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 22:49:40,039 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.52 vs. limit=22.5 2023-11-24 22:49:45,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3027140.0, ans=0.125 2023-11-24 22:49:54,152 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3027206.6666666665, ans=0.125 2023-11-24 22:49:55,225 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9200, loss[loss=0.04855, simple_loss=0.05728, pruned_loss=0.007315, audio_tagging_loss=0.01259, over 16694.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.09027, pruned_loss=0.0127, audio_tagging_loss=0.008664, over 3050428.98 frames. ], batch size: 64, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 22:50:16,204 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454100 2023-11-24 22:50:21,454 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.85 vs. limit=12.0 2023-11-24 22:50:29,192 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3027340.0, ans=0.125 2023-11-24 22:50:29,428 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=3027340.0, ans=0.125 2023-11-24 22:50:37,885 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=3027406.6666666665, ans=0.05 2023-11-24 22:50:57,386 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9250, loss[loss=0.05711, simple_loss=0.07337, pruned_loss=0.01116, audio_tagging_loss=0.009268, over 14590.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.09004, pruned_loss=0.01291, audio_tagging_loss=0.00861, over 3052913.55 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:51:03,933 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.41 vs. limit=12.0 2023-11-24 22:51:18,297 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454150 2023-11-24 22:51:24,365 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=3027673.3333333335, ans=0.125 2023-11-24 22:51:41,749 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.619e+01 9.253e+01 1.002e+02 1.218e+02, threshold=1.851e+02, percent-clipped=0.0 2023-11-24 22:51:49,517 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.18 vs. limit=15.0 2023-11-24 22:51:51,777 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=3027806.6666666665, ans=22.5 2023-11-24 22:51:52,741 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3027806.6666666665, ans=0.125 2023-11-24 22:51:58,229 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9300, loss[loss=0.08359, simple_loss=0.1197, pruned_loss=0.01677, audio_tagging_loss=0.006967, over 16598.00 frames. ], tot_loss[loss=0.0661, simple_loss=0.08939, pruned_loss=0.01274, audio_tagging_loss=0.008666, over 3050570.74 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:52:06,052 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.31 vs. limit=10.0 2023-11-24 22:52:11,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3027940.0, ans=0.125 2023-11-24 22:52:20,077 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454200 2023-11-24 22:52:25,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3028006.6666666665, ans=0.0 2023-11-24 22:52:29,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3028006.6666666665, ans=0.0 2023-11-24 22:52:44,739 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=3028073.3333333335, ans=0.2 2023-11-24 22:52:58,707 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=3028140.0, ans=0.0 2023-11-24 22:53:00,908 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9350, loss[loss=0.06999, simple_loss=0.09682, pruned_loss=0.01113, audio_tagging_loss=0.01044, over 15777.00 frames. ], tot_loss[loss=0.06626, simple_loss=0.08959, pruned_loss=0.01275, audio_tagging_loss=0.008723, over 3050367.95 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:53:01,268 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3028206.6666666665, ans=0.0 2023-11-24 22:53:22,130 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454250 2023-11-24 22:53:32,827 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3028340.0, ans=0.125 2023-11-24 22:53:45,747 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.804e+01 9.416e+01 1.006e+02 1.917e+02, threshold=1.883e+02, percent-clipped=1.0 2023-11-24 22:53:49,799 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3028473.3333333335, ans=0.125 2023-11-24 22:53:57,897 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3028473.3333333335, ans=0.1 2023-11-24 22:54:03,458 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9400, loss[loss=0.08075, simple_loss=0.107, pruned_loss=0.01826, audio_tagging_loss=0.008981, over 15831.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.09036, pruned_loss=0.01276, audio_tagging_loss=0.008832, over 3051907.74 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:54:24,306 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454300 2023-11-24 22:54:24,381 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=3028606.6666666665, ans=0.0 2023-11-24 22:54:49,645 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3028740.0, ans=0.0 2023-11-24 22:55:00,259 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3028806.6666666665, ans=0.125 2023-11-24 22:55:00,410 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=3028806.6666666665, ans=0.2 2023-11-24 22:55:01,329 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 22:55:05,008 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9450, loss[loss=0.06311, simple_loss=0.0798, pruned_loss=0.01518, audio_tagging_loss=0.008028, over 15464.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09123, pruned_loss=0.01301, audio_tagging_loss=0.008871, over 3058416.39 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:55:25,458 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3028940.0, ans=0.125 2023-11-24 22:55:26,439 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454350 2023-11-24 22:55:32,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3029006.6666666665, ans=0.125 2023-11-24 22:55:33,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=3029006.6666666665, ans=0.0 2023-11-24 22:55:49,906 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.559e+01 8.713e+01 9.194e+01 9.823e+01 1.241e+02, threshold=1.839e+02, percent-clipped=0.0 2023-11-24 22:56:07,686 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9500, loss[loss=0.06884, simple_loss=0.09414, pruned_loss=0.01114, audio_tagging_loss=0.01064, over 15070.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09166, pruned_loss=0.01305, audio_tagging_loss=0.008938, over 3055202.20 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:56:27,063 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=3029273.3333333335, ans=0.0 2023-11-24 22:56:29,261 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454400 2023-11-24 22:56:31,998 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3029340.0, ans=0.1 2023-11-24 22:56:39,120 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3029340.0, ans=0.125 2023-11-24 22:56:48,155 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=3029406.6666666665, ans=0.125 2023-11-24 22:57:03,403 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3029473.3333333335, ans=0.1 2023-11-24 22:57:08,769 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3029473.3333333335, ans=0.125 2023-11-24 22:57:10,763 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9550, loss[loss=0.06917, simple_loss=0.09089, pruned_loss=0.01144, audio_tagging_loss=0.01228, over 15353.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09146, pruned_loss=0.0129, audio_tagging_loss=0.008977, over 3052187.90 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 22:57:22,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3029606.6666666665, ans=0.0 2023-11-24 22:57:31,109 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454450 2023-11-24 22:57:41,387 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3029673.3333333335, ans=0.125 2023-11-24 22:57:44,844 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3029673.3333333335, ans=0.125 2023-11-24 22:57:50,063 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.81 vs. limit=12.0 2023-11-24 22:57:56,027 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.670e+01 8.429e+01 9.044e+01 9.550e+01 1.234e+02, threshold=1.809e+02, percent-clipped=0.0 2023-11-24 22:58:09,791 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3029806.6666666665, ans=0.0 2023-11-24 22:58:10,990 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3029806.6666666665, ans=0.125 2023-11-24 22:58:12,968 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9600, loss[loss=0.06905, simple_loss=0.09149, pruned_loss=0.01391, audio_tagging_loss=0.009405, over 14713.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09188, pruned_loss=0.01302, audio_tagging_loss=0.008969, over 3048670.56 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 22:58:13,293 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3029873.3333333335, ans=0.125 2023-11-24 22:58:14,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3029873.3333333335, ans=0.1 2023-11-24 22:58:14,595 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3029873.3333333335, ans=0.125 2023-11-24 22:58:18,073 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3029873.3333333335, ans=0.125 2023-11-24 22:58:23,582 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.82 vs. limit=15.0 2023-11-24 22:58:29,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3029940.0, ans=0.125 2023-11-24 22:58:30,271 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 22:58:31,742 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.33 vs. limit=22.5 2023-11-24 22:58:33,512 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454500 2023-11-24 22:59:14,826 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9650, loss[loss=0.06788, simple_loss=0.09943, pruned_loss=0.01207, audio_tagging_loss=0.006095, over 15700.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09048, pruned_loss=0.0128, audio_tagging_loss=0.008955, over 3043352.69 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 22:59:30,177 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=3030273.3333333335, ans=0.125 2023-11-24 22:59:36,988 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454550 2023-11-24 22:59:46,789 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3030340.0, ans=0.125 2023-11-24 22:59:58,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3030406.6666666665, ans=0.1 2023-11-24 23:00:01,102 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.723e+01 9.168e+01 1.005e+02 1.303e+02, threshold=1.834e+02, percent-clipped=0.0 2023-11-24 23:00:15,574 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=3030473.3333333335, ans=0.0 2023-11-24 23:00:18,272 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9700, loss[loss=0.05307, simple_loss=0.06778, pruned_loss=0.01024, audio_tagging_loss=0.008939, over 15179.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08989, pruned_loss=0.01278, audio_tagging_loss=0.008866, over 3047362.71 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:00:20,858 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3030540.0, ans=0.125 2023-11-24 23:00:26,013 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.03 vs. limit=10.0 2023-11-24 23:00:28,095 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3030540.0, ans=0.0 2023-11-24 23:00:34,179 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=3030606.6666666665, ans=0.0 2023-11-24 23:00:38,655 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454600 2023-11-24 23:00:56,589 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3030740.0, ans=0.0 2023-11-24 23:00:57,679 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3030740.0, ans=0.0 2023-11-24 23:00:57,744 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=3030740.0, ans=0.0 2023-11-24 23:00:57,822 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3030740.0, ans=0.0 2023-11-24 23:01:05,736 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.61 vs. limit=15.0 2023-11-24 23:01:10,223 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3030806.6666666665, ans=0.125 2023-11-24 23:01:20,992 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9750, loss[loss=0.04624, simple_loss=0.06119, pruned_loss=0.007571, audio_tagging_loss=0.008069, over 16377.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08975, pruned_loss=0.01265, audio_tagging_loss=0.008791, over 3054579.34 frames. ], batch size: 64, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:01:29,849 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3030873.3333333335, ans=0.125 2023-11-24 23:01:31,427 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3030873.3333333335, ans=0.1 2023-11-24 23:01:41,963 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454650 2023-11-24 23:01:50,935 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=3031006.6666666665, ans=0.2 2023-11-24 23:02:06,533 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.511e+01 8.357e+01 8.880e+01 9.692e+01 1.346e+02, threshold=1.776e+02, percent-clipped=0.0 2023-11-24 23:02:18,313 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3031140.0, ans=0.125 2023-11-24 23:02:22,675 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9800, loss[loss=0.04565, simple_loss=0.05728, pruned_loss=0.004581, audio_tagging_loss=0.01243, over 14858.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.09024, pruned_loss=0.01281, audio_tagging_loss=0.008765, over 3044404.07 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:02:28,810 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3031206.6666666665, ans=0.1 2023-11-24 23:02:44,563 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454700 2023-11-24 23:02:48,374 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3031340.0, ans=0.125 2023-11-24 23:02:57,103 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.52 vs. limit=15.0 2023-11-24 23:03:00,791 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.24 vs. limit=15.0 2023-11-24 23:03:04,368 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.40 vs. limit=12.0 2023-11-24 23:03:07,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3031406.6666666665, ans=0.0 2023-11-24 23:03:07,116 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3031406.6666666665, ans=0.125 2023-11-24 23:03:16,399 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 23:03:25,429 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9850, loss[loss=0.05633, simple_loss=0.07208, pruned_loss=0.01053, audio_tagging_loss=0.00976, over 15051.00 frames. ], tot_loss[loss=0.06657, simple_loss=0.09004, pruned_loss=0.01277, audio_tagging_loss=0.008778, over 3050096.54 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:03:29,988 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3031540.0, ans=0.125 2023-11-24 23:03:46,075 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=3031606.6666666665, ans=0.025 2023-11-24 23:03:47,048 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454750 2023-11-24 23:04:13,179 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.582e+01 8.592e+01 9.130e+01 1.016e+02 1.240e+02, threshold=1.826e+02, percent-clipped=0.0 2023-11-24 23:04:28,564 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9900, loss[loss=0.06787, simple_loss=0.08918, pruned_loss=0.01225, audio_tagging_loss=0.01103, over 14141.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.09009, pruned_loss=0.01267, audio_tagging_loss=0.008737, over 3050823.34 frames. ], batch size: 52, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:04:31,235 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3031873.3333333335, ans=0.125 2023-11-24 23:04:32,943 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.91 vs. limit=15.0 2023-11-24 23:04:35,943 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=3031873.3333333335, ans=0.125 2023-11-24 23:04:47,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3031940.0, ans=0.125 2023-11-24 23:04:49,708 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454800 2023-11-24 23:05:16,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3032073.3333333335, ans=0.1 2023-11-24 23:05:19,819 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3032140.0, ans=0.0 2023-11-24 23:05:29,338 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3032140.0, ans=0.0 2023-11-24 23:05:31,864 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 9950, loss[loss=0.08856, simple_loss=0.1201, pruned_loss=0.01928, audio_tagging_loss=0.009254, over 15070.00 frames. ], tot_loss[loss=0.06617, simple_loss=0.08994, pruned_loss=0.01245, audio_tagging_loss=0.008747, over 3057470.97 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:05:53,014 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454850 2023-11-24 23:05:59,007 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=3032340.0, ans=0.125 2023-11-24 23:06:09,181 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:06:18,926 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.463e+01 8.965e+01 9.807e+01 1.516e+02, threshold=1.793e+02, percent-clipped=0.0 2023-11-24 23:06:28,052 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.76 vs. limit=15.0 2023-11-24 23:06:33,797 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10000, loss[loss=0.06572, simple_loss=0.08045, pruned_loss=0.0126, audio_tagging_loss=0.0129, over 15540.00 frames. ], tot_loss[loss=0.06598, simple_loss=0.0895, pruned_loss=0.0124, audio_tagging_loss=0.008829, over 3054655.47 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:06:47,431 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3032606.6666666665, ans=0.125 2023-11-24 23:06:54,902 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454900 2023-11-24 23:07:20,898 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:07:35,229 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10050, loss[loss=0.06865, simple_loss=0.08942, pruned_loss=0.01687, audio_tagging_loss=0.007065, over 15383.00 frames. ], tot_loss[loss=0.06633, simple_loss=0.08989, pruned_loss=0.01251, audio_tagging_loss=0.008876, over 3049999.84 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:07:48,060 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3032940.0, ans=0.125 2023-11-24 23:07:56,656 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 454950 2023-11-24 23:08:07,946 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=3033006.6666666665, ans=0.125 2023-11-24 23:08:10,668 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0 2023-11-24 23:08:23,658 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.450e+01 9.036e+01 9.785e+01 1.299e+02, threshold=1.807e+02, percent-clipped=0.0 2023-11-24 23:08:26,570 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.99 vs. limit=22.5 2023-11-24 23:08:31,560 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3033140.0, ans=0.0 2023-11-24 23:08:37,249 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10100, loss[loss=0.06649, simple_loss=0.09078, pruned_loss=0.01155, audio_tagging_loss=0.009547, over 15204.00 frames. ], tot_loss[loss=0.06604, simple_loss=0.08935, pruned_loss=0.01242, audio_tagging_loss=0.008942, over 3042505.70 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:08:41,481 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3033206.6666666665, ans=0.125 2023-11-24 23:08:45,418 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3033206.6666666665, ans=0.0 2023-11-24 23:08:59,052 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455000 2023-11-24 23:09:02,039 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=3033340.0, ans=0.0 2023-11-24 23:09:08,202 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3033340.0, ans=0.125 2023-11-24 23:09:26,540 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 23:09:39,209 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=3033540.0, ans=0.0 2023-11-24 23:09:40,107 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10150, loss[loss=0.06492, simple_loss=0.09654, pruned_loss=0.009765, audio_tagging_loss=0.006883, over 14806.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08966, pruned_loss=0.01252, audio_tagging_loss=0.008958, over 3046534.25 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:10:01,314 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455050 2023-11-24 23:10:07,817 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 23:10:29,130 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.319e+01 8.639e+01 9.334e+01 9.887e+01 1.234e+02, threshold=1.867e+02, percent-clipped=0.0 2023-11-24 23:10:43,087 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10200, loss[loss=0.08428, simple_loss=0.1182, pruned_loss=0.01925, audio_tagging_loss=0.005938, over 14770.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09053, pruned_loss=0.01275, audio_tagging_loss=0.008935, over 3052580.44 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:10:49,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3033873.3333333335, ans=0.1 2023-11-24 23:11:04,524 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 23:11:04,582 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455100 2023-11-24 23:11:15,941 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3034006.6666666665, ans=0.2 2023-11-24 23:11:34,185 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=3034140.0, ans=0.0 2023-11-24 23:11:45,353 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10250, loss[loss=0.07142, simple_loss=0.08878, pruned_loss=0.01725, audio_tagging_loss=0.009784, over 15969.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09118, pruned_loss=0.01299, audio_tagging_loss=0.008919, over 3054561.35 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:11:49,196 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=3034206.6666666665, ans=0.2 2023-11-24 23:12:06,501 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455150 2023-11-24 23:12:13,378 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=3034340.0, ans=0.0 2023-11-24 23:12:14,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3034340.0, ans=0.125 2023-11-24 23:12:26,740 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.33 vs. limit=22.5 2023-11-24 23:12:33,697 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.500e+01 8.646e+01 9.167e+01 9.835e+01 1.179e+02, threshold=1.833e+02, percent-clipped=0.0 2023-11-24 23:12:34,150 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3034473.3333333335, ans=0.125 2023-11-24 23:12:35,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3034473.3333333335, ans=0.0 2023-11-24 23:12:47,008 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10300, loss[loss=0.06842, simple_loss=0.09065, pruned_loss=0.01403, audio_tagging_loss=0.00906, over 14684.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09118, pruned_loss=0.01292, audio_tagging_loss=0.008994, over 3051440.66 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:12:47,353 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=3034540.0, ans=0.07 2023-11-24 23:12:52,965 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.83 vs. limit=15.0 2023-11-24 23:12:54,979 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=3034540.0, ans=0.125 2023-11-24 23:12:55,208 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=3034540.0, ans=0.125 2023-11-24 23:12:59,131 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=3034606.6666666665, ans=0.05 2023-11-24 23:12:59,276 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3034606.6666666665, ans=0.1 2023-11-24 23:13:02,900 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=3034606.6666666665, ans=0.125 2023-11-24 23:13:08,702 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455200 2023-11-24 23:13:17,401 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=3034673.3333333335, ans=0.125 2023-11-24 23:13:17,511 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=3034673.3333333335, ans=0.0 2023-11-24 23:13:32,985 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3034740.0, ans=0.125 2023-11-24 23:13:50,473 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10350, loss[loss=0.07518, simple_loss=0.09789, pruned_loss=0.01446, audio_tagging_loss=0.01178, over 16571.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09164, pruned_loss=0.01288, audio_tagging_loss=0.009028, over 3048351.70 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:14:11,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455250 2023-11-24 23:14:17,914 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3035006.6666666665, ans=0.0 2023-11-24 23:14:19,052 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=3035006.6666666665, ans=0.2 2023-11-24 23:14:27,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=3035073.3333333335, ans=0.125 2023-11-24 23:14:38,310 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.693e+01 9.178e+01 1.025e+02 1.318e+02, threshold=1.836e+02, percent-clipped=0.0 2023-11-24 23:14:43,363 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=3035140.0, ans=0.125 2023-11-24 23:14:51,332 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10400, loss[loss=0.08168, simple_loss=0.1105, pruned_loss=0.0161, audio_tagging_loss=0.01033, over 14343.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09107, pruned_loss=0.01284, audio_tagging_loss=0.009049, over 3047453.33 frames. ], batch size: 52, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:14:55,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=3035206.6666666665, ans=0.2 2023-11-24 23:15:05,829 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3035273.3333333335, ans=0.125 2023-11-24 23:15:12,892 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455300 2023-11-24 23:15:27,043 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.88 vs. limit=12.0 2023-11-24 23:15:53,555 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10450, loss[loss=0.07302, simple_loss=0.09739, pruned_loss=0.01619, audio_tagging_loss=0.008136, over 15671.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09118, pruned_loss=0.01287, audio_tagging_loss=0.009063, over 3042344.57 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:16:14,914 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455350 2023-11-24 23:16:42,061 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.886e+01 9.517e+01 1.018e+02 1.385e+02, threshold=1.903e+02, percent-clipped=0.0 2023-11-24 23:16:56,071 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10500, loss[loss=0.07738, simple_loss=0.1093, pruned_loss=0.01413, audio_tagging_loss=0.00858, over 16248.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09186, pruned_loss=0.01296, audio_tagging_loss=0.008863, over 3050994.26 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:16:57,415 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=3035873.3333333335, ans=0.5 2023-11-24 23:16:57,973 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.36 vs. limit=6.0 2023-11-24 23:17:02,118 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3035873.3333333335, ans=0.0 2023-11-24 23:17:16,592 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455400 2023-11-24 23:17:32,843 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.81 vs. limit=6.0 2023-11-24 23:17:46,510 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3036140.0, ans=0.1 2023-11-24 23:17:48,145 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.51 vs. limit=15.0 2023-11-24 23:17:58,254 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10550, loss[loss=0.06569, simple_loss=0.0878, pruned_loss=0.01292, audio_tagging_loss=0.008875, over 15803.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09092, pruned_loss=0.01274, audio_tagging_loss=0.008739, over 3043733.66 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:17:59,123 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.78 vs. limit=15.0 2023-11-24 23:18:09,550 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=3036273.3333333335, ans=0.0 2023-11-24 23:18:10,856 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:18:19,461 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455450 2023-11-24 23:18:46,506 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.547e+01 9.162e+01 9.903e+01 1.156e+02, threshold=1.832e+02, percent-clipped=0.0 2023-11-24 23:18:52,743 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=3036473.3333333335, ans=0.2 2023-11-24 23:18:54,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3036473.3333333335, ans=0.125 2023-11-24 23:18:56,331 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.02 vs. limit=22.5 2023-11-24 23:18:59,121 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3036540.0, ans=0.125 2023-11-24 23:19:00,136 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10600, loss[loss=0.06807, simple_loss=0.08991, pruned_loss=0.01291, audio_tagging_loss=0.0102, over 15536.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.09024, pruned_loss=0.01276, audio_tagging_loss=0.008751, over 3039476.35 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:19:08,545 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3036540.0, ans=0.1 2023-11-24 23:19:22,439 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455500 2023-11-24 23:19:37,980 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3036740.0, ans=0.0 2023-11-24 23:19:40,411 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=3036740.0, ans=0.0 2023-11-24 23:19:44,916 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0 2023-11-24 23:19:49,438 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=3036806.6666666665, ans=0.04949747468305833 2023-11-24 23:19:54,675 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:19:54,688 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=3036806.6666666665, ans=0.035 2023-11-24 23:19:54,850 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3036806.6666666665, ans=0.125 2023-11-24 23:19:55,984 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3036806.6666666665, ans=0.1 2023-11-24 23:20:02,305 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.36 vs. limit=15.0 2023-11-24 23:20:03,939 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10650, loss[loss=0.07014, simple_loss=0.09365, pruned_loss=0.01406, audio_tagging_loss=0.00925, over 15752.00 frames. ], tot_loss[loss=0.06616, simple_loss=0.08946, pruned_loss=0.01269, audio_tagging_loss=0.008737, over 3032241.05 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:20:20,706 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=3036940.0, ans=0.04949747468305833 2023-11-24 23:20:24,020 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455550 2023-11-24 23:20:26,932 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.54 vs. limit=15.0 2023-11-24 23:20:27,793 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3037006.6666666665, ans=0.1 2023-11-24 23:20:29,252 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.31 vs. limit=15.0 2023-11-24 23:20:47,997 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.53 vs. limit=15.0 2023-11-24 23:20:52,473 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.714e+01 9.529e+01 1.031e+02 1.468e+02, threshold=1.906e+02, percent-clipped=0.0 2023-11-24 23:21:05,489 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10700, loss[loss=0.06417, simple_loss=0.07314, pruned_loss=0.01746, audio_tagging_loss=0.01014, over 14758.00 frames. ], tot_loss[loss=0.06612, simple_loss=0.08942, pruned_loss=0.01268, audio_tagging_loss=0.008728, over 3035341.57 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:21:26,180 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455600 2023-11-24 23:21:26,300 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3037273.3333333335, ans=0.0 2023-11-24 23:21:53,725 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=15.0 2023-11-24 23:22:07,804 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10750, loss[loss=0.06358, simple_loss=0.09526, pruned_loss=0.009665, audio_tagging_loss=0.006283, over 16638.00 frames. ], tot_loss[loss=0.0663, simple_loss=0.08966, pruned_loss=0.01281, audio_tagging_loss=0.00866, over 3043176.97 frames. ], batch size: 63, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:22:29,832 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455650 2023-11-24 23:22:56,330 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.544e+01 9.121e+01 9.679e+01 1.132e+02, threshold=1.824e+02, percent-clipped=0.0 2023-11-24 23:23:03,765 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=3037806.6666666665, ans=0.125 2023-11-24 23:23:10,665 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10800, loss[loss=0.06779, simple_loss=0.08796, pruned_loss=0.01447, audio_tagging_loss=0.009343, over 14923.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.09021, pruned_loss=0.01287, audio_tagging_loss=0.00868, over 3041131.53 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:23:30,776 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455700 2023-11-24 23:23:47,502 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.86 vs. limit=15.0 2023-11-24 23:23:49,559 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=3038073.3333333335, ans=0.0 2023-11-24 23:23:58,937 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3038140.0, ans=0.0 2023-11-24 23:24:11,714 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10850, loss[loss=0.05865, simple_loss=0.07797, pruned_loss=0.01052, audio_tagging_loss=0.009151, over 15427.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09033, pruned_loss=0.01291, audio_tagging_loss=0.008732, over 3051649.39 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:24:32,502 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455750 2023-11-24 23:24:59,977 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.004e+01 8.552e+01 9.146e+01 9.923e+01 1.188e+02, threshold=1.829e+02, percent-clipped=0.0 2023-11-24 23:25:00,422 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3038473.3333333335, ans=0.1 2023-11-24 23:25:07,133 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 23:25:13,544 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10900, loss[loss=0.07616, simple_loss=0.09494, pruned_loss=0.01875, audio_tagging_loss=0.00994, over 14750.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09074, pruned_loss=0.013, audio_tagging_loss=0.008729, over 3046981.96 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:25:14,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3038540.0, ans=0.125 2023-11-24 23:25:28,509 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3038606.6666666665, ans=0.125 2023-11-24 23:25:35,538 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455800 2023-11-24 23:25:38,502 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3038673.3333333335, ans=0.1 2023-11-24 23:25:42,164 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3038673.3333333335, ans=0.125 2023-11-24 23:25:58,733 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=3038740.0, ans=0.2 2023-11-24 23:26:08,808 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=3038806.6666666665, ans=0.5 2023-11-24 23:26:12,252 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=3038806.6666666665, ans=0.0 2023-11-24 23:26:16,214 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 10950, loss[loss=0.06811, simple_loss=0.08249, pruned_loss=0.01421, audio_tagging_loss=0.01266, over 14517.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.09041, pruned_loss=0.0128, audio_tagging_loss=0.008796, over 3047323.01 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:26:17,025 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.77 vs. limit=22.5 2023-11-24 23:26:37,329 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455850 2023-11-24 23:26:47,174 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=3039006.6666666665, ans=0.125 2023-11-24 23:26:49,942 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=3039006.6666666665, ans=0.125 2023-11-24 23:26:57,546 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.49 vs. limit=15.0 2023-11-24 23:27:05,052 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.762e+01 9.251e+01 9.897e+01 1.242e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 23:27:07,776 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=3039140.0, ans=0.125 2023-11-24 23:27:13,195 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:27:18,790 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11000, loss[loss=0.05697, simple_loss=0.07315, pruned_loss=0.009716, audio_tagging_loss=0.01068, over 15678.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09126, pruned_loss=0.01295, audio_tagging_loss=0.008835, over 3046300.15 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:27:26,061 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 23:27:31,141 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=3039273.3333333335, ans=0.0 2023-11-24 23:27:39,739 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455900 2023-11-24 23:27:49,298 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=3039340.0, ans=0.025 2023-11-24 23:27:56,399 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=3039406.6666666665, ans=0.0 2023-11-24 23:27:57,609 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=3039406.6666666665, ans=0.04949747468305833 2023-11-24 23:27:58,875 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3039406.6666666665, ans=0.1 2023-11-24 23:28:06,674 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.79 vs. limit=15.0 2023-11-24 23:28:08,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3039473.3333333335, ans=0.125 2023-11-24 23:28:12,521 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.51 vs. limit=12.0 2023-11-24 23:28:17,352 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.10 vs. limit=15.0 2023-11-24 23:28:18,825 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.43 vs. limit=15.0 2023-11-24 23:28:20,381 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11050, loss[loss=0.0711, simple_loss=0.1063, pruned_loss=0.01192, audio_tagging_loss=0.006029, over 15688.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09056, pruned_loss=0.01268, audio_tagging_loss=0.009016, over 3041221.91 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:28:28,302 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:28:29,420 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3039540.0, ans=0.125 2023-11-24 23:28:34,026 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3039606.6666666665, ans=0.125 2023-11-24 23:28:42,223 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 455950 2023-11-24 23:28:57,310 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3039740.0, ans=0.1 2023-11-24 23:29:08,958 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.642e+01 9.315e+01 1.003e+02 1.536e+02, threshold=1.863e+02, percent-clipped=0.0 2023-11-24 23:29:22,459 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11100, loss[loss=0.07401, simple_loss=0.09675, pruned_loss=0.01411, audio_tagging_loss=0.01153, over 16885.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09135, pruned_loss=0.01273, audio_tagging_loss=0.009085, over 3047957.94 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:29:29,859 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3039873.3333333335, ans=0.0 2023-11-24 23:29:29,924 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=3039873.3333333335, ans=0.0 2023-11-24 23:29:42,332 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=3039940.0, ans=0.125 2023-11-24 23:29:44,640 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456000 2023-11-24 23:29:45,958 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=3039940.0, ans=0.125 2023-11-24 23:30:28,639 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=3040206.6666666665, ans=0.2 2023-11-24 23:30:28,642 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=3040206.6666666665, ans=0.0 2023-11-24 23:30:30,067 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11150, loss[loss=0.06852, simple_loss=0.09371, pruned_loss=0.01368, audio_tagging_loss=0.007977, over 16263.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09143, pruned_loss=0.01276, audio_tagging_loss=0.00919, over 3051325.62 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:30:34,991 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3040206.6666666665, ans=0.0 2023-11-24 23:30:37,799 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.60 vs. limit=6.0 2023-11-24 23:30:39,196 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.34 vs. limit=15.0 2023-11-24 23:30:49,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=3040273.3333333335, ans=0.04949747468305833 2023-11-24 23:30:50,891 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456050 2023-11-24 23:30:51,320 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=3040273.3333333335, ans=15.0 2023-11-24 23:30:53,512 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3040340.0, ans=0.125 2023-11-24 23:31:10,337 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.80 vs. limit=15.0 2023-11-24 23:31:10,343 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.92 vs. limit=22.5 2023-11-24 23:31:14,627 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3040406.6666666665, ans=0.1 2023-11-24 23:31:18,465 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.591e+01 9.320e+01 9.957e+01 1.253e+02, threshold=1.864e+02, percent-clipped=0.0 2023-11-24 23:31:22,327 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3040473.3333333335, ans=0.1 2023-11-24 23:31:29,392 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3040473.3333333335, ans=0.0 2023-11-24 23:31:31,383 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11200, loss[loss=0.05975, simple_loss=0.07877, pruned_loss=0.01089, audio_tagging_loss=0.009473, over 15289.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09028, pruned_loss=0.01258, audio_tagging_loss=0.009296, over 3048290.40 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 32.0 2023-11-24 23:31:53,231 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456100 2023-11-24 23:32:33,943 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11250, loss[loss=0.06615, simple_loss=0.09494, pruned_loss=0.01163, audio_tagging_loss=0.007047, over 15936.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08989, pruned_loss=0.01248, audio_tagging_loss=0.009165, over 3049303.19 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:32:42,395 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3040873.3333333335, ans=0.125 2023-11-24 23:32:43,580 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=3040873.3333333335, ans=0.125 2023-11-24 23:32:50,947 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.78 vs. limit=22.5 2023-11-24 23:32:51,837 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3040940.0, ans=0.125 2023-11-24 23:32:55,770 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456150 2023-11-24 23:33:04,461 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3041006.6666666665, ans=0.0 2023-11-24 23:33:23,493 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.642e+01 8.456e+01 9.058e+01 9.747e+01 1.230e+02, threshold=1.812e+02, percent-clipped=0.0 2023-11-24 23:33:30,382 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3041140.0, ans=0.1 2023-11-24 23:33:35,476 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.79 vs. limit=12.0 2023-11-24 23:33:35,954 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11300, loss[loss=0.07585, simple_loss=0.1077, pruned_loss=0.01305, audio_tagging_loss=0.00896, over 16269.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09054, pruned_loss=0.01253, audio_tagging_loss=0.009094, over 3047498.84 frames. ], batch size: 61, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:33:46,263 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=3041206.6666666665, ans=0.07 2023-11-24 23:33:55,631 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3041273.3333333335, ans=0.125 2023-11-24 23:33:56,678 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456200 2023-11-24 23:34:07,306 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3041340.0, ans=0.125 2023-11-24 23:34:12,051 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=3041406.6666666665, ans=0.0 2023-11-24 23:34:34,249 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3041473.3333333335, ans=0.1 2023-11-24 23:34:37,459 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11350, loss[loss=0.06397, simple_loss=0.09111, pruned_loss=0.01205, audio_tagging_loss=0.006359, over 16993.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.09012, pruned_loss=0.0125, audio_tagging_loss=0.009018, over 3045446.33 frames. ], batch size: 63, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:34:42,881 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.10 vs. limit=6.0 2023-11-24 23:34:53,823 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3041606.6666666665, ans=0.1 2023-11-24 23:34:58,561 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456250 2023-11-24 23:35:19,790 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=3041740.0, ans=0.2 2023-11-24 23:35:28,353 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.570e+01 8.600e+01 9.248e+01 9.972e+01 1.236e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 23:35:31,030 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:35:39,400 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11400, loss[loss=0.05683, simple_loss=0.07199, pruned_loss=0.009034, audio_tagging_loss=0.0118, over 13430.00 frames. ], tot_loss[loss=0.06622, simple_loss=0.08965, pruned_loss=0.01252, audio_tagging_loss=0.008881, over 3039731.20 frames. ], batch size: 52, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:35:39,701 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3041873.3333333335, ans=0.1 2023-11-24 23:36:00,217 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456300 2023-11-24 23:36:07,845 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3042006.6666666665, ans=0.1 2023-11-24 23:36:19,626 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3042073.3333333335, ans=0.125 2023-11-24 23:36:24,476 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3042073.3333333335, ans=0.125 2023-11-24 23:36:35,855 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3042140.0, ans=0.0 2023-11-24 23:36:36,869 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3042140.0, ans=0.1 2023-11-24 23:36:41,457 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11450, loss[loss=0.07638, simple_loss=0.1053, pruned_loss=0.01391, audio_tagging_loss=0.009852, over 15556.00 frames. ], tot_loss[loss=0.0663, simple_loss=0.08959, pruned_loss=0.01261, audio_tagging_loss=0.008894, over 3043408.21 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:36:51,270 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=3042206.6666666665, ans=0.125 2023-11-24 23:37:02,358 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456350 2023-11-24 23:37:03,634 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3042273.3333333335, ans=0.125 2023-11-24 23:37:03,678 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:37:03,753 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3042273.3333333335, ans=0.1 2023-11-24 23:37:23,661 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3042406.6666666665, ans=0.2 2023-11-24 23:37:31,585 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.517e+01 9.262e+01 1.018e+02 1.314e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 23:37:31,797 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=3042473.3333333335, ans=0.125 2023-11-24 23:37:41,469 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=3042540.0, ans=0.95 2023-11-24 23:37:42,287 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11500, loss[loss=0.07291, simple_loss=0.1029, pruned_loss=0.01515, audio_tagging_loss=0.006302, over 15017.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09077, pruned_loss=0.01284, audio_tagging_loss=0.008771, over 3041927.67 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:38:03,717 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456400 2023-11-24 23:38:22,603 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3042740.0, ans=0.0 2023-11-24 23:38:44,368 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11550, loss[loss=0.0738, simple_loss=0.1091, pruned_loss=0.01347, audio_tagging_loss=0.005782, over 15666.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09151, pruned_loss=0.01287, audio_tagging_loss=0.008735, over 3046363.85 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:38:58,189 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3042940.0, ans=0.1 2023-11-24 23:39:05,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456450 2023-11-24 23:39:11,080 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=3043006.6666666665, ans=0.125 2023-11-24 23:39:19,201 WARNING [train_asr.py:1462] (3/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24 2023-11-24 23:39:20,718 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3043073.3333333335, ans=0.0 2023-11-24 23:39:20,806 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3043073.3333333335, ans=0.0 2023-11-24 23:39:34,490 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.770e+01 9.475e+01 1.009e+02 1.231e+02, threshold=1.895e+02, percent-clipped=0.0 2023-11-24 23:39:35,077 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.65 vs. limit=15.0 2023-11-24 23:39:37,286 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3043140.0, ans=0.0 2023-11-24 23:39:45,835 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11600, loss[loss=0.06692, simple_loss=0.09607, pruned_loss=0.0113, audio_tagging_loss=0.007588, over 14976.00 frames. ], tot_loss[loss=0.0668, simple_loss=0.09072, pruned_loss=0.01274, audio_tagging_loss=0.008702, over 3042977.29 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:39:59,527 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.55 vs. limit=15.0 2023-11-24 23:40:01,655 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3043273.3333333335, ans=0.1 2023-11-24 23:40:06,292 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456500 2023-11-24 23:40:32,475 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=3043406.6666666665, ans=0.0 2023-11-24 23:40:46,569 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11650, loss[loss=0.06193, simple_loss=0.08156, pruned_loss=0.01228, audio_tagging_loss=0.008871, over 15212.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.0911, pruned_loss=0.01292, audio_tagging_loss=0.00871, over 3043797.08 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:40:59,260 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3043606.6666666665, ans=0.0 2023-11-24 23:41:03,547 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3043606.6666666665, ans=0.0 2023-11-24 23:41:08,023 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456550 2023-11-24 23:41:30,503 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.14 vs. limit=12.0 2023-11-24 23:41:36,910 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.669e+01 8.725e+01 9.251e+01 1.014e+02 1.338e+02, threshold=1.850e+02, percent-clipped=0.0 2023-11-24 23:41:48,073 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11700, loss[loss=0.07234, simple_loss=0.1026, pruned_loss=0.01239, audio_tagging_loss=0.008638, over 14920.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.09014, pruned_loss=0.01286, audio_tagging_loss=0.008754, over 3045014.93 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:41:53,641 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:42:09,636 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456600 2023-11-24 23:42:33,683 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.61 vs. limit=15.0 2023-11-24 23:42:36,786 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3044140.0, ans=0.125 2023-11-24 23:42:46,824 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=3044140.0, ans=0.0 2023-11-24 23:42:50,842 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11750, loss[loss=0.06083, simple_loss=0.0777, pruned_loss=0.01156, audio_tagging_loss=0.01042, over 15525.00 frames. ], tot_loss[loss=0.06636, simple_loss=0.0897, pruned_loss=0.01279, audio_tagging_loss=0.008722, over 3050937.02 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:42:53,554 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3044206.6666666665, ans=0.125 2023-11-24 23:42:58,244 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=3044206.6666666665, ans=0.09899494936611666 2023-11-24 23:43:02,770 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3044273.3333333335, ans=0.125 2023-11-24 23:43:04,014 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:43:11,146 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456650 2023-11-24 23:43:26,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=3044406.6666666665, ans=0.125 2023-11-24 23:43:41,434 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3044473.3333333335, ans=0.1 2023-11-24 23:43:42,312 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.689e+01 9.260e+01 1.011e+02 1.469e+02, threshold=1.852e+02, percent-clipped=0.0 2023-11-24 23:43:51,899 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11800, loss[loss=0.05769, simple_loss=0.08358, pruned_loss=0.009007, audio_tagging_loss=0.006896, over 15406.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09096, pruned_loss=0.01307, audio_tagging_loss=0.008651, over 3050182.61 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:44:12,665 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456700 2023-11-24 23:44:25,533 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=3044673.3333333335, ans=0.125 2023-11-24 23:44:30,817 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=3044740.0, ans=15.0 2023-11-24 23:44:38,909 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.29 vs. limit=12.0 2023-11-24 23:44:44,649 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=3044806.6666666665, ans=0.07 2023-11-24 23:44:48,069 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3044806.6666666665, ans=0.125 2023-11-24 23:44:52,995 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11850, loss[loss=0.05354, simple_loss=0.07886, pruned_loss=0.006287, audio_tagging_loss=0.007827, over 14644.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09182, pruned_loss=0.01311, audio_tagging_loss=0.008735, over 3052015.58 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:45:04,596 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=3044940.0, ans=0.0 2023-11-24 23:45:14,840 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456750 2023-11-24 23:45:44,609 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.737e+01 9.408e+01 9.939e+01 1.281e+02, threshold=1.882e+02, percent-clipped=0.0 2023-11-24 23:45:54,845 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11900, loss[loss=0.07388, simple_loss=0.1003, pruned_loss=0.01388, audio_tagging_loss=0.009857, over 14600.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.09049, pruned_loss=0.01272, audio_tagging_loss=0.008914, over 3053830.19 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:46:00,368 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3045206.6666666665, ans=0.0 2023-11-24 23:46:07,411 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:46:15,644 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456800 2023-11-24 23:46:17,145 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3045273.3333333335, ans=0.0 2023-11-24 23:46:21,953 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=3045340.0, ans=0.5 2023-11-24 23:46:23,250 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=3045340.0, ans=0.125 2023-11-24 23:46:33,733 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.24 vs. limit=15.0 2023-11-24 23:46:45,367 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0 2023-11-24 23:46:56,505 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 11950, loss[loss=0.05454, simple_loss=0.07221, pruned_loss=0.007987, audio_tagging_loss=0.01045, over 15082.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.09023, pruned_loss=0.01277, audio_tagging_loss=0.008957, over 3051573.84 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 8.0 2023-11-24 23:47:13,957 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=3045606.6666666665, ans=0.0 2023-11-24 23:47:17,221 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456850 2023-11-24 23:47:17,523 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=3045606.6666666665, ans=0.2 2023-11-24 23:47:47,347 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.445e+01 8.455e+01 9.152e+01 9.873e+01 1.157e+02, threshold=1.830e+02, percent-clipped=0.0 2023-11-24 23:47:50,996 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=3045806.6666666665, ans=0.04949747468305833 2023-11-24 23:47:56,421 INFO [train_asr.py:1221] (3/4) Epoch 38, batch 12000, loss[loss=0.05759, simple_loss=0.07797, pruned_loss=0.01037, audio_tagging_loss=0.008228, over 15341.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.08972, pruned_loss=0.01269, audio_tagging_loss=0.00901, over 3052407.94 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0 2023-11-24 23:47:56,422 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 23:48:40,300 INFO [train_asr.py:1253] (3/4) Epoch 38, validation: loss=0.05738, simple_loss=0.0508, pruned_loss=0.005195, audio_tagging_loss=0.02678, over 4681554.00 frames. 2023-11-24 23:48:40,301 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 23:48:50,754 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3045940.0, ans=0.125 2023-11-24 23:48:53,109 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=3045940.0, ans=0.0 2023-11-24 23:48:55,618 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.65 vs. limit=12.0 2023-11-24 23:48:56,703 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0 2023-11-24 23:48:57,866 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.67 vs. limit=15.0 2023-11-24 23:48:59,668 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456900 2023-11-24 23:48:59,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3045940.0, ans=0.0 2023-11-24 23:49:37,963 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 0, loss[loss=0.07896, simple_loss=0.1078, pruned_loss=0.01193, audio_tagging_loss=0.0131, over 15471.00 frames. ], tot_loss[loss=0.07896, simple_loss=0.1078, pruned_loss=0.01193, audio_tagging_loss=0.0131, over 15471.00 frames. ], batch size: 55, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:49:37,963 INFO [train_asr.py:1244] (3/4) Computing validation loss 2023-11-24 23:50:14,445 INFO [train_asr.py:1253] (3/4) Epoch 39, validation: loss=0.0578, simple_loss=0.05083, pruned_loss=0.005244, audio_tagging_loss=0.02714, over 4681554.00 frames. 2023-11-24 23:50:14,446 INFO [train_asr.py:1254] (3/4) Maximum memory allocated so far is 26055MB 2023-11-24 23:50:32,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=3046086.6666666665, ans=0.0 2023-11-24 23:50:35,190 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3046086.6666666665, ans=0.1 2023-11-24 23:50:36,430 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=3046086.6666666665, ans=0.2 2023-11-24 23:50:40,662 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=3046153.3333333335, ans=0.0 2023-11-24 23:50:41,139 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.91 vs. limit=10.0 2023-11-24 23:50:41,794 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3046153.3333333335, ans=0.125 2023-11-24 23:50:48,826 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=3046153.3333333335, ans=0.0 2023-11-24 23:51:09,897 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 456950 2023-11-24 23:51:15,704 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 50, loss[loss=0.06896, simple_loss=0.08976, pruned_loss=0.01036, audio_tagging_loss=0.01373, over 14589.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09101, pruned_loss=0.01254, audio_tagging_loss=0.01662, over 690879.47 frames. ], batch size: 55, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:51:30,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=3046420.0, ans=0.05 2023-11-24 23:51:39,934 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 8.015e+01 9.557e+01 1.022e+02 1.093e+02 1.810e+02, threshold=2.044e+02, percent-clipped=0.0 2023-11-24 23:51:43,130 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=3046486.6666666665, ans=0.0 2023-11-24 23:52:00,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3046553.3333333335, ans=0.0 2023-11-24 23:52:11,443 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457000 2023-11-24 23:52:18,485 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 100, loss[loss=0.07672, simple_loss=0.0949, pruned_loss=0.01404, audio_tagging_loss=0.01523, over 15334.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09096, pruned_loss=0.01296, audio_tagging_loss=0.01594, over 1212908.73 frames. ], batch size: 57, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:52:39,643 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3046753.3333333335, ans=0.125 2023-11-24 23:52:55,314 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.92 vs. limit=15.0 2023-11-24 23:53:02,565 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3046886.6666666665, ans=0.1 2023-11-24 23:53:13,956 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.71 vs. limit=15.0 2023-11-24 23:53:14,573 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457050 2023-11-24 23:53:15,903 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3046953.3333333335, ans=0.125 2023-11-24 23:53:21,527 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 150, loss[loss=0.06974, simple_loss=0.1024, pruned_loss=0.0116, audio_tagging_loss=0.00692, over 14886.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09096, pruned_loss=0.01272, audio_tagging_loss=0.01438, over 1625670.22 frames. ], batch size: 56, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:53:28,680 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.80 vs. limit=15.0 2023-11-24 23:53:35,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3047086.6666666665, ans=0.125 2023-11-24 23:53:45,773 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.772e+01 8.928e+01 9.394e+01 1.010e+02 1.309e+02, threshold=1.879e+02, percent-clipped=0.0 2023-11-24 23:53:49,490 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0 2023-11-24 23:54:08,572 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.75 vs. limit=15.0 2023-11-24 23:54:18,136 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457100 2023-11-24 23:54:24,061 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 200, loss[loss=0.06363, simple_loss=0.08773, pruned_loss=0.01135, audio_tagging_loss=0.008415, over 14999.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09032, pruned_loss=0.0127, audio_tagging_loss=0.01281, over 1940682.07 frames. ], batch size: 55, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:54:24,661 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.16 vs. limit=15.0 2023-11-24 23:54:29,148 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3047353.3333333335, ans=0.0 2023-11-24 23:54:52,712 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3047486.6666666665, ans=0.125 2023-11-24 23:55:06,721 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=3047553.3333333335, ans=0.5 2023-11-24 23:55:14,156 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0 2023-11-24 23:55:19,673 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457150 2023-11-24 23:55:26,009 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 250, loss[loss=0.06794, simple_loss=0.08522, pruned_loss=0.01561, audio_tagging_loss=0.009722, over 14915.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09009, pruned_loss=0.01282, audio_tagging_loss=0.01155, over 2184942.80 frames. ], batch size: 54, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:55:41,161 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3047753.3333333335, ans=0.125 2023-11-24 23:55:50,702 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.824e+01 9.475e+01 1.039e+02 1.300e+02, threshold=1.895e+02, percent-clipped=0.0 2023-11-24 23:55:52,357 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=3047820.0, ans=0.2 2023-11-24 23:55:59,072 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.53 vs. limit=12.0 2023-11-24 23:56:03,429 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3047886.6666666665, ans=0.125 2023-11-24 23:56:20,583 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=3047953.3333333335, ans=0.025 2023-11-24 23:56:21,604 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457200 2023-11-24 23:56:28,287 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 300, loss[loss=0.06454, simple_loss=0.08971, pruned_loss=0.009888, audio_tagging_loss=0.0098, over 15270.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09207, pruned_loss=0.01307, audio_tagging_loss=0.01067, over 2373512.99 frames. ], batch size: 56, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:56:38,375 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=3048020.0, ans=0.125 2023-11-24 23:56:54,578 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=3048153.3333333335, ans=0.125 2023-11-24 23:56:55,583 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:56:56,843 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3048153.3333333335, ans=0.125 2023-11-24 23:56:58,704 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3048153.3333333335, ans=0.125 2023-11-24 23:57:04,641 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3048220.0, ans=0.1 2023-11-24 23:57:19,966 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=3048286.6666666665, ans=0.2 2023-11-24 23:57:20,375 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.00 vs. limit=12.0 2023-11-24 23:57:24,671 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457250 2023-11-24 23:57:26,122 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3048286.6666666665, ans=0.1 2023-11-24 23:57:30,467 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 350, loss[loss=0.05311, simple_loss=0.07249, pruned_loss=0.007722, audio_tagging_loss=0.009148, over 15678.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.0918, pruned_loss=0.01314, audio_tagging_loss=0.01004, over 2523648.54 frames. ], batch size: 60, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:57:30,705 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3048353.3333333335, ans=0.1 2023-11-24 23:57:34,764 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2023-11-24 23:57:44,261 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3048420.0, ans=0.125 2023-11-24 23:57:56,161 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.715e+01 9.321e+01 9.899e+01 1.393e+02, threshold=1.864e+02, percent-clipped=0.0 2023-11-24 23:58:05,289 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3048486.6666666665, ans=0.125 2023-11-24 23:58:06,838 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3048553.3333333335, ans=0.125 2023-11-24 23:58:25,979 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457300 2023-11-24 23:58:26,167 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=3048620.0, ans=0.2 2023-11-24 23:58:28,445 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3048620.0, ans=0.125 2023-11-24 23:58:32,353 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 400, loss[loss=0.0763, simple_loss=0.1086, pruned_loss=0.0151, audio_tagging_loss=0.006897, over 14729.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09127, pruned_loss=0.01303, audio_tagging_loss=0.009772, over 2637982.32 frames. ], batch size: 53, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:58:39,730 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3048686.6666666665, ans=0.1 2023-11-24 23:58:48,493 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=3048753.3333333335, ans=0.07 2023-11-24 23:59:01,226 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.37 vs. limit=22.5 2023-11-24 23:59:08,303 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.22 vs. limit=15.0 2023-11-24 23:59:16,965 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=3048886.6666666665, ans=0.0 2023-11-24 23:59:22,127 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3048953.3333333335, ans=0.125 2023-11-24 23:59:25,866 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3048953.3333333335, ans=0.1 2023-11-24 23:59:27,966 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457350 2023-11-24 23:59:33,731 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 450, loss[loss=0.05261, simple_loss=0.06519, pruned_loss=0.01036, audio_tagging_loss=0.009657, over 14427.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09183, pruned_loss=0.01307, audio_tagging_loss=0.009524, over 2726283.46 frames. ], batch size: 57, lr: 1.75e-03, grad_scale: 32.0 2023-11-24 23:59:35,762 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3049020.0, ans=0.125 2023-11-24 23:59:36,880 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3049020.0, ans=0.125 2023-11-24 23:59:59,407 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3049153.3333333335, ans=0.1 2023-11-25 00:00:00,318 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 7.663e+01 8.552e+01 9.288e+01 9.905e+01 1.638e+02, threshold=1.858e+02, percent-clipped=0.0 2023-11-25 00:00:01,713 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3049153.3333333335, ans=0.125 2023-11-25 00:00:15,564 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=3049220.0, ans=0.0 2023-11-25 00:00:19,032 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3049220.0, ans=0.125 2023-11-25 00:00:30,632 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457400 2023-11-25 00:00:36,780 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 500, loss[loss=0.07188, simple_loss=0.09802, pruned_loss=0.01674, audio_tagging_loss=0.006132, over 14547.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09153, pruned_loss=0.01308, audio_tagging_loss=0.009392, over 2797828.70 frames. ], batch size: 57, lr: 1.74e-03, grad_scale: 32.0 2023-11-25 00:00:45,180 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=3049353.3333333335, ans=0.125 2023-11-25 00:01:29,231 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3049620.0, ans=0.125 2023-11-25 00:01:32,578 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457450 2023-11-25 00:01:35,187 INFO [scaling.py:1118] (3/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2023-11-25 00:01:39,086 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 550, loss[loss=0.05974, simple_loss=0.08306, pruned_loss=0.008333, audio_tagging_loss=0.009877, over 15462.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09076, pruned_loss=0.01288, audio_tagging_loss=0.009334, over 2847348.57 frames. ], batch size: 56, lr: 1.74e-03, grad_scale: 32.0 2023-11-25 00:01:39,408 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3049686.6666666665, ans=0.125 2023-11-25 00:02:05,697 INFO [optim.py:476] (3/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.616e+01 9.440e+01 1.005e+02 1.281e+02, threshold=1.888e+02, percent-clipped=0.0 2023-11-25 00:02:06,490 INFO [scaling.py:1022] (3/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.99 vs. limit=15.0 2023-11-25 00:02:35,477 INFO [model.py:792] (3/4) Freeze_encoder: False; Current batch idx: 457500 2023-11-25 00:02:40,462 INFO [scaling.py:213] (3/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=3050020.0, ans=0.2 2023-11-25 00:02:41,293 INFO [train_asr.py:1221] (3/4) Epoch 39, batch 600, loss[loss=0.07662, simple_loss=0.1005, pruned_loss=0.0154, audio_tagging_loss=0.01096, over 16080.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09072, pruned_loss=0.01273, audio_tagging_loss=0.009349, over 2896546.30 frames. ], batch size: 61, lr: 1.74e-03, grad_scale: 32.0