Spaces:
Runtime error
Runtime error
File size: 33,926 Bytes
1bba990 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 |
2023-10-06 03:49:36,638 MJV2 INFO {'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/MJV2/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 7, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': True, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/MJV2', 'experiment_dir': './logs/MJV2', 'save_every_epoch': 20, 'name': 'MJV2', 'total_epoch': 300, 'pretrainG': 'assets/pretrained_v2/f0G40k.pth', 'pretrainD': 'assets/pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 1, 'save_every_weights': '1', 'if_cache_data_in_gpu': 1}
2023-10-06 03:49:41,021 MJV2 INFO loaded pretrained assets/pretrained_v2/f0G40k.pth
2023-10-06 03:49:41,611 MJV2 INFO <All keys matched successfully>
2023-10-06 03:49:41,612 MJV2 INFO loaded pretrained assets/pretrained_v2/f0D40k.pth
2023-10-06 03:49:42,543 MJV2 INFO <All keys matched successfully>
2023-10-06 03:50:16,825 MJV2 INFO Train Epoch: 1 [0%]
2023-10-06 03:50:16,826 MJV2 INFO [0, 0.0001]
2023-10-06 03:50:16,826 MJV2 INFO loss_disc=4.437, loss_gen=2.698, loss_fm=0.584,loss_mel=27.227, loss_kl=9.000
2023-10-06 03:50:17,898 MJV2 INFO ====> Epoch: 1 [2023-10-06 03:50:17] | (0:00:13.210963)
2023-10-06 03:50:18,450 MJV2 INFO ====> Epoch: 2 [2023-10-06 03:50:18] | (0:00:00.539312)
2023-10-06 03:50:18,940 MJV2 INFO ====> Epoch: 3 [2023-10-06 03:50:18] | (0:00:00.485277)
2023-10-06 03:50:19,435 MJV2 INFO ====> Epoch: 4 [2023-10-06 03:50:19] | (0:00:00.489769)
2023-10-06 03:50:19,952 MJV2 INFO ====> Epoch: 5 [2023-10-06 03:50:19] | (0:00:00.505072)
2023-10-06 03:50:20,535 MJV2 INFO ====> Epoch: 6 [2023-10-06 03:50:20] | (0:00:00.577565)
2023-10-06 03:50:21,114 MJV2 INFO ====> Epoch: 7 [2023-10-06 03:50:21] | (0:00:00.571026)
2023-10-06 03:50:21,723 MJV2 INFO ====> Epoch: 8 [2023-10-06 03:50:21] | (0:00:00.601737)
2023-10-06 03:50:22,325 MJV2 INFO ====> Epoch: 9 [2023-10-06 03:50:22] | (0:00:00.594052)
2023-10-06 03:50:22,978 MJV2 INFO ====> Epoch: 10 [2023-10-06 03:50:22] | (0:00:00.646191)
2023-10-06 03:50:23,624 MJV2 INFO ====> Epoch: 11 [2023-10-06 03:50:23] | (0:00:00.637738)
2023-10-06 03:50:24,261 MJV2 INFO ====> Epoch: 12 [2023-10-06 03:50:24] | (0:00:00.631217)
2023-10-06 03:50:24,886 MJV2 INFO ====> Epoch: 13 [2023-10-06 03:50:24] | (0:00:00.617479)
2023-10-06 03:50:25,533 MJV2 INFO ====> Epoch: 14 [2023-10-06 03:50:25] | (0:00:00.639175)
2023-10-06 03:50:26,161 MJV2 INFO ====> Epoch: 15 [2023-10-06 03:50:26] | (0:00:00.619377)
2023-10-06 03:50:26,819 MJV2 INFO ====> Epoch: 16 [2023-10-06 03:50:26] | (0:00:00.650853)
2023-10-06 03:50:27,484 MJV2 INFO ====> Epoch: 17 [2023-10-06 03:50:27] | (0:00:00.656293)
2023-10-06 03:50:28,089 MJV2 INFO ====> Epoch: 18 [2023-10-06 03:50:28] | (0:00:00.598655)
2023-10-06 03:50:28,626 MJV2 INFO ====> Epoch: 19 [2023-10-06 03:50:28] | (0:00:00.532480)
2023-10-06 03:50:29,157 MJV2 INFO Saving model and optimizer state at epoch 20 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:50:34,855 MJV2 INFO Saving model and optimizer state at epoch 20 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:50:38,736 MJV2 INFO saving ckpt MJV2_e20:Success.
2023-10-06 03:50:38,737 MJV2 INFO ====> Epoch: 20 [2023-10-06 03:50:38] | (0:00:10.105891)
2023-10-06 03:50:39,447 MJV2 INFO ====> Epoch: 21 [2023-10-06 03:50:39] | (0:00:00.701162)
2023-10-06 03:50:40,071 MJV2 INFO ====> Epoch: 22 [2023-10-06 03:50:40] | (0:00:00.616781)
2023-10-06 03:50:40,643 MJV2 INFO ====> Epoch: 23 [2023-10-06 03:50:40] | (0:00:00.560207)
2023-10-06 03:50:41,264 MJV2 INFO ====> Epoch: 24 [2023-10-06 03:50:41] | (0:00:00.614343)
2023-10-06 03:50:41,890 MJV2 INFO ====> Epoch: 25 [2023-10-06 03:50:41] | (0:00:00.618039)
2023-10-06 03:50:42,537 MJV2 INFO ====> Epoch: 26 [2023-10-06 03:50:42] | (0:00:00.638990)
2023-10-06 03:50:43,167 MJV2 INFO ====> Epoch: 27 [2023-10-06 03:50:43] | (0:00:00.621153)
2023-10-06 03:50:43,819 MJV2 INFO ====> Epoch: 28 [2023-10-06 03:50:43] | (0:00:00.644734)
2023-10-06 03:50:44,563 MJV2 INFO ====> Epoch: 29 [2023-10-06 03:50:44] | (0:00:00.735037)
2023-10-06 03:50:45,813 MJV2 INFO ====> Epoch: 30 [2023-10-06 03:50:45] | (0:00:01.214035)
2023-10-06 03:50:46,632 MJV2 INFO ====> Epoch: 31 [2023-10-06 03:50:46] | (0:00:00.808276)
2023-10-06 03:50:47,428 MJV2 INFO ====> Epoch: 32 [2023-10-06 03:50:47] | (0:00:00.786112)
2023-10-06 03:50:48,155 MJV2 INFO ====> Epoch: 33 [2023-10-06 03:50:48] | (0:00:00.718137)
2023-10-06 03:50:48,950 MJV2 INFO ====> Epoch: 34 [2023-10-06 03:50:48] | (0:00:00.787028)
2023-10-06 03:50:49,795 MJV2 INFO ====> Epoch: 35 [2023-10-06 03:50:49] | (0:00:00.835644)
2023-10-06 03:50:50,688 MJV2 INFO ====> Epoch: 36 [2023-10-06 03:50:50] | (0:00:00.883101)
2023-10-06 03:50:51,400 MJV2 INFO ====> Epoch: 37 [2023-10-06 03:50:51] | (0:00:00.700183)
2023-10-06 03:50:52,093 MJV2 INFO ====> Epoch: 38 [2023-10-06 03:50:52] | (0:00:00.684977)
2023-10-06 03:50:52,702 MJV2 INFO ====> Epoch: 39 [2023-10-06 03:50:52] | (0:00:00.604089)
2023-10-06 03:50:53,367 MJV2 INFO Saving model and optimizer state at epoch 40 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:51:00,311 MJV2 INFO Saving model and optimizer state at epoch 40 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:51:04,292 MJV2 INFO saving ckpt MJV2_e40:Success.
2023-10-06 03:51:04,292 MJV2 INFO ====> Epoch: 40 [2023-10-06 03:51:04] | (0:00:11.582100)
2023-10-06 03:51:04,910 MJV2 INFO ====> Epoch: 41 [2023-10-06 03:51:04] | (0:00:00.611901)
2023-10-06 03:51:05,430 MJV2 INFO ====> Epoch: 42 [2023-10-06 03:51:05] | (0:00:00.515362)
2023-10-06 03:51:05,960 MJV2 INFO ====> Epoch: 43 [2023-10-06 03:51:05] | (0:00:00.524792)
2023-10-06 03:51:06,496 MJV2 INFO ====> Epoch: 44 [2023-10-06 03:51:06] | (0:00:00.531301)
2023-10-06 03:51:07,028 MJV2 INFO ====> Epoch: 45 [2023-10-06 03:51:07] | (0:00:00.526814)
2023-10-06 03:51:07,565 MJV2 INFO ====> Epoch: 46 [2023-10-06 03:51:07] | (0:00:00.532899)
2023-10-06 03:51:08,100 MJV2 INFO ====> Epoch: 47 [2023-10-06 03:51:08] | (0:00:00.530152)
2023-10-06 03:51:08,649 MJV2 INFO ====> Epoch: 48 [2023-10-06 03:51:08] | (0:00:00.543636)
2023-10-06 03:51:09,182 MJV2 INFO ====> Epoch: 49 [2023-10-06 03:51:09] | (0:00:00.527989)
2023-10-06 03:51:09,714 MJV2 INFO ====> Epoch: 50 [2023-10-06 03:51:09] | (0:00:00.527200)
2023-10-06 03:51:10,281 MJV2 INFO ====> Epoch: 51 [2023-10-06 03:51:10] | (0:00:00.562587)
2023-10-06 03:51:11,008 MJV2 INFO ====> Epoch: 52 [2023-10-06 03:51:11] | (0:00:00.722259)
2023-10-06 03:51:11,806 MJV2 INFO ====> Epoch: 53 [2023-10-06 03:51:11] | (0:00:00.787369)
2023-10-06 03:51:12,535 MJV2 INFO ====> Epoch: 54 [2023-10-06 03:51:12] | (0:00:00.719133)
2023-10-06 03:51:13,213 MJV2 INFO ====> Epoch: 55 [2023-10-06 03:51:13] | (0:00:00.668560)
2023-10-06 03:51:13,814 MJV2 INFO ====> Epoch: 56 [2023-10-06 03:51:13] | (0:00:00.594812)
2023-10-06 03:51:14,550 MJV2 INFO ====> Epoch: 57 [2023-10-06 03:51:14] | (0:00:00.728585)
2023-10-06 03:51:15,360 MJV2 INFO ====> Epoch: 58 [2023-10-06 03:51:15] | (0:00:00.793917)
2023-10-06 03:51:15,973 MJV2 INFO ====> Epoch: 59 [2023-10-06 03:51:15] | (0:00:00.607627)
2023-10-06 03:51:17,197 MJV2 INFO Saving model and optimizer state at epoch 60 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:51:21,659 MJV2 INFO Saving model and optimizer state at epoch 60 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:51:25,817 MJV2 INFO saving ckpt MJV2_e60:Success.
2023-10-06 03:51:25,818 MJV2 INFO ====> Epoch: 60 [2023-10-06 03:51:25] | (0:00:09.836865)
2023-10-06 03:51:26,451 MJV2 INFO ====> Epoch: 61 [2023-10-06 03:51:26] | (0:00:00.625652)
2023-10-06 03:51:26,984 MJV2 INFO ====> Epoch: 62 [2023-10-06 03:51:26] | (0:00:00.528752)
2023-10-06 03:51:27,519 MJV2 INFO ====> Epoch: 63 [2023-10-06 03:51:27] | (0:00:00.529525)
2023-10-06 03:51:28,203 MJV2 INFO ====> Epoch: 64 [2023-10-06 03:51:28] | (0:00:00.679488)
2023-10-06 03:51:28,817 MJV2 INFO ====> Epoch: 65 [2023-10-06 03:51:28] | (0:00:00.605069)
2023-10-06 03:51:29,421 MJV2 INFO ====> Epoch: 66 [2023-10-06 03:51:29] | (0:00:00.597881)
2023-10-06 03:51:30,070 MJV2 INFO ====> Epoch: 67 [2023-10-06 03:51:30] | (0:00:00.642525)
2023-10-06 03:51:30,708 MJV2 INFO ====> Epoch: 68 [2023-10-06 03:51:30] | (0:00:00.632335)
2023-10-06 03:51:31,303 MJV2 INFO ====> Epoch: 69 [2023-10-06 03:51:31] | (0:00:00.587060)
2023-10-06 03:51:32,011 MJV2 INFO ====> Epoch: 70 [2023-10-06 03:51:32] | (0:00:00.701323)
2023-10-06 03:51:32,803 MJV2 INFO ====> Epoch: 71 [2023-10-06 03:51:32] | (0:00:00.779498)
2023-10-06 03:51:33,535 MJV2 INFO ====> Epoch: 72 [2023-10-06 03:51:33] | (0:00:00.722884)
2023-10-06 03:51:34,283 MJV2 INFO ====> Epoch: 73 [2023-10-06 03:51:34] | (0:00:00.739816)
2023-10-06 03:51:35,099 MJV2 INFO ====> Epoch: 74 [2023-10-06 03:51:35] | (0:00:00.799174)
2023-10-06 03:51:36,151 MJV2 INFO ====> Epoch: 75 [2023-10-06 03:51:36] | (0:00:01.039519)
2023-10-06 03:51:37,172 MJV2 INFO ====> Epoch: 76 [2023-10-06 03:51:37] | (0:00:01.005433)
2023-10-06 03:51:38,225 MJV2 INFO ====> Epoch: 77 [2023-10-06 03:51:38] | (0:00:01.040676)
2023-10-06 03:51:39,322 MJV2 INFO ====> Epoch: 78 [2023-10-06 03:51:39] | (0:00:01.082663)
2023-10-06 03:51:40,388 MJV2 INFO ====> Epoch: 79 [2023-10-06 03:51:40] | (0:00:01.044772)
2023-10-06 03:51:41,309 MJV2 INFO Saving model and optimizer state at epoch 80 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:51:46,761 MJV2 INFO Saving model and optimizer state at epoch 80 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:51:54,511 MJV2 INFO saving ckpt MJV2_e80:Success.
2023-10-06 03:51:54,512 MJV2 INFO ====> Epoch: 80 [2023-10-06 03:51:54] | (0:00:14.117232)
2023-10-06 03:51:55,186 MJV2 INFO ====> Epoch: 81 [2023-10-06 03:51:55] | (0:00:00.665237)
2023-10-06 03:51:55,804 MJV2 INFO ====> Epoch: 82 [2023-10-06 03:51:55] | (0:00:00.608887)
2023-10-06 03:51:56,422 MJV2 INFO ====> Epoch: 83 [2023-10-06 03:51:56] | (0:00:00.611605)
2023-10-06 03:51:57,034 MJV2 INFO ====> Epoch: 84 [2023-10-06 03:51:57] | (0:00:00.603133)
2023-10-06 03:51:57,735 MJV2 INFO ====> Epoch: 85 [2023-10-06 03:51:57] | (0:00:00.690840)
2023-10-06 03:51:59,189 MJV2 INFO ====> Epoch: 86 [2023-10-06 03:51:59] | (0:00:01.441537)
2023-10-06 03:52:00,082 MJV2 INFO ====> Epoch: 87 [2023-10-06 03:52:00] | (0:00:00.871573)
2023-10-06 03:52:01,017 MJV2 INFO ====> Epoch: 88 [2023-10-06 03:52:01] | (0:00:00.923933)
2023-10-06 03:52:01,642 MJV2 INFO ====> Epoch: 89 [2023-10-06 03:52:01] | (0:00:00.612630)
2023-10-06 03:52:02,263 MJV2 INFO ====> Epoch: 90 [2023-10-06 03:52:02] | (0:00:00.614190)
2023-10-06 03:52:02,906 MJV2 INFO ====> Epoch: 91 [2023-10-06 03:52:02] | (0:00:00.635062)
2023-10-06 03:52:03,471 MJV2 INFO ====> Epoch: 92 [2023-10-06 03:52:03] | (0:00:00.557012)
2023-10-06 03:52:04,042 MJV2 INFO ====> Epoch: 93 [2023-10-06 03:52:04] | (0:00:00.566189)
2023-10-06 03:52:04,637 MJV2 INFO ====> Epoch: 94 [2023-10-06 03:52:04] | (0:00:00.583105)
2023-10-06 03:52:05,203 MJV2 INFO ====> Epoch: 95 [2023-10-06 03:52:05] | (0:00:00.561535)
2023-10-06 03:52:05,890 MJV2 INFO ====> Epoch: 96 [2023-10-06 03:52:05] | (0:00:00.679752)
2023-10-06 03:52:06,455 MJV2 INFO ====> Epoch: 97 [2023-10-06 03:52:06] | (0:00:00.559880)
2023-10-06 03:52:07,007 MJV2 INFO ====> Epoch: 98 [2023-10-06 03:52:07] | (0:00:00.546814)
2023-10-06 03:52:07,558 MJV2 INFO ====> Epoch: 99 [2023-10-06 03:52:07] | (0:00:00.546054)
2023-10-06 03:52:08,143 MJV2 INFO Saving model and optimizer state at epoch 100 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:52:14,116 MJV2 INFO Saving model and optimizer state at epoch 100 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:52:24,697 MJV2 INFO saving ckpt MJV2_e100:Success.
2023-10-06 03:52:24,698 MJV2 INFO ====> Epoch: 100 [2023-10-06 03:52:24] | (0:00:17.136264)
2023-10-06 03:52:25,391 MJV2 INFO ====> Epoch: 101 [2023-10-06 03:52:25] | (0:00:00.679721)
2023-10-06 03:52:26,012 MJV2 INFO ====> Epoch: 102 [2023-10-06 03:52:26] | (0:00:00.613850)
2023-10-06 03:52:26,717 MJV2 INFO ====> Epoch: 103 [2023-10-06 03:52:26] | (0:00:00.699744)
2023-10-06 03:52:27,548 MJV2 INFO ====> Epoch: 104 [2023-10-06 03:52:27] | (0:00:00.822955)
2023-10-06 03:52:28,489 MJV2 INFO ====> Epoch: 105 [2023-10-06 03:52:28] | (0:00:00.925860)
2023-10-06 03:52:29,338 MJV2 INFO ====> Epoch: 106 [2023-10-06 03:52:29] | (0:00:00.836484)
2023-10-06 03:52:30,043 MJV2 INFO ====> Epoch: 107 [2023-10-06 03:52:30] | (0:00:00.691541)
2023-10-06 03:52:30,899 MJV2 INFO ====> Epoch: 108 [2023-10-06 03:52:30] | (0:00:00.843734)
2023-10-06 03:52:31,756 MJV2 INFO ====> Epoch: 109 [2023-10-06 03:52:31] | (0:00:00.846437)
2023-10-06 03:52:32,794 MJV2 INFO ====> Epoch: 110 [2023-10-06 03:52:32] | (0:00:01.028016)
2023-10-06 03:52:33,735 MJV2 INFO ====> Epoch: 111 [2023-10-06 03:52:33] | (0:00:00.932825)
2023-10-06 03:52:34,410 MJV2 INFO ====> Epoch: 112 [2023-10-06 03:52:34] | (0:00:00.668154)
2023-10-06 03:52:35,036 MJV2 INFO ====> Epoch: 113 [2023-10-06 03:52:35] | (0:00:00.618588)
2023-10-06 03:52:35,619 MJV2 INFO ====> Epoch: 114 [2023-10-06 03:52:35] | (0:00:00.576706)
2023-10-06 03:52:36,216 MJV2 INFO ====> Epoch: 115 [2023-10-06 03:52:36] | (0:00:00.590554)
2023-10-06 03:52:36,829 MJV2 INFO ====> Epoch: 116 [2023-10-06 03:52:36] | (0:00:00.605132)
2023-10-06 03:52:37,443 MJV2 INFO ====> Epoch: 117 [2023-10-06 03:52:37] | (0:00:00.607876)
2023-10-06 03:52:38,038 MJV2 INFO ====> Epoch: 118 [2023-10-06 03:52:38] | (0:00:00.586923)
2023-10-06 03:52:38,649 MJV2 INFO ====> Epoch: 119 [2023-10-06 03:52:38] | (0:00:00.605610)
2023-10-06 03:52:39,262 MJV2 INFO Saving model and optimizer state at epoch 120 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:52:44,533 MJV2 INFO Saving model and optimizer state at epoch 120 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:52:58,405 MJV2 INFO saving ckpt MJV2_e120:Success.
2023-10-06 03:52:58,408 MJV2 INFO ====> Epoch: 120 [2023-10-06 03:52:58] | (0:00:19.752815)
2023-10-06 03:52:59,169 MJV2 INFO ====> Epoch: 121 [2023-10-06 03:52:59] | (0:00:00.753933)
2023-10-06 03:52:59,879 MJV2 INFO ====> Epoch: 122 [2023-10-06 03:52:59] | (0:00:00.702880)
2023-10-06 03:53:00,663 MJV2 INFO ====> Epoch: 123 [2023-10-06 03:53:00] | (0:00:00.773274)
2023-10-06 03:53:01,556 MJV2 INFO ====> Epoch: 124 [2023-10-06 03:53:01] | (0:00:00.884808)
2023-10-06 03:53:02,377 MJV2 INFO ====> Epoch: 125 [2023-10-06 03:53:02] | (0:00:00.812450)
2023-10-06 03:53:03,142 MJV2 INFO ====> Epoch: 126 [2023-10-06 03:53:03] | (0:00:00.753508)
2023-10-06 03:53:03,844 MJV2 INFO ====> Epoch: 127 [2023-10-06 03:53:03] | (0:00:00.691548)
2023-10-06 03:53:04,487 MJV2 INFO ====> Epoch: 128 [2023-10-06 03:53:04] | (0:00:00.636253)
2023-10-06 03:53:05,506 MJV2 INFO ====> Epoch: 129 [2023-10-06 03:53:05] | (0:00:01.011236)
2023-10-06 03:53:06,566 MJV2 INFO ====> Epoch: 130 [2023-10-06 03:53:06] | (0:00:01.048733)
2023-10-06 03:53:07,309 MJV2 INFO ====> Epoch: 131 [2023-10-06 03:53:07] | (0:00:00.726694)
2023-10-06 03:53:07,885 MJV2 INFO ====> Epoch: 132 [2023-10-06 03:53:07] | (0:00:00.569697)
2023-10-06 03:53:08,504 MJV2 INFO ====> Epoch: 133 [2023-10-06 03:53:08] | (0:00:00.614753)
2023-10-06 03:53:09,109 MJV2 INFO ====> Epoch: 134 [2023-10-06 03:53:09] | (0:00:00.597936)
2023-10-06 03:53:09,678 MJV2 INFO ====> Epoch: 135 [2023-10-06 03:53:09] | (0:00:00.564048)
2023-10-06 03:53:10,280 MJV2 INFO ====> Epoch: 136 [2023-10-06 03:53:10] | (0:00:00.596084)
2023-10-06 03:53:10,928 MJV2 INFO ====> Epoch: 137 [2023-10-06 03:53:10] | (0:00:00.642712)
2023-10-06 03:53:11,547 MJV2 INFO ====> Epoch: 138 [2023-10-06 03:53:11] | (0:00:00.611499)
2023-10-06 03:53:12,150 MJV2 INFO ====> Epoch: 139 [2023-10-06 03:53:12] | (0:00:00.596271)
2023-10-06 03:53:12,750 MJV2 INFO Saving model and optimizer state at epoch 140 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:53:19,037 MJV2 INFO Saving model and optimizer state at epoch 140 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:53:34,706 MJV2 INFO saving ckpt MJV2_e140:Success.
2023-10-06 03:53:34,707 MJV2 INFO ====> Epoch: 140 [2023-10-06 03:53:34] | (0:00:22.552880)
2023-10-06 03:53:35,611 MJV2 INFO ====> Epoch: 141 [2023-10-06 03:53:35] | (0:00:00.894458)
2023-10-06 03:53:36,466 MJV2 INFO ====> Epoch: 142 [2023-10-06 03:53:36] | (0:00:00.834391)
2023-10-06 03:53:37,364 MJV2 INFO ====> Epoch: 143 [2023-10-06 03:53:37] | (0:00:00.887085)
2023-10-06 03:53:38,335 MJV2 INFO ====> Epoch: 144 [2023-10-06 03:53:38] | (0:00:00.961163)
2023-10-06 03:53:38,919 MJV2 INFO ====> Epoch: 145 [2023-10-06 03:53:38] | (0:00:00.576101)
2023-10-06 03:53:39,467 MJV2 INFO ====> Epoch: 146 [2023-10-06 03:53:39] | (0:00:00.543167)
2023-10-06 03:53:40,068 MJV2 INFO ====> Epoch: 147 [2023-10-06 03:53:40] | (0:00:00.595421)
2023-10-06 03:53:40,675 MJV2 INFO ====> Epoch: 148 [2023-10-06 03:53:40] | (0:00:00.599768)
2023-10-06 03:53:41,285 MJV2 INFO ====> Epoch: 149 [2023-10-06 03:53:41] | (0:00:00.603598)
2023-10-06 03:53:41,873 MJV2 INFO ====> Epoch: 150 [2023-10-06 03:53:41] | (0:00:00.582782)
2023-10-06 03:53:42,471 MJV2 INFO ====> Epoch: 151 [2023-10-06 03:53:42] | (0:00:00.591710)
2023-10-06 03:53:43,044 MJV2 INFO ====> Epoch: 152 [2023-10-06 03:53:43] | (0:00:00.566762)
2023-10-06 03:53:43,649 MJV2 INFO ====> Epoch: 153 [2023-10-06 03:53:43] | (0:00:00.600362)
2023-10-06 03:53:44,297 MJV2 INFO ====> Epoch: 154 [2023-10-06 03:53:44] | (0:00:00.640482)
2023-10-06 03:53:44,950 MJV2 INFO ====> Epoch: 155 [2023-10-06 03:53:44] | (0:00:00.645727)
2023-10-06 03:53:45,559 MJV2 INFO ====> Epoch: 156 [2023-10-06 03:53:45] | (0:00:00.602849)
2023-10-06 03:53:46,163 MJV2 INFO ====> Epoch: 157 [2023-10-06 03:53:46] | (0:00:00.592601)
2023-10-06 03:53:46,758 MJV2 INFO ====> Epoch: 158 [2023-10-06 03:53:46] | (0:00:00.588324)
2023-10-06 03:53:47,355 MJV2 INFO ====> Epoch: 159 [2023-10-06 03:53:47] | (0:00:00.591694)
2023-10-06 03:53:48,009 MJV2 INFO Saving model and optimizer state at epoch 160 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:53:51,031 MJV2 INFO Saving model and optimizer state at epoch 160 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:54:04,590 MJV2 INFO saving ckpt MJV2_e160:Success.
2023-10-06 03:54:04,590 MJV2 INFO ====> Epoch: 160 [2023-10-06 03:54:04] | (0:00:17.223768)
2023-10-06 03:54:05,412 MJV2 INFO ====> Epoch: 161 [2023-10-06 03:54:05] | (0:00:00.815681)
2023-10-06 03:54:06,371 MJV2 INFO ====> Epoch: 162 [2023-10-06 03:54:06] | (0:00:00.940540)
2023-10-06 03:54:07,297 MJV2 INFO ====> Epoch: 163 [2023-10-06 03:54:07] | (0:00:00.910478)
2023-10-06 03:54:07,997 MJV2 INFO ====> Epoch: 164 [2023-10-06 03:54:07] | (0:00:00.691057)
2023-10-06 03:54:08,684 MJV2 INFO ====> Epoch: 165 [2023-10-06 03:54:08] | (0:00:00.679572)
2023-10-06 03:54:09,316 MJV2 INFO ====> Epoch: 166 [2023-10-06 03:54:09] | (0:00:00.625722)
2023-10-06 03:54:09,945 MJV2 INFO ====> Epoch: 167 [2023-10-06 03:54:09] | (0:00:00.621918)
2023-10-06 03:54:10,621 MJV2 INFO ====> Epoch: 168 [2023-10-06 03:54:10] | (0:00:00.668251)
2023-10-06 03:54:11,274 MJV2 INFO ====> Epoch: 169 [2023-10-06 03:54:11] | (0:00:00.644905)
2023-10-06 03:54:11,883 MJV2 INFO ====> Epoch: 170 [2023-10-06 03:54:11] | (0:00:00.601653)
2023-10-06 03:54:12,429 MJV2 INFO ====> Epoch: 171 [2023-10-06 03:54:12] | (0:00:00.540736)
2023-10-06 03:54:12,985 MJV2 INFO ====> Epoch: 172 [2023-10-06 03:54:12] | (0:00:00.550989)
2023-10-06 03:54:13,534 MJV2 INFO ====> Epoch: 173 [2023-10-06 03:54:13] | (0:00:00.544413)
2023-10-06 03:54:14,111 MJV2 INFO ====> Epoch: 174 [2023-10-06 03:54:14] | (0:00:00.572376)
2023-10-06 03:54:14,781 MJV2 INFO ====> Epoch: 175 [2023-10-06 03:54:14] | (0:00:00.662149)
2023-10-06 03:54:15,370 MJV2 INFO ====> Epoch: 176 [2023-10-06 03:54:15] | (0:00:00.582636)
2023-10-06 03:54:15,959 MJV2 INFO ====> Epoch: 177 [2023-10-06 03:54:15] | (0:00:00.580516)
2023-10-06 03:54:16,542 MJV2 INFO ====> Epoch: 178 [2023-10-06 03:54:16] | (0:00:00.577851)
2023-10-06 03:54:17,187 MJV2 INFO ====> Epoch: 179 [2023-10-06 03:54:17] | (0:00:00.641118)
2023-10-06 03:54:17,913 MJV2 INFO Saving model and optimizer state at epoch 180 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:54:20,057 MJV2 INFO Saving model and optimizer state at epoch 180 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:54:34,201 MJV2 INFO saving ckpt MJV2_e180:Success.
2023-10-06 03:54:34,202 MJV2 INFO ====> Epoch: 180 [2023-10-06 03:54:34] | (0:00:17.006622)
2023-10-06 03:54:34,931 MJV2 INFO ====> Epoch: 181 [2023-10-06 03:54:34] | (0:00:00.719226)
2023-10-06 03:54:35,594 MJV2 INFO ====> Epoch: 182 [2023-10-06 03:54:35] | (0:00:00.654828)
2023-10-06 03:54:36,276 MJV2 INFO ====> Epoch: 183 [2023-10-06 03:54:36] | (0:00:00.673454)
2023-10-06 03:54:36,921 MJV2 INFO ====> Epoch: 184 [2023-10-06 03:54:36] | (0:00:00.637752)
2023-10-06 03:54:37,661 MJV2 INFO ====> Epoch: 185 [2023-10-06 03:54:37] | (0:00:00.731861)
2023-10-06 03:54:38,289 MJV2 INFO ====> Epoch: 186 [2023-10-06 03:54:38] | (0:00:00.619867)
2023-10-06 03:54:38,914 MJV2 INFO ====> Epoch: 187 [2023-10-06 03:54:38] | (0:00:00.618333)
2023-10-06 03:54:39,653 MJV2 INFO ====> Epoch: 188 [2023-10-06 03:54:39] | (0:00:00.730853)
2023-10-06 03:54:40,686 MJV2 INFO ====> Epoch: 189 [2023-10-06 03:54:40] | (0:00:01.022155)
2023-10-06 03:54:41,542 MJV2 INFO ====> Epoch: 190 [2023-10-06 03:54:41] | (0:00:00.840247)
2023-10-06 03:54:42,472 MJV2 INFO ====> Epoch: 191 [2023-10-06 03:54:42] | (0:00:00.919018)
2023-10-06 03:54:43,376 MJV2 INFO ====> Epoch: 192 [2023-10-06 03:54:43] | (0:00:00.892642)
2023-10-06 03:54:44,428 MJV2 INFO ====> Epoch: 193 [2023-10-06 03:54:44] | (0:00:01.044749)
2023-10-06 03:54:45,094 MJV2 INFO ====> Epoch: 194 [2023-10-06 03:54:45] | (0:00:00.658799)
2023-10-06 03:54:45,740 MJV2 INFO ====> Epoch: 195 [2023-10-06 03:54:45] | (0:00:00.637691)
2023-10-06 03:54:46,384 MJV2 INFO ====> Epoch: 196 [2023-10-06 03:54:46] | (0:00:00.639791)
2023-10-06 03:54:46,994 MJV2 INFO ====> Epoch: 197 [2023-10-06 03:54:46] | (0:00:00.604131)
2023-10-06 03:54:47,595 MJV2 INFO ====> Epoch: 198 [2023-10-06 03:54:47] | (0:00:00.593502)
2023-10-06 03:54:48,205 MJV2 INFO ====> Epoch: 199 [2023-10-06 03:54:48] | (0:00:00.602847)
2023-10-06 03:54:48,818 MJV2 INFO Saving model and optimizer state at epoch 200 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:54:54,252 MJV2 INFO Saving model and optimizer state at epoch 200 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:55:15,566 MJV2 INFO saving ckpt MJV2_e200:Success.
2023-10-06 03:55:15,570 MJV2 INFO ====> Epoch: 200 [2023-10-06 03:55:15] | (0:00:27.357656)
2023-10-06 03:55:16,646 MJV2 INFO Train Epoch: 201 [0%]
2023-10-06 03:55:16,647 MJV2 INFO [200, 9.753083879807726e-05]
2023-10-06 03:55:16,647 MJV2 INFO loss_disc=5.307, loss_gen=2.022, loss_fm=1.850,loss_mel=2.464, loss_kl=-0.004
2023-10-06 03:55:18,285 MJV2 INFO ====> Epoch: 201 [2023-10-06 03:55:18] | (0:00:02.702535)
2023-10-06 03:55:18,945 MJV2 INFO ====> Epoch: 202 [2023-10-06 03:55:18] | (0:00:00.643940)
2023-10-06 03:55:19,496 MJV2 INFO ====> Epoch: 203 [2023-10-06 03:55:19] | (0:00:00.546200)
2023-10-06 03:55:20,044 MJV2 INFO ====> Epoch: 204 [2023-10-06 03:55:20] | (0:00:00.543616)
2023-10-06 03:55:20,594 MJV2 INFO ====> Epoch: 205 [2023-10-06 03:55:20] | (0:00:00.544458)
2023-10-06 03:55:21,144 MJV2 INFO ====> Epoch: 206 [2023-10-06 03:55:21] | (0:00:00.545627)
2023-10-06 03:55:21,690 MJV2 INFO ====> Epoch: 207 [2023-10-06 03:55:21] | (0:00:00.540758)
2023-10-06 03:55:22,230 MJV2 INFO ====> Epoch: 208 [2023-10-06 03:55:22] | (0:00:00.536133)
2023-10-06 03:55:22,776 MJV2 INFO ====> Epoch: 209 [2023-10-06 03:55:22] | (0:00:00.540699)
2023-10-06 03:55:23,316 MJV2 INFO ====> Epoch: 210 [2023-10-06 03:55:23] | (0:00:00.535896)
2023-10-06 03:55:23,873 MJV2 INFO ====> Epoch: 211 [2023-10-06 03:55:23] | (0:00:00.551941)
2023-10-06 03:55:24,489 MJV2 INFO ====> Epoch: 212 [2023-10-06 03:55:24] | (0:00:00.611887)
2023-10-06 03:55:25,132 MJV2 INFO ====> Epoch: 213 [2023-10-06 03:55:25] | (0:00:00.635606)
2023-10-06 03:55:25,789 MJV2 INFO ====> Epoch: 214 [2023-10-06 03:55:25] | (0:00:00.649869)
2023-10-06 03:55:26,462 MJV2 INFO ====> Epoch: 215 [2023-10-06 03:55:26] | (0:00:00.666291)
2023-10-06 03:55:27,084 MJV2 INFO ====> Epoch: 216 [2023-10-06 03:55:27] | (0:00:00.614961)
2023-10-06 03:55:27,680 MJV2 INFO ====> Epoch: 217 [2023-10-06 03:55:27] | (0:00:00.590715)
2023-10-06 03:55:28,297 MJV2 INFO ====> Epoch: 218 [2023-10-06 03:55:28] | (0:00:00.609961)
2023-10-06 03:55:29,076 MJV2 INFO ====> Epoch: 219 [2023-10-06 03:55:29] | (0:00:00.774164)
2023-10-06 03:55:29,904 MJV2 INFO Saving model and optimizer state at epoch 220 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:55:40,216 MJV2 INFO Saving model and optimizer state at epoch 220 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:55:50,822 MJV2 INFO saving ckpt MJV2_e220:Success.
2023-10-06 03:55:50,825 MJV2 INFO ====> Epoch: 220 [2023-10-06 03:55:50] | (0:00:21.740222)
2023-10-06 03:55:51,594 MJV2 INFO ====> Epoch: 221 [2023-10-06 03:55:51] | (0:00:00.755274)
2023-10-06 03:55:52,252 MJV2 INFO ====> Epoch: 222 [2023-10-06 03:55:52] | (0:00:00.650103)
2023-10-06 03:55:52,873 MJV2 INFO ====> Epoch: 223 [2023-10-06 03:55:52] | (0:00:00.613572)
2023-10-06 03:55:53,508 MJV2 INFO ====> Epoch: 224 [2023-10-06 03:55:53] | (0:00:00.627875)
2023-10-06 03:55:54,112 MJV2 INFO ====> Epoch: 225 [2023-10-06 03:55:54] | (0:00:00.599724)
2023-10-06 03:55:54,714 MJV2 INFO ====> Epoch: 226 [2023-10-06 03:55:54] | (0:00:00.593971)
2023-10-06 03:55:55,264 MJV2 INFO ====> Epoch: 227 [2023-10-06 03:55:55] | (0:00:00.545647)
2023-10-06 03:55:55,807 MJV2 INFO ====> Epoch: 228 [2023-10-06 03:55:55] | (0:00:00.538006)
2023-10-06 03:55:56,435 MJV2 INFO ====> Epoch: 229 [2023-10-06 03:55:56] | (0:00:00.622771)
2023-10-06 03:55:57,066 MJV2 INFO ====> Epoch: 230 [2023-10-06 03:55:57] | (0:00:00.624770)
2023-10-06 03:55:57,756 MJV2 INFO ====> Epoch: 231 [2023-10-06 03:55:57] | (0:00:00.683317)
2023-10-06 03:55:58,446 MJV2 INFO ====> Epoch: 232 [2023-10-06 03:55:58] | (0:00:00.683250)
2023-10-06 03:55:59,151 MJV2 INFO ====> Epoch: 233 [2023-10-06 03:55:59] | (0:00:00.695991)
2023-10-06 03:56:00,160 MJV2 INFO ====> Epoch: 234 [2023-10-06 03:56:00] | (0:00:00.996514)
2023-10-06 03:56:00,985 MJV2 INFO ====> Epoch: 235 [2023-10-06 03:56:00] | (0:00:00.816494)
2023-10-06 03:56:02,025 MJV2 INFO ====> Epoch: 236 [2023-10-06 03:56:02] | (0:00:01.028410)
2023-10-06 03:56:03,145 MJV2 INFO ====> Epoch: 237 [2023-10-06 03:56:03] | (0:00:01.106992)
2023-10-06 03:56:04,001 MJV2 INFO ====> Epoch: 238 [2023-10-06 03:56:04] | (0:00:00.835461)
2023-10-06 03:56:04,674 MJV2 INFO ====> Epoch: 239 [2023-10-06 03:56:04] | (0:00:00.657481)
2023-10-06 03:56:05,306 MJV2 INFO Saving model and optimizer state at epoch 240 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:56:15,733 MJV2 INFO Saving model and optimizer state at epoch 240 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:56:25,590 MJV2 INFO saving ckpt MJV2_e240:Success.
2023-10-06 03:56:25,591 MJV2 INFO ====> Epoch: 240 [2023-10-06 03:56:25] | (0:00:20.909496)
2023-10-06 03:56:26,332 MJV2 INFO ====> Epoch: 241 [2023-10-06 03:56:26] | (0:00:00.729054)
2023-10-06 03:56:26,965 MJV2 INFO ====> Epoch: 242 [2023-10-06 03:56:26] | (0:00:00.626688)
2023-10-06 03:56:27,585 MJV2 INFO ====> Epoch: 243 [2023-10-06 03:56:27] | (0:00:00.613657)
2023-10-06 03:56:28,228 MJV2 INFO ====> Epoch: 244 [2023-10-06 03:56:28] | (0:00:00.638880)
2023-10-06 03:56:28,857 MJV2 INFO ====> Epoch: 245 [2023-10-06 03:56:28] | (0:00:00.621633)
2023-10-06 03:56:29,480 MJV2 INFO ====> Epoch: 246 [2023-10-06 03:56:29] | (0:00:00.616075)
2023-10-06 03:56:30,111 MJV2 INFO ====> Epoch: 247 [2023-10-06 03:56:30] | (0:00:00.625484)
2023-10-06 03:56:30,692 MJV2 INFO ====> Epoch: 248 [2023-10-06 03:56:30] | (0:00:00.573028)
2023-10-06 03:56:31,241 MJV2 INFO ====> Epoch: 249 [2023-10-06 03:56:31] | (0:00:00.544812)
2023-10-06 03:56:31,841 MJV2 INFO ====> Epoch: 250 [2023-10-06 03:56:31] | (0:00:00.595134)
2023-10-06 03:56:32,551 MJV2 INFO ====> Epoch: 251 [2023-10-06 03:56:32] | (0:00:00.700838)
2023-10-06 03:56:33,302 MJV2 INFO ====> Epoch: 252 [2023-10-06 03:56:33] | (0:00:00.744068)
2023-10-06 03:56:34,225 MJV2 INFO ====> Epoch: 253 [2023-10-06 03:56:34] | (0:00:00.914344)
2023-10-06 03:56:35,074 MJV2 INFO ====> Epoch: 254 [2023-10-06 03:56:35] | (0:00:00.835004)
2023-10-06 03:56:35,976 MJV2 INFO ====> Epoch: 255 [2023-10-06 03:56:35] | (0:00:00.886215)
2023-10-06 03:56:36,874 MJV2 INFO ====> Epoch: 256 [2023-10-06 03:56:36] | (0:00:00.887418)
2023-10-06 03:56:37,825 MJV2 INFO ====> Epoch: 257 [2023-10-06 03:56:37] | (0:00:00.939730)
2023-10-06 03:56:38,786 MJV2 INFO ====> Epoch: 258 [2023-10-06 03:56:38] | (0:00:00.942782)
2023-10-06 03:56:39,669 MJV2 INFO ====> Epoch: 259 [2023-10-06 03:56:39] | (0:00:00.875538)
2023-10-06 03:56:40,611 MJV2 INFO Saving model and optimizer state at epoch 260 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:56:42,480 MJV2 INFO Saving model and optimizer state at epoch 260 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:56:57,918 MJV2 INFO saving ckpt MJV2_e260:Success.
2023-10-06 03:56:57,919 MJV2 INFO ====> Epoch: 260 [2023-10-06 03:56:57] | (0:00:18.239052)
2023-10-06 03:56:58,627 MJV2 INFO ====> Epoch: 261 [2023-10-06 03:56:58] | (0:00:00.691500)
2023-10-06 03:56:59,222 MJV2 INFO ====> Epoch: 262 [2023-10-06 03:56:59] | (0:00:00.587209)
2023-10-06 03:56:59,819 MJV2 INFO ====> Epoch: 263 [2023-10-06 03:56:59] | (0:00:00.591587)
2023-10-06 03:57:00,400 MJV2 INFO ====> Epoch: 264 [2023-10-06 03:57:00] | (0:00:00.573412)
2023-10-06 03:57:00,949 MJV2 INFO ====> Epoch: 265 [2023-10-06 03:57:00] | (0:00:00.543658)
2023-10-06 03:57:01,498 MJV2 INFO ====> Epoch: 266 [2023-10-06 03:57:01] | (0:00:00.544362)
2023-10-06 03:57:02,057 MJV2 INFO ====> Epoch: 267 [2023-10-06 03:57:02] | (0:00:00.554105)
2023-10-06 03:57:02,607 MJV2 INFO ====> Epoch: 268 [2023-10-06 03:57:02] | (0:00:00.545201)
2023-10-06 03:57:03,160 MJV2 INFO ====> Epoch: 269 [2023-10-06 03:57:03] | (0:00:00.548308)
2023-10-06 03:57:03,712 MJV2 INFO ====> Epoch: 270 [2023-10-06 03:57:03] | (0:00:00.547160)
2023-10-06 03:57:04,264 MJV2 INFO ====> Epoch: 271 [2023-10-06 03:57:04] | (0:00:00.546357)
2023-10-06 03:57:04,811 MJV2 INFO ====> Epoch: 272 [2023-10-06 03:57:04] | (0:00:00.542045)
2023-10-06 03:57:05,356 MJV2 INFO ====> Epoch: 273 [2023-10-06 03:57:05] | (0:00:00.540888)
2023-10-06 03:57:05,905 MJV2 INFO ====> Epoch: 274 [2023-10-06 03:57:05] | (0:00:00.544092)
2023-10-06 03:57:06,500 MJV2 INFO ====> Epoch: 275 [2023-10-06 03:57:06] | (0:00:00.590528)
2023-10-06 03:57:07,145 MJV2 INFO ====> Epoch: 276 [2023-10-06 03:57:07] | (0:00:00.637404)
2023-10-06 03:57:08,002 MJV2 INFO ====> Epoch: 277 [2023-10-06 03:57:08] | (0:00:00.848961)
2023-10-06 03:57:08,943 MJV2 INFO ====> Epoch: 278 [2023-10-06 03:57:08] | (0:00:00.921772)
2023-10-06 03:57:09,899 MJV2 INFO ====> Epoch: 279 [2023-10-06 03:57:09] | (0:00:00.942280)
2023-10-06 03:57:10,780 MJV2 INFO Saving model and optimizer state at epoch 280 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:57:17,977 MJV2 INFO Saving model and optimizer state at epoch 280 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:57:33,219 MJV2 INFO saving ckpt MJV2_e280:Success.
2023-10-06 03:57:33,221 MJV2 INFO ====> Epoch: 280 [2023-10-06 03:57:33] | (0:00:23.314079)
2023-10-06 03:57:33,921 MJV2 INFO ====> Epoch: 281 [2023-10-06 03:57:33] | (0:00:00.688499)
2023-10-06 03:57:34,534 MJV2 INFO ====> Epoch: 282 [2023-10-06 03:57:34] | (0:00:00.602221)
2023-10-06 03:57:35,168 MJV2 INFO ====> Epoch: 283 [2023-10-06 03:57:35] | (0:00:00.628034)
2023-10-06 03:57:35,796 MJV2 INFO ====> Epoch: 284 [2023-10-06 03:57:35] | (0:00:00.620763)
2023-10-06 03:57:36,394 MJV2 INFO ====> Epoch: 285 [2023-10-06 03:57:36] | (0:00:00.591844)
2023-10-06 03:57:36,943 MJV2 INFO ====> Epoch: 286 [2023-10-06 03:57:36] | (0:00:00.544362)
2023-10-06 03:57:37,489 MJV2 INFO ====> Epoch: 287 [2023-10-06 03:57:37] | (0:00:00.541447)
2023-10-06 03:57:38,049 MJV2 INFO ====> Epoch: 288 [2023-10-06 03:57:38] | (0:00:00.555393)
2023-10-06 03:57:38,654 MJV2 INFO ====> Epoch: 289 [2023-10-06 03:57:38] | (0:00:00.599946)
2023-10-06 03:57:39,299 MJV2 INFO ====> Epoch: 290 [2023-10-06 03:57:39] | (0:00:00.636895)
2023-10-06 03:57:39,926 MJV2 INFO ====> Epoch: 291 [2023-10-06 03:57:39] | (0:00:00.619722)
2023-10-06 03:57:40,549 MJV2 INFO ====> Epoch: 292 [2023-10-06 03:57:40] | (0:00:00.615366)
2023-10-06 03:57:41,203 MJV2 INFO ====> Epoch: 293 [2023-10-06 03:57:41] | (0:00:00.647124)
2023-10-06 03:57:41,829 MJV2 INFO ====> Epoch: 294 [2023-10-06 03:57:41] | (0:00:00.617758)
2023-10-06 03:57:42,548 MJV2 INFO ====> Epoch: 295 [2023-10-06 03:57:42] | (0:00:00.711541)
2023-10-06 03:57:43,594 MJV2 INFO ====> Epoch: 296 [2023-10-06 03:57:43] | (0:00:01.035691)
2023-10-06 03:57:44,518 MJV2 INFO ====> Epoch: 297 [2023-10-06 03:57:44] | (0:00:00.916965)
2023-10-06 03:57:45,387 MJV2 INFO ====> Epoch: 298 [2023-10-06 03:57:45] | (0:00:00.860600)
2023-10-06 03:57:46,124 MJV2 INFO ====> Epoch: 299 [2023-10-06 03:57:46] | (0:00:00.728063)
2023-10-06 03:57:46,744 MJV2 INFO Saving model and optimizer state at epoch 300 to ./logs/MJV2/G_2333333.pth
2023-10-06 03:57:50,985 MJV2 INFO Saving model and optimizer state at epoch 300 to ./logs/MJV2/D_2333333.pth
2023-10-06 03:58:03,341 MJV2 INFO saving ckpt MJV2_e300:Success.
2023-10-06 03:58:03,343 MJV2 INFO ====> Epoch: 300 [2023-10-06 03:58:03] | (0:00:17.211895)
2023-10-06 03:58:03,344 MJV2 INFO Training is done. The program is closed.
2023-10-06 03:58:07,679 MJV2 INFO saving final ckpt:Success.
|