title: Sudden reinitialization of train and validation loss and slow covergence while training CNN in Pytorch
body: <p>I am a beginner in deep learning, especially in designing neural networks from scratch and I am trying to build a B/W image colorizer using the Autoencoder concept in Pytorch while referring this Keras architecture from <a href="https://blog.floydhub.com/colorizing-b-w-photos-with-neural-networks/" rel="nofollow noreferrer">here</a> using the dataset <a href="https://www.floydhub.com/emilwallner/datasets/colornet/2/images" rel="nofollow noreferrer">here</a>. The Keras architecture for reference:</p>
  <pre><code>#Design the neural network
  model = Sequential()
  model.add(InputLayer(input_shape=(256, 256, 1)))
  model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
  model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=2))
  model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
  model.add(Conv2D(128, (3, 3), activation='relu', padding='same', strides=2))
  model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
  model.add(Conv2D(256, (3, 3), activation='relu', padding='same', strides=2))
  model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
  model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
  model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
  model.add(UpSampling2D((2, 2)))
  model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
  model.add(UpSampling2D((2, 2)))
  model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
  model.add(Conv2D(2, (3, 3), activation='tanh', padding='same'))
  model.add(UpSampling2D((2, 2)))
  
  # Finish model
  model.compile(optimizer='rmsprop', loss='mse')
  
  # Image transformer
  datagen = ImageDataGenerator(
      shear_range=0.2,
      zoom_range=0.2,
      rotation_range=20,
      horizontal_flip=True)
  
  # Generate training data
  batch_size = 50
  
  def image_a_b_gen(batch_size):
      for batch in datagen.flow(Xtrain, batch_size=batch_size):
          lab_batch = rgb2lab(batch)
          X_batch = lab_batch[:,:,:,0]
          Y_batch = lab_batch[:,:,:,1:] / 128
          yield (X_batch.reshape(X_batch.shape+(1,)), Y_batch)
  
  # Train model
  TensorBoard(log_dir='/output')
  model.fit_generator(image_a_b_gen(batch_size), steps_per_epoch=10000, epochs=1)
  </code></pre>
  <p>And this is the architecture that I have built in PyTorch:</p>
  <pre><code>class AutoEncoder(nn.Module):
      def __init__(self):
          super().__init__()
  
          self.model = nn.Sequential(
          nn.Conv2d(1, 64, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Conv2d(64, 128, (3,3), padding = (1,1), stride = (2,2)),
          nn.ReLU(),
          nn.Conv2d(128, 128, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Conv2d(128, 256, (3,3), padding = (1,1), stride = (2,2)),
          nn.ReLU(),
          nn.Conv2d(256, 256, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Conv2d(256, 512, (3,3), padding = (1,1), stride = (2,2)),
          nn.ReLU(),
          nn.Conv2d(512, 512, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Conv2d(512, 256, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Conv2d(256, 128, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Upsample(scale_factor= (2,2)),
          nn.Conv2d(128, 64, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Upsample(scale_factor= (2,2)),
          nn.Conv2d(64, 32, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Conv2d(32, 2, (3,3), padding = (1,1)),
          nn.ReLU(),
          nn.Upsample(scale_factor= (2,2)),       
  )
  
    def forward(self, input):
         output = self.model(input)
         return output
  </code></pre>
  <p>And initialize like this:</p>
  <pre><code>model = AutoEncoder()
  loss_func = nn.MSELoss()
  optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
  </code></pre>
  <p>This is the training function (trained on Google Colab) with validation incorporated:</p>
  <pre><code>  def train(model, optimizer, loss_func, train_loader, val_loader, epochs, device = 'cuda'):
    model = model.to(device)
    loss_func = loss_func.to(device)
    training_loss = 0
    valid_loss = 0
    save_path = {'grayscale': '/content/outputs/gray/', 'colorized': '/content/outputs/color/',
                 'target': '/content/outputs/target/'}
  
    for epoch in range(e+1, e+1+epochs):
      print('Starting epoch {}'.format(epoch))
      epoch_start = time.time()
      training_loss = 0.0
      valid_loss = 0.0
  
      model.train()
      b = 1
      for batch in train_loader:
        batch_start = time.time()
        optimizer.zero_grad()
        img_gray, img_ab, og_image = batch
        input, target = img_gray, img_ab
        input, target = input.to(device), target.to(device)
        output = model(input)
        loss = loss_func(output, target)
        loss.backward()
        optimizer.step()
  
        training_loss += loss.data.item()
        if b % 20 == 0:
          print('Epoch: {} , Batch: {} Training Loss: {:.4f}, Time: {:.2f}'.format(
          epoch, b, loss.data.item(), time.time() - batch_start))
        b += 1
  
      training_loss /= len(train_loader)
  
      model.eval()
      b = 1
      for batch in val_loader:
        batch_start = time.time()
        img_gray, img_ab, og_image = batch
        input, target = img_gray, img_ab
        input, target = input.to(device), target.to(device)
        output = model(input)
        loss = loss_func(output, target)
  
        valid_loss += loss.data.item()
  
        j = random.randint(1,9)
        save_name = 'epoch-{}batch-{}img-{}.jpg'.format(epoch, b, j)
        save_image(input[j].detach().cpu(), output[j].detach().cpu(), target[j].detach().cpu(),
                   og_image[j].detach().cpu(), save_path=save_path, save_name=save_name)
          
        print('Epoch: {} , Batch: {} Valid Loss: {:.4f}, Time: {:.2f}'.format(
        epoch, b, loss.data.item(), time.time() - batch_start))
  
        b += 1
  
      valid_loss /= len(val_loader)
  
      # model saving code here
    
      print('Epoch: {} , Training Loss: {:.4f}, Validation Loss: {:.4f}, Time: {:.2f}'
      .format(epoch, training_loss, valid_loss, time.time() - epoch_start))
  </code></pre>
  <p>However, on training this model, I found that the convergence speed was slower than I expected and also the training and validation loss shoots up suddenly (reinitializes to loss that was there at epoch 1) at a certain epoch (epoch 40) as seen in the logs below and then increases. I cannot ascertain the reason for this happening even though I did search the net for it. I did overfit the model on a single image to check if the model is able to learn images, and it gives great results after about 200 epochs on that single image.</p>
  <p>Any other suggestions to improve training speed are also welcome.</p>
  <p>The logs:</p>
  <pre><code>Starting epoch 35
  Epoch: 35 , Batch: 20 Training Loss: 106.5381, Time: 0.68
  Epoch: 35 , Batch: 40 Training Loss: 164.4004, Time: 0.67
  Epoch: 35 , Batch: 60 Training Loss: 187.2683, Time: 0.67
  Epoch: 35 , Batch: 80 Training Loss: 125.9881, Time: 0.68
  Epoch: 35 , Batch: 100 Training Loss: 178.2198, Time: 0.67
  Epoch: 35 , Batch: 120 Training Loss: 145.3067, Time: 0.68
  Epoch: 35 , Batch: 140 Training Loss: 172.2734, Time: 0.67
  Epoch: 35 , Batch: 160 Training Loss: 161.0511, Time: 0.68
  Epoch: 35 , Batch: 180 Training Loss: 154.7270, Time: 0.67
  Epoch: 35 , Batch: 200 Training Loss: 105.7089, Time: 0.67
  Epoch: 35 , Batch: 220 Training Loss: 136.7364, Time: 0.67
  Epoch: 35 , Batch: 240 Training Loss: 144.3123, Time: 0.67
  Epoch: 35 , Batch: 1 Valid Loss: 150.3127, Time: 0.36
  Epoch: 35 , Batch: 2 Valid Loss: 136.5788, Time: 0.36
  Epoch: 35 , Batch: 3 Valid Loss: 145.5104, Time: 0.37
  Epoch: 35 , Batch: 4 Valid Loss: 174.9318, Time: 0.35
  Epoch: 35 , Batch: 5 Valid Loss: 169.0320, Time: 0.36
  Epoch: 35 , Batch: 6 Valid Loss: 104.9420, Time: 0.36
  Epoch: 35 , Batch: 7 Valid Loss: 139.7437, Time: 0.37
  Epoch: 35 , Batch: 8 Valid Loss: 142.1164, Time: 0.37
  Epoch: 35 , Batch: 9 Valid Loss: 162.5985, Time: 0.35
  Epoch: 35 , Batch: 10 Valid Loss: 195.1854, Time: 0.35
  Epoch: 35 , Batch: 11 Valid Loss: 134.6287, Time: 0.35
  Epoch: 35 , Batch: 12 Valid Loss: 136.0625, Time: 0.35
  Epoch: 35 , Batch: 13 Valid Loss: 210.8838, Time: 0.35
  Epoch: 35 , Batch: 14 Valid Loss: 171.1289, Time: 0.35
  Epoch: 35 , Batch: 15 Valid Loss: 158.6969, Time: 0.34
  Epoch: 35 , Batch: 16 Valid Loss: 245.2010, Time: 0.36
  Epoch: 35 , Batch: 17 Valid Loss: 163.9438, Time: 0.36
  Epoch: 35 , Batch: 18 Valid Loss: 148.4983, Time: 0.36
  Epoch: 35 , Batch: 19 Valid Loss: 153.7577, Time: 0.35
  Epoch: 35 , Batch: 20 Valid Loss: 180.0079, Time: 0.34
  Epoch: 35 , Batch: 21 Valid Loss: 123.2093, Time: 0.36
  Epoch: 35 , Batch: 22 Valid Loss: 125.0577, Time: 0.36
  Epoch: 35 , Batch: 23 Valid Loss: 167.9037, Time: 0.35
  Epoch: 35 , Batch: 24 Valid Loss: 142.5038, Time: 0.35
  Epoch: 35 , Batch: 25 Valid Loss: 204.0228, Time: 0.36
  Epoch: 35 , Batch: 26 Valid Loss: 138.4986, Time: 0.31
  Epoch: 35 , Batch: 27 Valid Loss: 245.4016, Time: 0.35
  Epoch: 35 , Batch: 28 Valid Loss: 147.9031, Time: 0.35
  Epoch: 35 , Batch: 29 Valid Loss: 86.9302, Time: 0.35
  Epoch: 35 , Batch: 30 Valid Loss: 144.6022, Time: 0.35
  Epoch: 35 , Batch: 31 Valid Loss: 171.0132, Time: 0.36
  Epoch: 35 , Batch: 32 Valid Loss: 171.6215, Time: 0.26
  
  Epoch: 35 , Training Loss: 158.2567, Validation Loss: 159.1384, Time: 419.94
  
  Starting epoch 36
  Epoch: 36 , Batch: 20 Training Loss: 105.9032, Time: 0.68
  Epoch: 36 , Batch: 40 Training Loss: 166.2677, Time: 0.67
  Epoch: 36 , Batch: 60 Training Loss: 187.5138, Time: 0.68
  Epoch: 36 , Batch: 80 Training Loss: 126.4907, Time: 0.67
  Epoch: 36 , Batch: 100 Training Loss: 177.6801, Time: 0.67
  Epoch: 36 , Batch: 120 Training Loss: 145.7679, Time: 0.67
  Epoch: 36 , Batch: 140 Training Loss: 172.1092, Time: 0.67
  Epoch: 36 , Batch: 160 Training Loss: 161.7235, Time: 0.67
  Epoch: 36 , Batch: 180 Training Loss: 154.2764, Time: 0.68
  Epoch: 36 , Batch: 200 Training Loss: 105.0589, Time: 0.67
  Epoch: 36 , Batch: 220 Training Loss: 137.1901, Time: 0.67
  Epoch: 36 , Batch: 240 Training Loss: 143.7092, Time: 0.68
  Epoch: 36 , Batch: 1 Valid Loss: 151.7165, Time: 0.36
  Epoch: 36 , Batch: 2 Valid Loss: 137.6912, Time: 0.35
  Epoch: 36 , Batch: 3 Valid Loss: 148.1911, Time: 0.36
  Epoch: 36 , Batch: 4 Valid Loss: 178.2061, Time: 0.36
  Epoch: 36 , Batch: 5 Valid Loss: 171.2887, Time: 0.35
  Epoch: 36 , Batch: 6 Valid Loss: 105.7988, Time: 0.36
  Epoch: 36 , Batch: 7 Valid Loss: 139.2858, Time: 0.35
  Epoch: 36 , Batch: 8 Valid Loss: 143.3132, Time: 0.36
  Epoch: 36 , Batch: 9 Valid Loss: 164.0428, Time: 0.35
  Epoch: 36 , Batch: 10 Valid Loss: 197.8514, Time: 0.35
  Epoch: 36 , Batch: 11 Valid Loss: 135.2920, Time: 0.36
  Epoch: 36 , Batch: 12 Valid Loss: 136.8798, Time: 0.37
  Epoch: 36 , Batch: 13 Valid Loss: 213.7531, Time: 0.36
  Epoch: 36 , Batch: 14 Valid Loss: 173.2047, Time: 0.34
  Epoch: 36 , Batch: 15 Valid Loss: 159.9836, Time: 0.36
  Epoch: 36 , Batch: 16 Valid Loss: 250.1139, Time: 0.36
  Epoch: 36 , Batch: 17 Valid Loss: 166.5989, Time: 0.36
  Epoch: 36 , Batch: 18 Valid Loss: 149.8430, Time: 0.35
  Epoch: 36 , Batch: 19 Valid Loss: 156.4111, Time: 0.32
  Epoch: 36 , Batch: 20 Valid Loss: 182.4751, Time: 0.36
  Epoch: 36 , Batch: 21 Valid Loss: 125.2163, Time: 0.36
  Epoch: 36 , Batch: 22 Valid Loss: 127.7892, Time: 0.36
  Epoch: 36 , Batch: 23 Valid Loss: 171.6647, Time: 0.35
  Epoch: 36 , Batch: 24 Valid Loss: 145.8683, Time: 0.36
  Epoch: 36 , Batch: 25 Valid Loss: 205.0257, Time: 0.35
  Epoch: 36 , Batch: 26 Valid Loss: 141.0681, Time: 0.32
  Epoch: 36 , Batch: 27 Valid Loss: 248.7428, Time: 0.36
  Epoch: 36 , Batch: 28 Valid Loss: 148.8966, Time: 0.36
  Epoch: 36 , Batch: 29 Valid Loss: 88.4646, Time: 0.36
  Epoch: 36 , Batch: 30 Valid Loss: 143.6590, Time: 0.35
  Epoch: 36 , Batch: 31 Valid Loss: 173.3025, Time: 0.35
  Epoch: 36 , Batch: 32 Valid Loss: 174.2860, Time: 0.24
  
  Epoch: 36 , Training Loss: 158.3800, Validation Loss: 161.1226, Time: 419.45
  
  Starting epoch 37
  Epoch: 37 , Batch: 20 Training Loss: 105.6317, Time: 0.68
  Epoch: 37 , Batch: 40 Training Loss: 165.0392, Time: 0.67
  Epoch: 37 , Batch: 60 Training Loss: 188.5125, Time: 0.68
  Epoch: 37 , Batch: 80 Training Loss: 126.0623, Time: 0.67
  Epoch: 37 , Batch: 100 Training Loss: 177.5551, Time: 0.67
  Epoch: 37 , Batch: 120 Training Loss: 145.3477, Time: 0.67
  Epoch: 37 , Batch: 140 Training Loss: 175.2243, Time: 0.68
  Epoch: 37 , Batch: 160 Training Loss: 160.9049, Time: 0.68
  Epoch: 37 , Batch: 180 Training Loss: 153.3031, Time: 0.68
  Epoch: 37 , Batch: 200 Training Loss: 104.0302, Time: 0.67
  Epoch: 37 , Batch: 220 Training Loss: 135.7108, Time: 0.67
  Epoch: 37 , Batch: 240 Training Loss: 142.8567, Time: 0.67
  Epoch: 37 , Batch: 1 Valid Loss: 150.1673, Time: 0.36
  Epoch: 37 , Batch: 2 Valid Loss: 136.7532, Time: 0.36
  Epoch: 37 , Batch: 3 Valid Loss: 144.9570, Time: 0.36
  Epoch: 37 , Batch: 4 Valid Loss: 173.6571, Time: 0.32
  Epoch: 37 , Batch: 5 Valid Loss: 166.9340, Time: 0.35
  Epoch: 37 , Batch: 6 Valid Loss: 104.1471, Time: 0.35
  Epoch: 37 , Batch: 7 Valid Loss: 138.5468, Time: 0.36
  Epoch: 37 , Batch: 8 Valid Loss: 141.9434, Time: 0.35
  Epoch: 37 , Batch: 9 Valid Loss: 163.1604, Time: 0.35
  Epoch: 37 , Batch: 10 Valid Loss: 195.1403, Time: 0.35
  Epoch: 37 , Batch: 11 Valid Loss: 134.0441, Time: 0.36
  Epoch: 37 , Batch: 12 Valid Loss: 135.6687, Time: 0.36
  Epoch: 37 , Batch: 13 Valid Loss: 210.3593, Time: 0.36
  Epoch: 37 , Batch: 14 Valid Loss: 170.5062, Time: 0.35
  Epoch: 37 , Batch: 15 Valid Loss: 157.5619, Time: 0.35
  Epoch: 37 , Batch: 16 Valid Loss: 243.9199, Time: 0.35
  Epoch: 37 , Batch: 17 Valid Loss: 163.0798, Time: 0.36
  Epoch: 37 , Batch: 18 Valid Loss: 147.0687, Time: 0.36
  Epoch: 37 , Batch: 19 Valid Loss: 152.7307, Time: 0.36
  Epoch: 37 , Batch: 20 Valid Loss: 179.4877, Time: 0.34
  Epoch: 37 , Batch: 21 Valid Loss: 122.9611, Time: 0.35
  Epoch: 37 , Batch: 22 Valid Loss: 124.0443, Time: 0.35
  Epoch: 37 , Batch: 23 Valid Loss: 165.3088, Time: 0.36
  Epoch: 37 , Batch: 24 Valid Loss: 142.3539, Time: 0.35
  Epoch: 37 , Batch: 25 Valid Loss: 202.2605, Time: 0.35
  Epoch: 37 , Batch: 26 Valid Loss: 137.7361, Time: 0.35
  Epoch: 37 , Batch: 27 Valid Loss: 242.8415, Time: 0.35
  Epoch: 37 , Batch: 28 Valid Loss: 148.3895, Time: 0.35
  Epoch: 37 , Batch: 29 Valid Loss: 86.0574, Time: 0.36
  Epoch: 37 , Batch: 30 Valid Loss: 144.1157, Time: 0.35
  Epoch: 37 , Batch: 31 Valid Loss: 170.8322, Time: 0.35
  Epoch: 37 , Batch: 32 Valid Loss: 171.6427, Time: 0.25
  
  Epoch: 37 , Training Loss: 158.1220, Validation Loss: 158.3868, Time: 417.97
  
  Starting epoch 38
  Epoch: 38 , Batch: 20 Training Loss: 105.7252, Time: 0.68
  Epoch: 38 , Batch: 40 Training Loss: 163.4334, Time: 0.67
  Epoch: 38 , Batch: 60 Training Loss: 185.7129, Time: 0.68
  Epoch: 38 , Batch: 80 Training Loss: 125.1632, Time: 0.68
  Epoch: 38 , Batch: 100 Training Loss: 178.0677, Time: 0.67
  Epoch: 38 , Batch: 120 Training Loss: 145.0643, Time: 0.67
  Epoch: 38 , Batch: 140 Training Loss: 172.1782, Time: 0.67
  Epoch: 38 , Batch: 160 Training Loss: 160.3347, Time: 0.68
  Epoch: 38 , Batch: 180 Training Loss: 151.9679, Time: 0.68
  Epoch: 38 , Batch: 200 Training Loss: 103.2668, Time: 0.67
  Epoch: 38 , Batch: 220 Training Loss: 134.6251, Time: 0.67
  Epoch: 38 , Batch: 240 Training Loss: 141.8546, Time: 0.67
  Epoch: 38 , Batch: 1 Valid Loss: 150.4089, Time: 0.35
  Epoch: 38 , Batch: 2 Valid Loss: 136.9565, Time: 0.36
  Epoch: 38 , Batch: 3 Valid Loss: 144.5437, Time: 0.35
  Epoch: 38 , Batch: 4 Valid Loss: 172.9728, Time: 0.36
  Epoch: 38 , Batch: 5 Valid Loss: 166.5804, Time: 0.35
  Epoch: 38 , Batch: 6 Valid Loss: 103.4765, Time: 0.35
  Epoch: 38 , Batch: 7 Valid Loss: 138.3381, Time: 0.35
  Epoch: 38 , Batch: 8 Valid Loss: 141.9496, Time: 0.35
  Epoch: 38 , Batch: 9 Valid Loss: 163.2548, Time: 0.35
  Epoch: 38 , Batch: 10 Valid Loss: 194.6338, Time: 0.36
  Epoch: 38 , Batch: 11 Valid Loss: 134.2889, Time: 0.35
  Epoch: 38 , Batch: 12 Valid Loss: 136.0695, Time: 0.35
  Epoch: 38 , Batch: 13 Valid Loss: 209.8143, Time: 0.36
  Epoch: 38 , Batch: 14 Valid Loss: 170.3180, Time: 0.35
  Epoch: 38 , Batch: 15 Valid Loss: 157.3526, Time: 0.35
  Epoch: 38 , Batch: 16 Valid Loss: 243.4948, Time: 0.36
  Epoch: 38 , Batch: 17 Valid Loss: 162.8065, Time: 0.36
  Epoch: 38 , Batch: 18 Valid Loss: 146.0056, Time: 0.36
  Epoch: 38 , Batch: 19 Valid Loss: 152.3638, Time: 0.35
  Epoch: 38 , Batch: 20 Valid Loss: 179.5833, Time: 0.34
  Epoch: 38 , Batch: 21 Valid Loss: 122.8715, Time: 0.36
  Epoch: 38 , Batch: 22 Valid Loss: 123.2680, Time: 0.36
  Epoch: 38 , Batch: 23 Valid Loss: 164.3962, Time: 0.36
  Epoch: 38 , Batch: 24 Valid Loss: 142.2115, Time: 0.33
  Epoch: 38 , Batch: 25 Valid Loss: 202.0091, Time: 0.35
  Epoch: 38 , Batch: 26 Valid Loss: 137.4782, Time: 0.36
  Epoch: 38 , Batch: 27 Valid Loss: 241.4298, Time: 0.35
  Epoch: 38 , Batch: 28 Valid Loss: 148.7050, Time: 0.35
  Epoch: 38 , Batch: 29 Valid Loss: 86.0582, Time: 0.34
  Epoch: 38 , Batch: 30 Valid Loss: 144.0733, Time: 0.33
  Epoch: 38 , Batch: 31 Valid Loss: 170.6064, Time: 0.36
  Epoch: 38 , Batch: 32 Valid Loss: 171.6329, Time: 0.25
  
  Epoch: 38 , Training Loss: 157.1817, Validation Loss: 158.1235, Time: 418.16
  
  Starting epoch 39
  Epoch: 39 , Batch: 20 Training Loss: 105.1230, Time: 0.68
  Epoch: 39 , Batch: 40 Training Loss: 165.4860, Time: 0.67
  Epoch: 39 , Batch: 60 Training Loss: 185.0349, Time: 0.67
  Epoch: 39 , Batch: 80 Training Loss: 124.9884, Time: 0.67
  Epoch: 39 , Batch: 100 Training Loss: 176.9023, Time: 0.67
  Epoch: 39 , Batch: 120 Training Loss: 144.6198, Time: 0.67
  Epoch: 39 , Batch: 140 Training Loss: 170.2020, Time: 0.67
  Epoch: 39 , Batch: 160 Training Loss: 158.3938, Time: 0.67
  Epoch: 39 , Batch: 180 Training Loss: 151.8574, Time: 0.68
  Epoch: 39 , Batch: 200 Training Loss: 103.9747, Time: 0.67
  Epoch: 39 , Batch: 220 Training Loss: 134.2870, Time: 0.67
  Epoch: 39 , Batch: 240 Training Loss: 141.7696, Time: 0.68
  Epoch: 39 , Batch: 1 Valid Loss: 151.1508, Time: 0.34
  Epoch: 39 , Batch: 2 Valid Loss: 136.8730, Time: 0.35
  Epoch: 39 , Batch: 3 Valid Loss: 145.3472, Time: 0.36
  Epoch: 39 , Batch: 4 Valid Loss: 173.3746, Time: 0.36
  Epoch: 39 , Batch: 5 Valid Loss: 167.3553, Time: 0.35
  Epoch: 39 , Batch: 6 Valid Loss: 103.7990, Time: 0.36
  Epoch: 39 , Batch: 7 Valid Loss: 139.3748, Time: 0.36
  Epoch: 39 , Batch: 8 Valid Loss: 142.6575, Time: 0.35
  Epoch: 39 , Batch: 9 Valid Loss: 163.5934, Time: 0.35
  Epoch: 39 , Batch: 10 Valid Loss: 194.6525, Time: 0.35
  Epoch: 39 , Batch: 11 Valid Loss: 135.1157, Time: 0.36
  Epoch: 39 , Batch: 12 Valid Loss: 136.4958, Time: 0.36
  Epoch: 39 , Batch: 13 Valid Loss: 209.0710, Time: 0.36
  Epoch: 39 , Batch: 14 Valid Loss: 170.3045, Time: 0.36
  Epoch: 39 , Batch: 15 Valid Loss: 157.5657, Time: 0.35
  Epoch: 39 , Batch: 16 Valid Loss: 243.3302, Time: 0.36
  Epoch: 39 , Batch: 17 Valid Loss: 163.2397, Time: 0.37
  Epoch: 39 , Batch: 18 Valid Loss: 146.7547, Time: 0.35
  Epoch: 39 , Batch: 19 Valid Loss: 153.3413, Time: 0.36
  Epoch: 39 , Batch: 20 Valid Loss: 179.2610, Time: 0.36
  Epoch: 39 , Batch: 21 Valid Loss: 123.0379, Time: 0.35
  Epoch: 39 , Batch: 22 Valid Loss: 123.5644, Time: 0.36
  Epoch: 39 , Batch: 23 Valid Loss: 165.4950, Time: 0.35
  Epoch: 39 , Batch: 24 Valid Loss: 142.4183, Time: 0.35
  Epoch: 39 , Batch: 25 Valid Loss: 203.7861, Time: 0.35
  Epoch: 39 , Batch: 26 Valid Loss: 137.3413, Time: 0.35
  Epoch: 39 , Batch: 27 Valid Loss: 241.8771, Time: 0.36
  Epoch: 39 , Batch: 28 Valid Loss: 148.8467, Time: 0.36
  Epoch: 39 , Batch: 29 Valid Loss: 85.9060, Time: 0.34
  Epoch: 39 , Batch: 30 Valid Loss: 145.1030, Time: 0.31
  Epoch: 39 , Batch: 31 Valid Loss: 171.1640, Time: 0.36
  Epoch: 39 , Batch: 32 Valid Loss: 172.1705, Time: 0.26
  
  Epoch: 39 , Training Loss: 156.5736, Validation Loss: 158.5428, Time: 417.88
  
  Starting epoch 40
  Epoch: 40 , Batch: 20 Training Loss: 104.7642, Time: 0.68
  Epoch: 40 , Batch: 40 Training Loss: 199.1840, Time: 0.64
  Epoch: 40 , Batch: 60 Training Loss: 203.7560, Time: 0.64
  Epoch: 40 , Batch: 80 Training Loss: 164.9527, Time: 0.64
  Epoch: 40 , Batch: 100 Training Loss: 186.1097, Time: 0.64
  Epoch: 40 , Batch: 120 Training Loss: 155.4772, Time: 0.64
  Epoch: 40 , Batch: 140 Training Loss: 192.1701, Time: 0.65
  Epoch: 40 , Batch: 160 Training Loss: 193.4585, Time: 0.65
  Epoch: 40 , Batch: 180 Training Loss: 171.5463, Time: 0.64
  Epoch: 40 , Batch: 200 Training Loss: 117.4412, Time: 0.64
  Epoch: 40 , Batch: 220 Training Loss: 152.0421, Time: 0.64
  Epoch: 40 , Batch: 240 Training Loss: 143.3178, Time: 0.64
  Epoch: 40 , Batch: 1 Valid Loss: 156.7238, Time: 0.35
  Epoch: 40 , Batch: 2 Valid Loss: 147.8145, Time: 0.36
  Epoch: 40 , Batch: 3 Valid Loss: 159.7137, Time: 0.35
  Epoch: 40 , Batch: 4 Valid Loss: 191.3246, Time: 0.32
  Epoch: 40 , Batch: 5 Valid Loss: 178.6363, Time: 0.36
  Epoch: 40 , Batch: 6 Valid Loss: 109.2589, Time: 0.35
  Epoch: 40 , Batch: 7 Valid Loss: 144.4912, Time: 0.36
  Epoch: 40 , Batch: 8 Valid Loss: 150.4129, Time: 0.36
  Epoch: 40 , Batch: 9 Valid Loss: 171.8096, Time: 0.35
  Epoch: 40 , Batch: 10 Valid Loss: 210.0710, Time: 0.36
  Epoch: 40 , Batch: 11 Valid Loss: 142.5819, Time: 0.35
  Epoch: 40 , Batch: 12 Valid Loss: 140.7213, Time: 0.36
  Epoch: 40 , Batch: 13 Valid Loss: 221.1351, Time: 0.36
  Epoch: 40 , Batch: 14 Valid Loss: 182.1034, Time: 0.36
  Epoch: 40 , Batch: 15 Valid Loss: 172.3420, Time: 0.35
  Epoch: 40 , Batch: 16 Valid Loss: 264.6909, Time: 0.35
  Epoch: 40 , Batch: 17 Valid Loss: 177.7539, Time: 0.36
  Epoch: 40 , Batch: 18 Valid Loss: 159.2539, Time: 0.35
  Epoch: 40 , Batch: 19 Valid Loss: 167.3184, Time: 0.36
  Epoch: 40 , Batch: 20 Valid Loss: 188.2323, Time: 0.35
  Epoch: 40 , Batch: 21 Valid Loss: 134.6375, Time: 0.36
  Epoch: 40 , Batch: 22 Valid Loss: 137.9939, Time: 0.36
  Epoch: 40 , Batch: 23 Valid Loss: 188.5730, Time: 0.36
  Epoch: 40 , Batch: 24 Valid Loss: 157.5786, Time: 0.36
  Epoch: 40 , Batch: 25 Valid Loss: 215.2611, Time: 0.35
  Epoch: 40 , Batch: 26 Valid Loss: 146.4763, Time: 0.36
  Epoch: 40 , Batch: 27 Valid Loss: 257.7494, Time: 0.36
  Epoch: 40 , Batch: 28 Valid Loss: 156.5165, Time: 0.35
  Epoch: 40 , Batch: 29 Valid Loss: 94.2914, Time: 0.34
  Epoch: 40 , Batch: 30 Valid Loss: 146.4372, Time: 0.35
  Epoch: 40 , Batch: 31 Valid Loss: 180.1848, Time: 0.36
  Epoch: 40 , Batch: 32 Valid Loss: 178.5307, Time: 0.24
  
  Epoch: 40 , Training Loss: 176.2027, Validation Loss: 169.7069, Time: 414.79
  
  Starting epoch 41
  Epoch: 41 , Batch: 20 Training Loss: 128.6025, Time: 0.64
  Epoch: 41 , Batch: 40 Training Loss: 199.1840, Time: 0.64
  Epoch: 41 , Batch: 60 Training Loss: 203.7560, Time: 0.64
  Epoch: 41 , Batch: 80 Training Loss: 164.9526, Time: 0.64
  Epoch: 41 , Batch: 100 Training Loss: 186.1097, Time: 0.64
  Epoch: 41 , Batch: 120 Training Loss: 155.4771, Time: 0.65
  Epoch: 41 , Batch: 140 Training Loss: 192.1701, Time: 0.65
  Epoch: 41 , Batch: 160 Training Loss: 193.4585, Time: 0.64
  Epoch: 41 , Batch: 180 Training Loss: 171.5462, Time: 0.64
  Epoch: 41 , Batch: 200 Training Loss: 117.4412, Time: 0.64
  Epoch: 41 , Batch: 220 Training Loss: 152.0420, Time: 0.65
  Epoch: 41 , Batch: 240 Training Loss: 143.3178, Time: 0.65
  Epoch: 41 , Batch: 1 Valid Loss: 156.7238, Time: 0.35
  Epoch: 41 , Batch: 2 Valid Loss: 147.8145, Time: 0.35
  Epoch: 41 , Batch: 3 Valid Loss: 159.7137, Time: 0.35
  Epoch: 41 , Batch: 4 Valid Loss: 191.3246, Time: 0.36
  Epoch: 41 , Batch: 5 Valid Loss: 178.6363, Time: 0.36
  Epoch: 41 , Batch: 6 Valid Loss: 109.2589, Time: 0.35
  Epoch: 41 , Batch: 7 Valid Loss: 144.4912, Time: 0.34
  Epoch: 41 , Batch: 8 Valid Loss: 150.4129, Time: 0.36
  Epoch: 41 , Batch: 9 Valid Loss: 171.8096, Time: 0.36
  Epoch: 41 , Batch: 10 Valid Loss: 210.0710, Time: 0.36
  Epoch: 41 , Batch: 11 Valid Loss: 142.5818, Time: 0.36
  Epoch: 41 , Batch: 12 Valid Loss: 140.7213, Time: 0.36
  Epoch: 41 , Batch: 13 Valid Loss: 221.1351, Time: 0.35
  Epoch: 41 , Batch: 14 Valid Loss: 182.1034, Time: 0.36
  Epoch: 41 , Batch: 15 Valid Loss: 172.3419, Time: 0.35
  Epoch: 41 , Batch: 16 Valid Loss: 264.6909, Time: 0.36
  Epoch: 41 , Batch: 17 Valid Loss: 177.7538, Time: 0.35
  Epoch: 41 , Batch: 18 Valid Loss: 159.2538, Time: 0.36
  Epoch: 41 , Batch: 19 Valid Loss: 167.3183, Time: 0.36
  Epoch: 41 , Batch: 20 Valid Loss: 188.2322, Time: 0.35
  Epoch: 41 , Batch: 21 Valid Loss: 134.6374, Time: 0.36
  Epoch: 41 , Batch: 22 Valid Loss: 137.9939, Time: 0.36
  Epoch: 41 , Batch: 23 Valid Loss: 188.5730, Time: 0.36
  Epoch: 41 , Batch: 24 Valid Loss: 157.5785, Time: 0.36
  Epoch: 41 , Batch: 25 Valid Loss: 215.2611, Time: 0.36
  Epoch: 41 , Batch: 26 Valid Loss: 146.4763, Time: 0.36
  Epoch: 41 , Batch: 27 Valid Loss: 257.7494, Time: 0.36
  Epoch: 41 , Batch: 28 Valid Loss: 156.5165, Time: 0.35
  Epoch: 41 , Batch: 29 Valid Loss: 94.2914, Time: 0.35
  Epoch: 41 , Batch: 30 Valid Loss: 146.4371, Time: 0.35
  Epoch: 41 , Batch: 31 Valid Loss: 180.1848, Time: 0.36
  Epoch: 41 , Batch: 32 Valid Loss: 178.5307, Time: 0.26
  Epoch: 41 , Training Loss: 178.2872, Validation Loss: 169.7068, Time: 414.65
  </code></pre>