title: PyTorch: Testing with torchvision.datasets.ImageFolder and DataLoader
body: <p>I'm a newbie trying to make this PyTorch CNN work with the <a href="https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/" rel="noreferrer">Cats&amp;Dogs dataset from kaggle</a>. As there are no targets for the test images, I manually classified some of the test images and put the class in the filename, to be able to test (maybe should have just used some of the train images).</p>
  
  <p>I used the torchvision.datasets.ImageFolder class to load the train and test images. The training seems to work.</p>
  
  <p>But what do I need to do to make the test-routine work? I don't know, how to connect my test_data_loader with the test loop at the bottom, via test_x and test_y. </p>
  
  <p>The Code is based on <a href="https://github.com/MorvanZhou/PyTorch-Tutorial/blob/master/tutorial-contents/401_CNN.py" rel="noreferrer">this MNIST example CNN.</a> There, something like this is used right after the loaders are created. But I failed to rewrite it for my dataset:</p>
  
  <pre><code>test_x = Variable(torch.unsqueeze(test_data.test_data, dim=1), volatile=True).type(torch.FloatTensor)[:2000]/255.   # shape from (2000, 28, 28) to (2000, 1, 28, 28), value in range(0,1)
  test_y = test_data.test_labels[:2000]
  </code></pre>
  
  <p>The Code:</p>
  
  <pre><code>import os
  import numpy as np
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  from torch.autograd import Variable
  import torch.utils.data as data
  import torchvision
  from torchvision import transforms
  
  EPOCHS = 2
  BATCH_SIZE = 10
  LEARNING_RATE = 0.003
  TRAIN_DATA_PATH = "./train_cl/"
  TEST_DATA_PATH = "./test_named_cl/"
  TRANSFORM_IMG = transforms.Compose([
      transforms.Resize(256),
      transforms.CenterCrop(256),
      transforms.ToTensor(),
      transforms.Normalize(mean=[0.485, 0.456, 0.406],
                           std=[0.229, 0.224, 0.225] )
      ])
  
  train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
  train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)
  test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
  test_data_loader  = data.DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4) 
  
  class CNN(nn.Module):
      # omitted...
  
  if __name__ == '__main__':
  
      print("Number of train samples: ", len(train_data))
      print("Number of test samples: ", len(test_data))
      print("Detected Classes are: ", train_data.class_to_idx) # classes are detected by folder structure
  
      model = CNN()    
      optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
      loss_func = nn.CrossEntropyLoss()    
  
      # Training and Testing
      for epoch in range(EPOCHS):        
          for step, (x, y) in enumerate(train_data_loader):
              b_x = Variable(x)   # batch x (image)
              b_y = Variable(y)   # batch y (target)
              output = model(b_x)[0]          
              loss = loss_func(output, b_y)   
              optimizer.zero_grad()           
              loss.backward()                 
              optimizer.step()
  
              # Test -&gt; this is where I have no clue
              if step % 50 == 0:
                  test_x = Variable(test_data_loader)
                  test_output, last_layer = model(test_x)
                  pred_y = torch.max(test_output, 1)[1].data.squeeze()
                  accuracy = sum(pred_y == test_y) / float(test_y.size(0))
                  print('Epoch: ', epoch, '| train loss: %.4f' % loss.data[0], '| test accuracy: %.2f' % accuracy)
  </code></pre>