How to improve my image classifier to recognize real world images
I would like to train a hand gesture classifier with pytorch.
  The dataset images looks like this.

I tried to use resnet34 and some kind of data augmentation. I got a high accuracy on test set, but low accuracy when trying to recognize my own gesture in the real world. It works fine when the background is white, goes crazy when other things(my face, chair, bed etc.) appeared in the background. Maybe that's because test images have a pure background, so how can I improve my classifier? 
Also I want to add a 'non-gesture' category in my claasifier as well. How can I do that?
This is my data augmentation transforms:

transform = torchvision.transforms.Compose([
         torchvision.transforms.Grayscale(3),
         torchvision.transforms.RandomHorizontalFlip(),
         torchvision.transforms.RandomRotation(20),
         torchvision.transforms.RandomResizedCrop(64, (0.6, 1.2)),
         torchvision.transforms.ToTensor(),
         torchvision.transforms.Normalize((0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),
    ])


Any advice is appreciated. Thanks!

Aside from changes to your classifier, you should look at your training data:

Ask yourself:


Is my sample size big enough?
If you have too few images to train on, no data augmentation in the world will make up for that. Aim to acquire a large, heterogenous dataset with even label distribution.
Does your training data accurately reflect the circumstances you want to use your classifier in. The images you supplied seem to have a light background, maybe try to get images of hand gestures with different backgrounds.


After that you should take a look at your classifier and improve it. Since you didn't include your model I can't comment on that.
