binary image digit convolution neural network has a very high confidence when nothing is in the image
I've been training my convolutional neural network on closely cropped printed digits, similar in style to the Mnist dataset. It works perfectly, with close to 100 percent on both the training and test data.  

At first I trained it on  4 channel binary images where white was '255' and black was '0'. It has 10 outputs, one for each digit, which i then softmax in order to get a probability for each category.

However, I want to use a sliding window technique in order to analyse a page of digits. This is impossible because, for a fully white input, it returns almost %100 confidence that it is a 4, while pretty much nothing for everything else.

I thought it might be the fact the neural net was training on the white space in the image rather than the black space, as the black pixels had value 0, so I inverted the images and trained the network again.  again, it simply returns really  close to 100% confidence on a fully white image.

For both, it returns low percentages for each class when the image is completely black,as it should, although 4 is still the highest

I don't understand the intuition behind this, so any help would be great, even if you could just say its not usual behaviour. Is this to be expected? should i create another class for things that arent digits aswell and train it on that?

heres my neural network: 
its fully convolutional so that i can implement fast sliding window with it, but the last convolutions are equivalent to fully connected layers

   

class fully_convolutional_1channel(nn.Module):
    def __init__(self): 

        super(fully_convolutional_1channel, self).__init__()

        self.conv1 = nn.Conv2d(1, 6, 5)

        self.pool = nn.MaxPool2d(2, 2)

        self.conv2 = nn.Conv2d(6, 16, 5)

        self.fconv1 = nn.Conv2d(16, 120, (4,2))


        self.fconv2 = nn.Conv2d(120, 84,  1)
        self.fconv3 = nn.Conv2d(84, 10, 1)


    def forward(self, x):

        x = self.conv1(x)

        x = F.relu(x)
        #relu does not change size



        x = self.pool(x)

        x = self.pool(F.relu(self.conv2(x)))

        #x = x.view(-1, 16 * 4 * 2)


        x = F.relu(self.fconv1(x))

        x = F.relu(self.fconv2(x))


        x = self.fconv3(x)

        #print(list(x.size))

        return x



  should i create another class for things that arent digits aswell and train it on that?


Yes. Your network has been trained to recognize digits, not white spaces. You could either re-train the network with this strategy or preprocess the page of digits to avoid feeding the network with white spaces.
