Convolutional neural network giving high confidence on wrong classification
Since your model seems to be doing fine on the correct hand signs, I don't think there is anything wrong. It's a common misconception that you can interpret the model output as a confidence measure. Consider the following example.

Let's say you have a binary classifer and you learn to distinguish numbers between -1 and 0 in one class labelled as 0 and numbers between 0 and 1 in the other labelled as 1. Your training data is in these intervals and is labeled correctly. Your classifier in the end will be something like

$$ y = \sigma(wx)$$

with $\sigma$ as the sigmoid function. When you train, you will quickly learn that $w &gt; 0$. The more you train, the larger $|w|$ will become untill you get something close to 0 for all $x \in [-1,0)$ and something close to 1 for $x \in (0,1]$. All perfect and normal so far. 

Now you pass the number $x=10$ to the model. The query does not make sense, as this $x$ does not belong to any of the classes the model was trained on and has nothing to do with the training data. There is no reason to assume that the model would give any useful output for $x=10$. But still, the model will the super confident that $10 \in (0,1]$ since for $x=10$ the model will output a value closer to 1 than for any of the training examples. This is clearly not correct.

I hope this clarifies that the interpretation of the classifier output as confidence can be very misleading. If the classes are close together in feature space and you query with an example that is somewhat between the training data, then you could take it as such, but for input that is not represented in the training data, the numbers you will get don't have any meaning at all.

In your application you could introduce the counterexamples to the training data and label them as "other", but that would be risky as well. If you really need a confidence measure, look into Bayesian Neural Networks (e.g. Bayes-By-Backprop, Dropout or Batch norm), other Bayesian methods or bootstrapping.