Softmax classifier never allows for 100% probability in LSTM?
If want you mean by 100% certainty is:

1,

0, 

0,

0,

as by your example, you cannot obtain this type of output using soft-max. For this to happen you will need that the softmax equals 0 for those i that are not correct, and this simply cannot happen because:

$$ e^{h_i} \neq 0 $$ $$ \forall h_i \in \mathbb{R} $$

See this link for better explanation. Note that $ e^x $ can be close to 0 but it will never be 0. 
The kind of output that you want is the one that the max function would output, sadly the max function is not differentiable so a soft version is needed hence softmax.

I think that the problem is that you are using softmax after tanh, and this so you must eliminate one of the "squashing" functions, see this discussion

The reason to use softmax is if the classes are mutually exclusives, also one of the advantages of using softmax is that it reduces sensitivity to outliers by giving at least some probability to the other values. See this links for a more detailed explanation 1 and 2.