How to implement gradient descent for a tanh() activation function for a single layer perceptron?
How can I circumvent this problem?


TLDR : Normalize your input data.

Why?

Notice how tanh actually works on the input data:

&gt;&gt;&gt; np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000])) 
array([-1. , -1., -1. , -0.76159416,  0.,0.76159416,  1. ,1.   , 1])

If your data input to tanh has a magnitude of 10 or more, tanh produces an output of 1. That means it treats 10, 50, 100, 1000 the same. We don't want that.

This would explain


  the input vector with the weights turns out to be too large, which makes tanh(x)=1tanh(x)=1 and 1−o2=01−o2=0, so I can't learn.


Instead, we normalize the data (by dividing the input data by 1000 ( there are other ways of doing this)):

&gt;&gt;&gt; np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]) / 1000)
array([-0.76159416, -0.09966799, -0.00999967, -0.001, 0. ,0.001,  0.00999967,  0.09966799,  0.76159416])

Now, tanh treats all of them differently.

Note
You are doing a classification task. Using mean squared error as a Cost Function won't yield the best results here. You should instead use Cross Entropy Loss with One Hot Vectors and softmax. 

As to why should be a whole different answer.
Here is a link for the same.