Is my approach to building an RNN to predict the probability that the word is in English appropriate? Goal
To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding).
Example
input:  hello 
output: 100%

input:  nmnmn 
output: 0%

Approach
Here is my approach.
RNN
I have built an RNN with the following specifications: (the subscript $i$ means a specific time step)
The vectors (neurons):
$$
x_i \in \mathbb{R}^n \\
s_i \in \mathbb{R}^m \\
h_i \in \mathbb{R}^m \\
b_i \in \mathbb{R}^n \\
y_i \in \mathbb{R}^n \\
$$
The matrices (weights):
$$
U \in \mathbb{R}^{m \times n} \\
W \in \mathbb{R}^{m \times m} \\
V \in \mathbb{R}^{n \times m} \\
$$
This is how each time step is being fed forward:
$$
y_i = softmax(b_i) \\
b_i = V h_i \\
h_i = f(s_i) \\
s_i = U x_i + W h_{i-1} \\
$$
Note that the $ + W h_{i-1}$ will not be used on the first layer.
Losses
Then, for the loss of each layer, I used cross entropy ($t_i$ is the target, or expected output at time $i$):
$$
L_i = -\sum_{j=1}^{n} t_{i,j} \ln(y_{i,j})
$$
Then, the total loss of the network:
$$
L = \sum L_i
$$
RNN diagram
Here is a picture of the network that I drew:

Data pre-processing
Here is how data is fed into the network:
Each word is split into characters, and every character is split into a one-hot vector. Two special tokens START and END are being appended to the word from the beginning and the end. Then the input at each time step will be every sequential character without END, and the output at each time step will be the following character to the input.
Example
Here is an example:

Start with a word: &quot;cat&quot;
Split it into characters and append the special tags: START  c  a  t  END
Transform into one-hot vectors: $v_1, v_2, v_3, v_4, v_5$
Then the input is $v_1, v_2, v_3, v_4$ and the output $v_2, v_3, v_4, v_5$

Dataset
For the dataset, I used a list of English words.
Since I am working with English characters, the size of the input and output is $n=26+2=28$ (the $+2$ is for the extra START and END tags).
Hyper-parameters
Here are some more specifications:

Hidden size: $m=100$
Learning rate:  $0.001$
Number of training cycles:  $15000$ (each cycle is a loss calculation and backpropagation of a random word)
Activation function: $f(x) = \tanh(x)$

Problem/question
However, when I run my model, I get that the probability of some word being valid is about 0.9 regardless of the input.
For the probability of a word begin valid, I used the value at the last layer of the RNN at the position of END tag after feeding forward the word.
I wrote a gradient checking algorithm and the gradients seem to check up.
Is there conceptually something wrong with my neural network?
I played a bit with $m$, the learning rate, and the number of cycles, but nothing really improved the performance.
while using a neural network for this type of problem is not the ideal use-case, it is a good exercise.
In terms of conceptual issues, the most concerning that I see is the loss: $\sum_{i=1}^N L_i$.
First issue, is that it validates loss at each time step equivalently. This is probably not ideal because in the example (cat), we dont expect it to know its English from just c or ca, but at cat. The quickest fix and probably the best, is to just use $L = L_N$. Though an argument would be that the model should become more and more aware as it gets more letters, and this loss function doesnt achieve that, so another solution would be to add another fixed parameter that you can play with: $L = \sum_i r^{-i}L_i$ where $0 \lt r \lt 1$. Note that $r^{-i}$ can be replaced with any function that increases in size.
Another issue with the loss is it doesnt normalize, meaning on average, larger words will hold more weight to the model than smaller ones, while this may be intended it should be noted and considered (also note if you do end up going with $L=L_N$ this will no longer be a concern.
Hope this helps