Number of parameters in Simple RNNs
Keras SimpleRNN is a fully connected RNN so indeed each unit connected with all other units. So the equation becomes:

(input_feature +1) x units + units x units
2x50 +2500
+1 comes from the bias