Is an Arma model equivalent to a 1-layer Recurrent Neural Network without activation function? Given a time series $f(t)$ to forecast, let us consider an Arma model of the form:
$$
f(t) = c + \sum_{i=1}^p a_i f(t-i) + e(t) + \sum_{j=1}^q b_j e(t-j)
$$

where $e(t)$ are the forecast errors.

On the train set, if $f(t)$ is the ground truth, then we define its estimate obtained with this model as $\widetilde{f}(t) = f(t) + e(t)$.

Let $m = min(p,q)$, we can rewrite the first equation as:
$$
\widetilde{f}(t) = c + \sum_{i=1}^m (a_i + b_i) f(t-i) + \sum_{i=m+1}^p a_i f(t-i) - \sum_{j=1}^q b_j \widetilde{f}(t-j)
$$
Then after reparametrization can be rewritten as:
$$
\widetilde{f}(t) = c + \sum_{i=1}^k c_i f(t-i) - \sum_{j=1}^q b_j \widetilde{f}(t-j)
$$
Which is the equation of a 1-layer recurrent neural network (RNN) without activation function.

So, are Arma models a subset of RNNs or is there a flaw in this reasoning ?
It's correct. The reason it sounds so weird is that a 1-layer-NN without activation function is simply a linear map, so it's equivalent to any linear model, the only difference being the inputs having some interpretation.

This even holds true for any NN, no matter the number of layers, without activation functions. The reason: A k-layer-NN is just k matrix multiplications (=linear maps) with activation functions in between. If you remove the latter you are left with a long chain of matrices that you can simply multiply to get one matrix that now defines your linear map.

Many deep learning researches concluded that giving the networks the possibility to create non-linear maps is what makes deep learning so powerful, even when using seemingly unsophisticated activation functions such as the ReLU.