Is an Arma model equivalent to a 1-layer Recurrent Neural Network without activation function?
It's correct. The reason it sounds so weird is that a 1-layer-NN without activation function is simply a linear map, so it's equivalent to any linear model, the only difference being the inputs having some interpretation.

This even holds true for any NN, no matter the number of layers, without activation functions. The reason: A k-layer-NN is just k matrix multiplications (=linear maps) with activation functions in between. If you remove the latter you are left with a long chain of matrices that you can simply multiply to get one matrix that now defines your linear map.

Many deep learning researches concluded that giving the networks the possibility to create non-linear maps is what makes deep learning so powerful, even when using seemingly unsophisticated activation functions such as the ReLU.