Attention network without hidden state?
Though translations are not on a word-word basis, nonetheless there is a strong merit in retaining the sequence of the words at the encoder side. There is a huge penalty to pay for this as this causes serialization, but in spite of it LSTMs and GRU's became so popular so one can imagine that sequence order matters. After the encoder is done processing in sequence, the final state that is generated is sort of a sentence embedding and contains the essence of the sentence. This is a good starting point for the decoder to pick and use. Unlike what you have assumed, the model does not look 'only' at the context generated by the attention layer to make predictions. It also uses the prev LSTM state along with the context (and the last translated word to make the next prediction). If you trace back the prev LSTM state right to the beginning of the decoder, you can see it has its origins in the final LSTM state of the encoder.
Having said that, your question still is very pertinent. The concept of Attention is so powerful that with self attention and multi-head attention, it is now possible to do away with the RNN at the encoder end altogether and just use the representation generated purely by the process of 'Attention'. But even here the authors of the landmark paper - Attention is all you need - add in a small hack to retain the order of the sequence of the words of the input sentence. This seems to improve the model in better prediction.