What are the exact differences between Deep Learning, Deep Neural Networks, Artificial Neural Networks and further terms?
I will try to explain it in the simplest way I can-

Deep Learning - Deep learning is a machine learning technique that teaches computers to do what comes naturally to humans: learn by example. It is a subclass of Machine Learning where (in layman terms) we use only Neural Networks. https://www.mathworks.com/discovery/deep-learning.html

Deep Neural Network (DNN) - This is just a term for a neural network with many layers and many nodes in layers, which is not usually found in a shallow network. There is no clear boundary between a shallow and deep neural network. I suggest you watch Andrew Ng's course on deep learning to get a better understanding.

Artificial Neural Network (ANN) - This is just another term for a neural network, all neural networks are ANN's.

Feedforward Neural Network - It is a type of neural network where there is no feedback connections. In technical terms, the information flows only in one direction (input to output) in the forward propagation stage.
https://towardsdatascience.com/deep-learning-feedforward-neural-network-26a6705dbdc7


  Shouldn't there also be for instance Deep Recurrent Neural Networks?


There is no need for Deep Recurrent Neural Networks, it would simply be Recurrent Neural Networks.


  Is it correct that Deep Neural Networks must feedforward Neural Networks?


No, RNN is a type of Deep Neural Network which is not a feedforward NN.

So, to answer your questions,


  
  Does the term Deep Neural Networks ONLY belong to feedforward Neural Networks? 
  


No, the term Deep Neural Networks belong to all Neural Networks with multiple layers. (Architecture does not matter). 


  
  Is the term Deep Neural Networks a collective term for ALL Artificial Neural Networks with multiple layers, or just for all feedforward Neural Networks with multiple layers? 
  


Yes, it is a collective term for ALL Artificial Neural Networks with multiple layers.


  
  Would it be more accurate to use the collective term "Deep Learning Architectures" with strictly separated subclasses as Wikipedia suggests?
  


Deep learning architectures just refer to the different neural network architectures used for different tasks - CNN for image processing, RNN for sequence (text, audio etc) processing etc. So, when you are doing a particular task, you can use the associated architecture name. For general purpose, you can just use the term DNN.

Hope this clears your confusion.