What happens when the agent faces a state that never before encountered? I have a network with nodes and links, each of them with a certain amount of resources (that can take discrete values) at the initial state. At random time steps, a service is generated, and, based on the agent's action, the network status changes, reducing some of those nodes and links resources.
The number of all possible states that the network can have is too large to calculate, especially since there is the random factor when generating the services.
Let's say that I set the state space large enough (for example, 5000), and I use Q-Learning for 1000 episodes. Afterwards, when I test the agent ($\max Q(s,a)$), what could happen if the agent faces a state that did not encounter during the training phase?
Having too many states to actually visit is a common problem in RL. This is exactly why we often use function approximation. If you replace your q table with a good function approximator such as a neural network, it should be able to generelize well to states it has not yet encountered.
If you do not use a function approximator but stick with a table, the agent will have no idea what to do when it encounters a new state. For more information,  see Reinforcement Learning  by Sutton and Barto, chapter 9.