Handle non-existing states in q-learning I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating state to Q-values. 

I created a State vector DS = [s1, s2, ..., sN] containing all "desired" states. So the Q-table has the form of Q-table =[DS, Q-values]. 

On the other hand, my agent follows a trajectory. Playing action a at state s (which is a point of the trajectory) leads the agent to another state s' (another point of the trajectory). However, I don't have the s' state in the initially desired states vector DS. 

One solution is to add new states to the DS vector while the Q-learning algorithm is running, but I do not want to add new states. 

Any other ideas on how to handle this problem?
You have two options, either interpolate or restrict the actions only to values that produce states which are in your state vector. 

The simplest interpolation scheme is a linear interpolation, which works as follows (assuming DS contains a set of grid points in increasing order). For a state $s'$ you can locate its closest neighbours from the array DS and the value in state $s'$ will, then, be a weighted average of the values in those neighbouring states. Formally, $$Q(s') = \frac{s_{i+1} - s'}{s_{i+1}-s_i}Q(s_i) + \frac{s'-s_i}{s_{i+1}-s_i}Q(s_{i+1}),$$ for $s_i &lt; s'&lt; s_{i+1}$, where $s_i$ and $s_{i+1}$ are the neighbours, $s_i$ is the $i$-th element of DS(i.e. si = DS[i] = Q-table[i,1]), and $Q(s_i)$ is related to the Q-table as Qsi = Q-table[i,2] (asuming array indexing starts from 1). 

Restricting the actions would work as follows. For simplicity, assume that the agent chooses the next state directly, i.e. $s'=a$. Then, if you have an array of actions A = [a1, a2, ... , aM], $M \le N$ then each $a_i$ needs to be present in the state array DS (i.e. A is a subset of DS, formally $A \subseteq S$). This may not be desirable but it is an option.