Will reinforcement learning work if states wont get repeated again?
I am not convinced that framing a document retrieval success as reinforcement learning problem will be easy to get working. 

However, your core question is answerable. This problem occurs in other problems that are too large to explore all possible states. For instance, in the game of Go, the AlphaZero system cannot possibly explore all game positions. Similarly, in Atari games, many of which have also had reinforcement learning applied successfully, it is very unlikely that the agent has seen all possible states of the system - typically it will have trained on a million video frames.

The answer to this issue is to use some form of function approximation that can generalise to new unseen data. That is precisely what supervised machine learning models do, so the RL agent will use one internally, typically a linear regression or neural network, so that values and or policy learned from states it has explored will be associated with similar unseen states (for some interpretation of similar).