How to use a different model to deep neural network with reinforcement learning based on DQN?
Well, if you remove the DNN, I would not call that a Deep Q-Network anymore.. but it is definitely possible to remove that and still consider the approach as Reinforcement Learning.

Actually, the function of the deep neural network is just to approximate the Q-value function. The DNN is just a function in the form $Q_\theta(s, a)$ and based on samples, you adjust $\theta$ to minimize the error. 

In practice, you could use any function you want. Obviously, some functions will work better than others. There are simpler approaches based on linear regression, least squares, kernel-based approaches, etc.

With respect to unsupervised learning.. mm. I guess you could force it somehow, but keep in mind that it would not totally make sense given that RL already provides you feedback so it makes sense that you use it.

Some interested advantage of neural networks is that they can find common elements in $\phi$ and rapidly abstract away some noise that if is not relevant with respect to predicting the Q-value of certain state/action.

I noticed that you would like to use something like random forests, etc. There is nothing to stop you from doing that. In particular, random forests are not ideal because they don't deal well with non-stationary. Instead they require bit batches of data. However, this does not mean, it is impossible.

In DQN, after you sample from the environment, you obtain feedback from the environment in the form of rewards associated with state/action pairs. RL combines this information with previous predictions using TD (there are some variations like TD($\lambda$), etc.). What TD gives you is a better estimate of $Q(s,a)$ and your predictor (in this case, your random forest) should learn that when the input is $(s,a)$ the correct answer is $Q(s,a)$.

It is likely that you've seen the same inputs before (specially for initial states) but because TD improves the reference value, you would need to drop these previous values and replace them with the most recent ones. Random forest would need to get this whole batch again.

In general, I accept that while both cases are supervised learning, adjusting DQN to work with non-parametric methods such as random forests can be inconvenient.

If I had to guess based on the comments, I think your main question circles around the distinction between parametric and non-parametric supervised learning. In that case, it is fair to say that most non-parametric methods are not easy to combine with DQN.