Clamping Q function to it's theoretical maximum, yes or no?
As MountainCar is often solved with $\gamma = 1$ and a negative reward per timestep, you would immediately hit a problem with your ability to calculate a maximum action value in that case. However, I don't think that is your problem here, the discounted return with positive reward at the end should still encourage desired behaviour for the problem.

It is likely that you are experiencing known problems with RL, and the "deadly triad":


Function approximation (neural network)
on a bootstrap method (Q-Learning or any TD-learning approach)
off policy (learning optimal policy from non-optimal behaviour*, which is a feature of Q-learning)


This combination is often unstable and difficult to train. Your Q value clamping is one way to help stabilise values. Some features of DQN approach are also designed to deal with this issue:


Experience replay. Agent does not learn online, but instead puts each sample (S, A, R, S') into a memory table and trains the neural network on mini-batches sampled from this memory. Commonly this minibatch update is run on every step (after enough experience collected) and might be e.g. size 32. So learning updates happen faster than experience is collected.
Frozen bootstrap target network. The network used to calculate $\operatorname{max}_{a'} Q(s', a')$ when setting the target values to learn (in td target $R + \gamma \operatorname{max}_{a'} Q(s', a')$) is kept stable, and refreshed to use a copy of the current weights after a certain number of steps (e.g. 100 or 1000, or once every 10 episodes).




* Technically off-policy learning is learning the cost function for any policy $\pi$ from a different behaviour policy $b$, but the common case is for control systems attempting to find an optimal policy from an exploratory policy.