Tuning the model parameters vs the parameter of optimizer for Deep Neural Networks?
The short answer, is that you want the fastest way to reach the performance you expect/desire. This would mean first playing with some hyperparameters like learning rate, initialisation strategy, trying different optimizers, etc.
Having said that, if you managed to write your training process such that it is trivial to swap out the model completely for another one, you can also give that a go.

If you have a model that is known to work well for a given type of task, but is really under-performing (e.g. not converging or a large difference to results in the literature), then I would first check for some common issues in the training process:

data quality: are the inputs scaled correctly? Should you add some
data volume: if you don't have many samples, can you get more or use augmentations?
loss function: are the outputs of the model being correctly consumed and the losses correctly computed? are you even using the right loss formulation?

This is essentially debugging your training setup.
I would convince myself as far as possible that none of these exist before building a different architecture or a radically different approach. The main reason for this, is that I assume you did some research into the problem and the type of models that are known to perform well - so if this doesn't work, what makes you think another architecture will?  :)