How do you determine the architecture of a model(keras)? I have solved quite a few kaggle playground problems lately, but I can't understand how to come up with good enough model architecture which gets 0.9+ validation accuracy and without overfitting.

Is there some formula, or is it some hit and trial method for determining filters and neurons.
And also I'm always using relu activation in hidden layers(read it somewhere) when will other activations be used in hidden ones.

Can you lay down some guidelines that I should follow?. Considering I mostly work with image classification problems.

And I you have some other tips as well you can share in your answer.
There is no theoretical understanding, that would take a problem and specify the optimal network architecture for you. So no, there is no formula, I'm afraid. What does exist, are strategies to arrive at good solutions.

Good strategies are to start simple and try to make iterative improvements. Define a single KPI (error metric) that you're trying to optimize, build the simplest model you can imagine that will hopefully work a little bit, then try to improve from there. You get improvements by trying to understand why your model does not perform optimally. You need to be able to distinguish between underfitting and overfitting to make the right modifications. Study the errors that your model makes on specific images and try to make improvements to address those mistakes with changes to the model.

I think you may find this course on Coursera interesting: Structuring Machine Learning Projects.