How do you determine the architecture of a model(keras)?
There is no theoretical understanding, that would take a problem and specify the optimal network architecture for you. So no, there is no formula, I'm afraid. What does exist, are strategies to arrive at good solutions.

Good strategies are to start simple and try to make iterative improvements. Define a single KPI (error metric) that you're trying to optimize, build the simplest model you can imagine that will hopefully work a little bit, then try to improve from there. You get improvements by trying to understand why your model does not perform optimally. You need to be able to distinguish between underfitting and overfitting to make the right modifications. Study the errors that your model makes on specific images and try to make improvements to address those mistakes with changes to the model.

I think you may find this course on Coursera interesting: Structuring Machine Learning Projects.