Finding the appropriate CNN Model Architecture and Parameters
In many cases in deep learning it works well  to start off with a model which has a very high capacity and potentially overfits. From thereon you can reduce the model capacity to narrow the gap between train and validation error. In this chapter of the Deep Learning Book by Goodwell you find a good description of manual hyperparameter selection and how they influence model capacity.

Moreover, for many tasks well engineered solutions already exist. So check what worked for similar tasks and try out these architectures. For example, MNIST handwriting recognition is somewhat similar to your task. Wikipedia gives several architectures which work well on MNIST. 

The article "Handwritten Digit Recognition using Convolutional Neural Networks in Python with Keras" contains architectures for MNIST, too. Again, this might be close enough to your tasks so I suggest to give these a try. The article includes an architecture very similar to yours but also a more complex one. 

In other cases you could also check out pre-trained models. But here it might not even be required here.