Too little or too much maxpooling?
Using max-pooling is not a good idea on its own. The reason is that by employing that, you ignore $75$% of the information each time. If your input is a signal which is quite a small pattern, as the number you have referred, it is better not to use max-pool that much. Although the use of max-pooling adds relative spatial invariance to the objects in the input and is useful for classification tasks, the main reason it is employed nowadays is to reduce the number of parameters to train. For instance, for images which belong to $R^{224\times224}$, it is wise to use them for some layers due to lessening the number of training parameters significantly. For images with smaller sizes, the input signal on its own has a smaller number of entries, features, and by employing max-pooling, you are chiefly discarding information which may be necessary and prominent for generalisation. You are actually shrugging them off!