How to prevent the optimization algorithm from just shrinking the image?
I doubt your question can be answered in any detail without knowing more about what you're trying to achieve with these patterns.  The general answer to "my scoring function rewards something I don't want" is either to fix the scoring function or to add a penalty to counteract it, but I'm sure you were already aware of that.

Depending the optimization method you're using, you could possibly also try biasing your sample generation method away from local optima in the unwanted part of the parameter space, but that's generally only effective if there's some other, better local optimum available elsewhere for the process to converge to instead.  If the unwanted local optimum is actually the global optimum for your scoring function, then any decent optimizer will sooner or later find its way there.  If that's the case, you really do just need to find a way to fix your scoring.

Anyway, since you say in the comments that smaller patterns are objectively better in some sense, I assume they must also have some property that makes them objectively worse in some other way. You might want to think about what exactly that property is, and try to penalize that.