How to prevent the optimization algorithm from just shrinking the image? I'm trying to optimize a set of patterns in an image, but unfortunately they only way I have of evaluating the images results in smaller versions of the pattern scoring much better than larger versions.

As a result, the optimization methods I've tried all just end up shrinking the pattern down until they run into aliasing issues, and start becoming erratic.

I've tried normalizing the images by resizing the patterns to all be the same size at each step, but the optimization algorithm quickly just adds 'decoy' pixels to the corners, defeating the normalization.

I've considered trying to do the optimization with a vectorized form, but evaluating a pattern requires it to be rasterized to a particular size anyway.
Vectorizing the pattern also doesn't solve the problem with normalization and adding decoy geometry.

How can I prevent the optimization process from just scaling down the image?
I doubt your question can be answered in any detail without knowing more about what you're trying to achieve with these patterns.  The general answer to "my scoring function rewards something I don't want" is either to fix the scoring function or to add a penalty to counteract it, but I'm sure you were already aware of that.

Depending the optimization method you're using, you could possibly also try biasing your sample generation method away from local optima in the unwanted part of the parameter space, but that's generally only effective if there's some other, better local optimum available elsewhere for the process to converge to instead.  If the unwanted local optimum is actually the global optimum for your scoring function, then any decent optimizer will sooner or later find its way there.  If that's the case, you really do just need to find a way to fix your scoring.

Anyway, since you say in the comments that smaller patterns are objectively better in some sense, I assume they must also have some property that makes them objectively worse in some other way. You might want to think about what exactly that property is, and try to penalize that.