Modulo vs Scaling vs Capping when performing filters I'm currently taking a graphics course and we've recently covered cross-correlation with regards to using a filter matrix that is applied to a region of pixels in a sort of continual raster-type scan across a larger input image to produce some output image.

The formula is denoted here:


G is the output image, h is the kernel/box filter matrix, F is the input image

The classic one that was introduced is a 3x3 box filter for blurring an input image. The box filter is a 3x3 matrix with each cell value being equally set to 1/9. We've been dealing primarily with grayscale images with values 0 (black) to 255 (white).

In this case, the largest possible output pixel value would be if the 3x3 input image area contained all white pixels in which case we'd have:

[(1/9) * 255 ] * 9 = 255

This is fine, but what if our kernel filter matrix instead had 9 cell values all equal to 0.5?

Then the largest potential output value would be:

[0.5 * 255] * 9 = 1147.5

Once we've completely finished producing our new output image matrix, how do we handle values above 255 and below 0?

It seems like there are three possible approaches:

Capping

Cap all output values to 0 and 255. Any value below 0 is capped to 0, any value above 255 is capped to 255.

Modulo

Use modulo. In this case it would be 1147.5 % 255 = 125.5 rounded to 126.

Scaling

Take the highest output value and use it to scale all the values proportionally. If our example output image pixel values were 1147.5, 100, and 255, then we would perform the following operations:


(1147.5 / 1147.5) * 255 = 255
(100 / 1147.5)    * 255 = 22.2 = 22
(255 / 1147.5)    * 255 = 56.6 = 57


I just don't get which one of these I'm supposed to be using for common graphics operations. I have the same question when combining images together (such as multiply, divide, screen, etc).
When doing a filter operation such as a blur, in many cases the filter itself will be normalized so its values sum to 1.0, precisely to avoid this problem.

In cases where out-of-range outputs do need to be handled, the most common way is what you called capping. In graphics it's more commonly known as clamping or saturating. This is the default behavior if, for instance, you run a pixel shader on a GPU and output out-of-range values.

Renormalizing the output image might be necessary in some special cases depending on what you're doing. I can't think of anytime that I've seen the modulo behavior used—it would be a pretty weird operation to do on a visual image.