How do you choose the optimal number of taps in a windowed image scaling filter?
Images quantized to some bit depth have error due to the quantization. Your filter also introduces error, compared to using an infinitely large ideal filter. Using larger and larger filters, eventually you will arrive at a knee in the total error vs filter size diagram where those two errors are equal. After that the quantization error dominates so it doesn't pay off much to improve the filter further.

A commonly used error metric is peak signal-to-noise ratio.