How do you choose the optimal number of taps in a windowed image scaling filter? I'm attempting to use a filter to scale an image and for the most part, I have something that appears to work.  But now I want to be able to generate a filter with an optimal number of taps.  I know in general that the more taps, the better.  But there are diminishing returns in how much each additional tap will increase the quality of the image.  So how would one go about picking a number of taps that makes a good tradeoff between implementation complexity and image quality?  Are there standard measurements that would allow different filters to be evaluated to some "ideal" filter?



Thinking about this more, I think this is really two questions.  One is a matter of deciding which windowed sinc to use (in this case, I've already settled on using lanczos2 or 3) -- that is a more general question that has various tradeoffs depending on the application.  But once that decision is made, the number of taps is easily determined.  For lanczos2, the ideal number of taps is 4 and for lanczos3, the ideal number of taps is 6.  The reason it's not 5 or 7 is that one of those taps will always be zero due to the windowing.  

In the more general case, I think the easy way to state how many taps to have for a given window is to simply say:

taps = max - min


In the case of lanczos2, the max is 2 and the min is -2.  Therefore, taps = 4.
Images quantized to some bit depth have error due to the quantization. Your filter also introduces error, compared to using an infinitely large ideal filter. Using larger and larger filters, eventually you will arrive at a knee in the total error vs filter size diagram where those two errors are equal. After that the quantization error dominates so it doesn't pay off much to improve the filter further.

A commonly used error metric is peak signal-to-noise ratio.