What is a 'notch' in image processing?
In standard 1D signal processing, notch filters are filters that try to remove one specific frequency, and pass all others.

I say "try to" because this is not usually achievable, but it comes close.

Steve W. Smiths's DSP Guide has a nice plot (Figure 19.6b below) that shows the frequency response of such filters.

In image processing, it will be the same thing, but instead of a 1D filter it'll be a 2D filter probably comprised of two 1D notch filters --- one in each image direction.