Why use convolutional NNs for a visual inspection task over classic CV template matching? I had an interesting discussion come up based on a project we were working on: why use a CNN visual inspection system over a template matching algorithm?

Background: I had shown a demo of a simple CNN vision system (webcam + laptop) that detected if a particular type of object was "broken"/defective or not - in this case, a PCB circuit board. My CNN model was shown examples of the proper and broken circuit boards (about 100 images of each) on a static background. Our model used the first few conv/maxpool layers of pre-trained VGG16 (on imagenet), and then we added a few more trainable convs/pools, with a few denses, leading to a dim-3 one hot encoded vectored output for classification: (is_empty, has_good_product, has_defective_product).

The model trained pretty easily and reached 99% validation acc no problems; we also trained with various data augmentation since we know our dataset was small. In practice, it worked about 9 times out of 10, but a few random translations/rotations of the same circuit board would occasionally put it in the opposite class. Perhaps more aggressive data augmentation would have helped. Anyways, for a prototype concept project we were happy.

Now we were presenting to another engineer and his colleague, and he brought up the argument that NNs are overkill for this, should just use template matching, why would one want to do CNNs?

We didn't have a great answer for why our approach could be better in certain applications (e.g. other parts to inspect). Some points we brought up:

1) More robust to invariances (through e.g. data augmentation)

2) Can do online learning to improve the system (e.g. human can tell the software which examples it got wrong)

3) No need to set thresholds like in classical computer vision algorithms
What do you guys think, are there more advantages for a CNN system for this type of inspection task? In what cases would it be better than template matching?

A few more random ideas for when deep NNs could be the tech for the job: for systems that require 3D depth sensing as part of the input, or any type of object that can be deformed/stretched/squished but still be "good" and not defective (e.g. a stuffed animal, wires, etc). Curious to hear your thoughts :)
The answer depends on the task.  Template matching can work for some tasks but not for all.  CNNs potentially have the ability to generalize to unseen inputs that don't match any of your templates, so can potentially generalize better.

But whether CNNs will beat template matching will depend on the specific task and what specifically you're trying to achieve.  This is an empirical science; ultimately, the way you find out which works better is to try them both -- or learn from others who have tried them (e.g., by reading the literature).  I don't think you're going to find some theory or taxonomy that is going to substitute for empirical evaluation on real-world data.