Computer vision: Identifying different items in screenshot Lets say I have a screenshot like this:



I want to be able to detect/localize each item on the floor, however, there 1) can be any number of items in the image and 2) each item is different

I have a candidate list of all possible items. In reality each one is labelled and separated into individual image files:



I've thought about training something like a convnet, but that feels like it might be slow because I'd need to segment each screenshot with multiple sliding windows and feed each one forward through the net. Creating those sliding window segments for each screenshot will likely take a long time. I'd like the entire detection process to be completed quickly (&lt;1sec)

Whats the most efficient way of doing this detection task? I will be implementing this using Javascript

The main problems are:


The item count is unknown. There could be 0 items in the screenshot, or there could be many
There are a lot of possible targets, each one of which are different in shape, colour, and structure
Javascript isn't the fastest of languages for this type of work. This is a semi strict requirement unless its absolutely impossible to do this using JS. The fallback language would be Python
This is a good case for computer vision.  You know exactly what the icon looks like (basically pixel-perfect), and you want to find whether it exists anywhere in the screenshot.  This is an image matching problem.

Normalized cross-correlation (NCC) is a standard way to check whether small image $I$ (your icon)  appears anywhere in the large image $I'$: you compute the NCC and then look for a location where the value of the NCC is large; that is your candidate match.  You can now apply this to the set of candidate icons: for each candidate icon, search whether it appears anywhere in the screenshot, using NCC.  OpenCV and other computer vision libraries should include normalized implementations of the NCC computation.

For speed, you can also try using image pyramids, but try using NCC directly on the full screenshot first.