Alternative to sliding window neural network (was: Object detect (or) image classification at specific locations in the frame) Recent advances in Deeplearning and dedicated hardware has made it possible to detect images with a much better accuracy than ever. Neural networks are the gold standard for computer vision application and are used widely in the industry, for example for internet search engines and autonomous cars. In real life problems, the image contains of regions with different objects. It is not enough to only identify the picture but elements of the picture.

A while ago an alternative to the well known sliding window algorithm was described in the literature, called Region Proposal Networks. It is basically a convolution neural network which was extended by a region vector.

Problem that I am trying to solve:

In a given video frame, I want to pick some region of interests (literally), and perform classification on those regions.

How is it currently implemented


Capture the video frame
Split the video frame into multiple images each representing a region of interest
Perform image classification(inference) on each of the image (corresponding to a part of the frame)
Aggregate the results of #3 


Problem with the current approach

Multiple inferences per frame.

Question

I am looking for a solution where I specify the locations of interest in a frame, and inference task, be it object detection (or) image classification, is performed only on those regions.Can you please point to me the references which I need to study (or) use to do this.
There are many different problems in computer vision. Four of them are well described by the top image here:


Classification: Given an image, say what is on it (a single thing)
Classification+Localization: Given an image, say what is on it and draw an axis-aligned bounding box (AABB) around it
Object detection: Given an image, draw AABBs around every object and classify those objects
Semantic segmentation: See Survey of semantic segmentation
Instance segmentatation: Like semantic segmentation, but if there are multiple cats then they should be recognized as different objects.


Your question seems to be about object detection. The relevant papers here are:


SSD: Single Shot MultiBox Detector: Code
You Only Look Once: Unified, Real-Time Object Detection (YOLO): Website with video
YOLOv3


If you actually already have the regions, then you can simply perform classification on them. When you pad / scale / crop them, you can batch-predict them.