Title: An End-to-End Approach to Natural Language Object Retrieval via Context-Aware Deep Reinforcement Learning,
Abstract: We propose an end-to-end approach to the natural language object retrieval
task, which localizes an object within an image according to a natural language
description, i.e., referring expression. Previous works divide this problem
into two independent stages: first, compute region proposals from the image
without the exploration of the language description; second, score the object
proposals with regard to the referring expression and choose the top-ranked
proposals. The object proposals are generated independently from the referring
expression, which makes the proposal generation redundant and even irrelevant
to the referred object. In this work, we train an agent with deep reinforcement
learning, which learns to move and reshape a bounding box to localize the
object according to the referring expression. We incorporate both the spatial
and temporal context information into the training procedure. By simultaneously
exploiting local visual information, the spatial and temporal context and the
referring language a priori, the agent selects an appropriate action to take at
each time. A special action is defined to indicate when the agent finds the
referred object, and terminate the procedure. We evaluate our model on various
datasets, and our algorithm significantly outperforms the compared algorithms.
Notably, the accuracy improvement of our method over the recent method GroundeR
and SCRC on the ReferItGame dataset are 7.67% and 18.25%, respectively.