1. A method for querying a database of videos, comprising: extracting a set of descriptors for each image in a video of a scene; aggregating the sets of descriptors for the video to form a descriptor matrix; determining a product of a low-rank descriptor matrix and a selection matrix using an orthogonal non-negative matrix factorization of the descriptor matrix, such that each column of the selection matrix has only one nonzero entry and associates a column in the descriptor matrix to a corresponding column in the low-rank descriptor matrix; wherein the factorization comprises: determining a low-rank factor by a non-negative least squares minimization; and determining a selection matrix factor by minimizing a proximal point least squares problem and retaining a largest entry in every column of the selection matrix and setting all other entries to zero; and querying the database of videos including at least one video using the low-rank descriptor matrix to produce a search result, wherein the steps are performed in a processor.