Compact and discriminative visual codebooks are preferred in many visual recognition tasks. In the literature, a number of works have taken the approach of hierarchically merging visual words of an initial large-sized codebook, but implemented this approach with different merging criteria. In this work, we propose a single probabilistic framework to unify these merging criteria, by identifying two key factors: the function used to model class-conditional distribution and the method used to estimate the distribution parameters. More importantly, by adopting new distribution functions and or parameter estimation methods, our framework can readily produce a spectrum of novel merging criteria. Three of them are specifically focused in this work. In the first criterion, we adopt the multinomial distribution with Bayesian method; In the second criterion, we integrate Gaussian distribution with maximum likelihood parameter estimation. In the third criterion, which shows the best merging performance, we propose a max-margin-based parameter estimation method and apply it with multinomial distribution. Extensive experimental study is conducted to systematically analyse the performance of the above three criteria and compare them with existing ones. As demonstrated, the best criterion obtained in our framework achieves the overall best merging performance among the comparable merging criteria developed in the literature.
@cite_18: We introduce a novel distributional clustering algorithm that maximizes the mutual information per cluster between data and given categories. This algorithm can be considered as a bottom up hard version of the recently introduced "Information Bottleneck Method". The algorithm is compared with the top-down soft version of the information bottleneck method and a relationship between the hard and soft results is established. We demonstrate the algorithm on the 20 Newsgroups data set. For a subset of two news-groups we achieve compression by 3 orders of magnitudes loosing only 10 of the original mutual information.
@cite_25: We present an approach to determine the category and location of objects in images. It performs very fast categorization of each pixel in an image, a brute-force approach made feasible by three key developments: First, our method reduces the size of a large generic dictionary (on the order of ten thousand words) to the low hundreds while increasing classification performance compared to k-means. This is achieved by creating a discriminative dictionary tailored to the task by following the information bottleneck principle. Second, we perform feature-based categorization efficiently on a dense grid by extending the concept of integral images to the computation of local histograms. Third, we compute SIFT descriptors densely in linear time. We compare our method to the state of the art and find that it excels in accuracy and simplicity, performing better while assuming less.
: In @cite_25 , the mutual information, @math , between @math and class labels @math is used to measure its discriminative power as where @math denotes the @math th word of @math and @math and @math are estimated with the @math th bins of training histograms. At each level @math , the words @math and @math whose mergence maximizes @math are identified and merged. As noted in @cite_25 , this criterion can be related to agglomerative information bottleneck @cite_18 .