In this paper, we propose training very deep neural networks (DNNs) for supervised learning of hash codes. Existing methods in this context train relatively "shallow" networks limited by the issues arising in back propagation (e.e. vanishing gradients) as well as computational efficiency. We propose a novel and efficient training algorithm inspired by alternating direction method of multipliers (ADMM) that overcomes some of these limitations. Our method decomposes the training process into independent layer-wise local updates through auxiliary variables. Empirically we observe that our training algorithm always converges and its computational complexity is linearly proportional to the number of edges in the networks. Empirically we manage to train DNNs with 64 hidden layers and 1024 nodes per layer for supervised hashing in about 3 hours using a single GPU. Our proposed very deep supervised hashing (VDSH) method significantly outperforms the state-of-the-art on several benchmark datasets.
@cite_40: This paper describes a general purpose method for solving convex optimization problems in a distributed computing environment. In particular, if the problem data includes a large linear operator or matrix (A ), the method allows for handling each sub-block of (A ) on a separate machine. The approach works as follows. First, we define a canonical problem form called graph form, in which we have two sets of variables related by a linear operator (A ), such that the objective function is separable across these two sets of variables. Many types of problems are easily expressed in graph form, including cone programs and a wide variety of regularized loss minimization problems from statistics, like logistic regression, the support vector machine, and the lasso. Next, we describe graph projection splitting, a form of Douglas–Rachford splitting or the alternating direction method of multipliers, to solve graph form problems serially. Finally, we derive a distributed block splitting algorithm based on graph projection splitting. In a statistical or machine learning context, this allows for training models exactly with a huge number of both training examples and features, such that each processor handles only a subset of both. To the best of our knowledge, this is the only general purpose method with this property. We present several numerical experiments in both the serial and distributed settings.
@cite_3: Optimization is an important tool used in decision science and for the analysis of physical systems used in This space but at the book requires substantial background. Numerical optimization presents a graduate text, in continuous presents. Mmor mathematical optimization in understanding of initiative. I acknowledge the necessary to use, optimization it talks extensively about algorithmic performance and thinking. The optimization they delve into, the new ideas progressing through more thoroughly updated throughout.
In contrast our ADMM-based method is more suitable and efficient for solving regularized loss minimization as has been shown in the Block-Splitting algorithm @cite_40 . ADMM solves optimization (possibly nonconvex) problems with equality constraints by decomposing an objective into several disjoint sub-objectives using new auxiliary variables so that the original objective can be optimized iteratively using coordinate descent. With small additional computational cost we circumvent the need for relaxation of penalty related parameters as required in this context @cite_3 .