Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) is a momentum version of stochastic gradient descent with properly injected Gaussian noise to find a global minimum. In this paper, non-asymptotic convergence analysis of SGHMC is given in the context of non-convex optimization, where subsampling techniques are used over an i.i.d dataset for gradient updates. Our results complement those of [RRT17] and improve on those of [GGZ18].
@cite_30: In this paper, we revisit the recently established theoretical guarantees for the convergence of the Langevin Monte Carlo algorithm of sampling from a smooth and (strongly) log-concave density. We improve the existing results when the convergence is measured in the Wasserstein distance and provide further insights on the very tight relations between, on the one hand, the Langevin Monte Carlo for sampling and, on the other hand, the gradient descent for optimization. Finally, we also establish guarantees for the convergence of a version of the Langevin Monte Carlo algorithm that is based on noisy evaluations of the gradient.
@cite_18: We seek a global minimum of @math . The solution to @math will find local minima. Using the idea of simulated annealing, we consider the diffusion process, @math , @math , where @math is the n-dimensional standard Brownian motion and @math is the annealing rate which decreases to zero as t goes to @math . Under suitable condition on @math , we prove that @math converges weakly to a probability measure @math if for large t, @math with @math , where @math has a simple expression involving the action function of the dynamical system @math , @math concentrates on the global minima of U and is the weak limit of the Gibbs densities @math .The above result can also be formulated as follows: consider the Fokker–Planck equation (forward equation) [ t V(t,y) = ...
@cite_22: 
@cite_7: An algorithm of the form @math , where @math is a smooth function on @math , @math is a sequence of @math -valued random variables, @math is a sequence of independent standard d-dimensional Gaussian random variables, @math and @math for k large, is considered. An algorithm of this type arises by adding slowly decreasing white Gaussian noise to a stochastic gradient algorithm. It is shown, under suitable conditions on @math , @math , A, and B, that @math converges in probability to the set of global minima of @math . No prior information is assumed as to what bounded region contains a global minimum. The analysis is based on the asymptotic behavior of the related diffusion process @math , where @math is a standard d-dimensional Wiener process and @math for t large.
@cite_6: The convergence of a class of Metropolis-type Markov-chain annealing algorithms for global optimization of a smooth function @math on @math is established. No prior information is assumed as to what bounded region contains a global minimum. The analysis contained herein is based on writing the Metropolis-type algorithm in the form of a recursive stochastic algorithm @math , where @math is a standard white Gaussian sequence, @math are random variables, and @math , @math for k large. Convergence results for @math are then applied from our previous work [SIAM Journal on Control and Optimization, 29 (1991), pp. 999–1018]. Since the analysis of @math is based on the asymptotic behavior of the related Langevin-type Markov diffusion annealing algorithm @math , where @math is a standard Wiener process and $c(t) = ...
@cite_24: Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SGMCMC algorithm. Furthermore, we extend recent SG-MCMC methods with two key components: i) adaptive preconditioners (as in ADAgrad or RMSprop), and ii) adaptive element-wise momentum weights. The zerotemperature limit gives a novel stochastic optimization method with adaptive elementwise momentum weights, while conventional optimization methods only have a shared, static momentum weight. Under certain assumptions, our theoretical analysis suggests the proposed simulated annealing approach converges close to the global optima. Experiments on several deep neural network models show state-of-the-art results compared to related stochastic optimization algorithms.
@cite_19: Soit M une variete de Riemann compacte connexe a N dimensions de produit interieur et soit U:M→R 1 une fonction lisse donnee. Pour β>0, on definit #7B-L β :C ∞ (M)→C ∞ (M) par [#7B-L β O]=exp(βU)•⊇•exp(−βU)•⊇O, ou ⊇ et ⊇• sont le gradient et la divergence sur M. Soit λ(β) la taille de l'ecart entre 0 et le reste du spectre de #7B-L β . On etudie le comportement asymptotique de λ(β)
@cite_2: We study the asymptotic behavior as timet → + ∞ of certain nonstationary Markov chains, and prove the convergence of the annealing algorithm in Monte Carlo simulations. We find that in the limitt → + ∞, a nonstationary Markov chain may exhibit “phase transitions.” Nonstationary Markov chains in general, and the annealing algorithm in particular, lead to biased estimators for the expectation values of the process. We compute the leading terms in the bias and the variance of the sample-means estimator. We find that the annealing algorithm converges if the temperatureT(t) goes to zero no faster thanC log(t t0) ast→+∞, with a computable constantC andt0 the initial time. The bias and the variance of the sample-means estimator in the annealing algorithm go to zero likeO(t−1+e) for some 0⩽e<1, with e=0 only in very special circumstances. Our results concerning the convergence of the annealing algorithm, and the rate of convergence to zero of the bias and the variance of the sample-means estimator, provide a rigorous procedure for choosing the optimal “annealing schedule.” This optimal choice reflects the competition between two physical effects: (a) The “adiabatic” effect, whereby if the temperature is loweredtoo abruptly the system may end up not in a ground state but in a nearby metastable state, and (b) the “super-cooling” effect, whereby if the temperature is loweredtoo slowly the system will indeed approach the ground state(s) but may do so extremely slowly.
@cite_13: Annealing is the process of slowly cooling a physical system in order to obtain states with globally minimum energy. By simulating such a process, near globally-minimum-cost solutions can be found for very large optimization problems. The purpose of this paper is to review the basic theory of simulated annealing, to survey its recent applications, and to survey the theoretical approaches that have been used to study the technique. The applications include image restoration, combinatorial optimization (eg VLSI routing and placement), code design for communication systems and certain aspects of artificial intelligence. The theoretical tools for analysis include the theory of nonstationary Markov chains, statistical physics analysis techniques, large deviation theory and singular perturbation theory.
Finding an approximate minimizer is similar to sampling distributions concentrate around the true minimizer. This well known connection give rise to the study of simulated annealing algorithms, see @cite_22 , @cite_2 , @cite_13 , @cite_18 , @cite_19 , @cite_7 , @cite_6 . Recently, there are many studies further investigate this connection by means of non asymptotic convergence of Langevin based algorithms and in stochastic non-convex optimization and large-scale data analysis, @cite_24 , @cite_30 .