I implemented downhill simplex simulated annealing algorithm. Algorithm is very hard to tune, w.r.t. parameters including cooling schedule, starting temperature... My first question is about convergence. In general, independently from choice of initial parameters, is it sure that algorithm converges to global optimal solution?  More precisely: I use SA with downhill simplex based on Num.recipes, however, I am adding some specific actions for observing constraints. Do you think my additional conditions for constraints are cause for bad convergence? Then, I would like your advice on how to go on with solver conception: for instance, my algo is tested on data in 10-D space (10 parameters). It goes as follows: a preprocessing (monte carlo) picks up a good starting point. Then I run simplex downhill --> does not converge. I run simulated annealing (with simplex downhill to go to next step), and it remains far to solution (RMSE == 0.001). Excel solver on these data does the job very good. Excel method is generalized reduced gradient. Should i go on with fine tuning for SA, or go to non-derivative-free methods s.a. GRG ?  Finally, to speak the same language: with my example, convergence is not reach AT ALL. i don't have time to reach a local minimum either, since I have put some exit conditions when the simplex gets stuck. what does one mean in general with 'solver does not converge'? does it mean that termination condition (s.a. go below a tolerance value) is met and solution is sub optimal, or can it mean that termination condition is never met?Thanks & regards.EDITMy objective function is given explicitly. It can, for instance, look like$F(p) = \sum_{j=1}^{n}\Bigg(y_{j} - (p_{1} + p_{2}\exp{\Big(-\frac{(x_{i}-p_{3})^{2}}{p_{4}}\Big)} + p_{5}\exp{\Big(-\frac{(x_{j}-p_{6})^{2}}{p_{7}}\Big)} + p_{8}\exp{\Big(-\frac{(x_{j}-p_{9})^{2}}{p_{10}}\Big))}\Bigg)^{2}$so, the squared residuals with a 'three Gaussians + step' model.This is an example, solver should solve for any combination of functions among gaussians, exponential, power, polynomial, linear, rational with 'reasonable' number of parameters.Box constraints on parameters are specified, with form $l_{i} \leq p_{i} \leq u_{i}$.
Simulated Annealing proof of convergence
optimization;algorithms;iterative method;random;stochastic
There is a theorem that syas that a black box algorithm is guaranteed to find the global minimum of an arbitrary smooth (i.e., twice continuously differentiable) function if and only if it samples points densely in the search space. Here dense is meant in the topological sense, i.e., it must sample points in arbirarily small neighborhoods of every point. In this case, the worst case complexity is $O(\Big(\frac{d}{\delta}\Big)^{-n})$, [the Latex parser chokes with nested big brackets] where $d$ is the diameter of the search space and $\delta$ the guaranteed error in $x$. Edit: While this looks like being the worst case complexity for locating an $\delta$-accurate $x$ rather than for locating a point for a $\epsilon$-accurate $f$, the complexity for the latter is as bad (even for the rather big value of $\epsilon=(f_{first}-f_{global})/2$), as one can easily construct a smooth function which interpolates all data points so far but takes much smaller values in the center of the largest open ball not containing one of the points evaluated.Thus guaranteed convergence to a global minimum is worthless in practice. (For example, uniformly random search has a convergence guarantee to the global minimizer, whereas most practically fast algorithms don't have one.)Note that simulated annealing usually performs much worse than modern methods. Rather use code recommended in:Comparison of derivative-free optimization algorithms (2012, by Nick Sahinidis)Black-Box Optimization Benchmarking (BBOB) 2012(by Auger, Hansen, et al.)