title: Tolerance for termination in Nelder-Mead optimization
body: <p>I am trying to optimize a certain function using the Nelder-Mead method and I need help understanding some of the arguments. I am fairly new to the world of numerical optimizations so, please, forgive my ignorance of what might be obvious to more experienced users. I note that I already looked at <a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html#optimize-minimize-neldermead" rel="nofollow noreferrer">minimize(method=’Nelder-Mead’)</a> and at <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html" rel="nofollow noreferrer">scipy.optimize.minimize</a> but it was not of as much help as I would have hoped. I am trying to optimize function $f$ under two conditions: (i) I want the optimization to stop once the $f$ value is below a certain value and (ii) once the argument is around the optimal value, I don't want the optimizer to increase the step again (i.e., once it gets below the threshold value and stays below for a couple of iterations, I would like the optimization to terminate). Here is the optimization code I use:</p>
  
  <pre><code>scipy.optimize.minimize(fun=f, x0=init_pos, method="nelder-mead",
                          options={"initial_simplex": simplex, 
                                   "disp": True, "maxiter" : 25, 
                                   "fatol": 0.50, "adaptive": True})
  </code></pre>
  
  <p>where <code>f</code> is my function (<em>f : RxR -> [0,sqrt(2))</em>). I understand that <code>x0=init_pos</code> are initial values for <em>f</em>, <code>"initial_simplex": simplex</code> is the initial triangle (in my 2D case), <code>"maxiter" : 25</code> means that the optimizer will run up to 25 iterations before terminating. </p>
  
  <p>Here are things I do not understand/I am not sure about:</p>
  
  <ol>
  <li><p>The website <a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html#optimize-minimize-neldermead" rel="nofollow noreferrer">1</a> says "fatol: Absolute error in func(xopt) between iterations that is acceptable for convergence." Since the optimal value for my function is <em>f(xopt)=0</em>, does <code>"fatol": 0.50</code> mean that the optimization will terminate once the <em>f(x)</em> will have the value of 0.5 or less? If not, how do I modify the condition for termination (in my case, how do I assure that it does stop once <em>f(x)&lt;=0.5</em>)? I am ok if the optimizer runs a few more iterations around the region giving <em>&lt;0.5</em> but right now it tends to jump out of the near optimal region in a completely random way and I would like to be able to prevent it (if possible). </p></li>
  <li><p>Likewise, as far as I understand, "xatol: Absolute error in xopt between iterations that is acceptable for convergence." means that the optimization will terminate once the difference between the optimal and the present arguments is at most <code>xatol</code>. Since in principle I do not know a-priori what the <em>xopt</em> is, does it mean in practice that once |x_n - x_(n+1)|, the optimizer will stop? If no, is there a way of adding a constraint to stop the function once it is near the optimal point?</p></li>
  </ol>
  
  <p>I will appreciate if someone can either answer or give me a better reference than the SciPy documentation. </p>