title: Is there any "standard" way to calculate the numerical gradient?
body: <p>I am trying to calculate the numerical gradient of a smooth function in c++. And the parameter value could vary from zero to a very large number(maybe 1e10 to 1e20?)</p>
  
  <p>I used the function f(x,y) = 10*x^3 + y^3 as a testbench, but I found that if x or y is too large, I can't get correct gradient.</p>
  
  <p>Here is my code to calculate the graidient:</p>
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;cmath&gt;
  #include &lt;cassert&gt;
  using namespace std;
  double f(double x, double y)
  {
      // black box expensive function
      return 10 * pow(x, 3) + pow(y, 3);
  }
  int main()
  {
      // double x = -5897182590.8347721;
      // double y = 269857217.0017581;
      double x = 1.13041e+19;
      double y = -5.49756e+14;
      const double epsi = 1e-4;
  
      double f1 = f(x, y);
      double f2 = f(x, y+epsi);
      double f3 = f(x, y-epsi);
      cout &lt;&lt; f1 &lt;&lt; endl;
      cout &lt;&lt; f2 &lt;&lt; endl;
      cout &lt;&lt; f3 &lt;&lt; endl;
      cout &lt;&lt; f1 - f2 &lt;&lt; endl; // 0
      cout &lt;&lt; f2 - f3 &lt;&lt; endl; // 0
      return 0;
  }
  </code></pre>
  
  <p>If I use the above code to calculate the gradient, the gradient would be zero!</p>
  
  <p>The testbench function, 10*x^3 + y^3, is just a demo, the real problem I need to solve is actually a black box function.</p>
  
  <p>So, is there any "standard" way to calculate the numerical gradient?</p>