<p dir="auto">Hello,</p>
<p dir="auto">I am looking at the gradient computation in Gaussian Process Kernel module. My understanding is that there we are trying to compute <math-renderer class="js-inline-math" style="display: inline" data-static-url="https://github.githubassets.com/static" data-run-id="a887106cd854ad6ec07494230699608b">$$\partial{K(x,x')}{\partial \theta}$$</math-renderer>, where <math-renderer class="js-inline-math" style="display: inline" data-static-url="https://github.githubassets.com/static" data-run-id="a887106cd854ad6ec07494230699608b">$$\theta$$</math-renderer> is a hyperparameter. However, I am not sure that is what is computed in the code:</p>
<p dir="auto">For example, the ConstantKernel has:<br>
</p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/scikit-learn/scikit-learn/blob/f339609ab80ee79924cc739edc774182f4870c7c/sklearn/gaussian_process/kernels.py#L1013">scikit-learn/sklearn/gaussian_process/kernels.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 1013
      in
      <a data-pjax="true" class="commit-tease-sha" href="/scikit-learn/scikit-learn/commit/f339609ab80ee79924cc739edc774182f4870c7c">f339609</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L1013" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1013"></td>
          <td id="LC1013" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">self</span>.<span class="pl-s1">constant_value</span>, </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
While I think instead of filling in the constant value, we should fill in just 1.<p></p>
<p dir="auto">Another example is the RBF kernel,<br>
</p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/scikit-learn/scikit-learn/blob/f339609ab80ee79924cc739edc774182f4870c7c/sklearn/gaussian_process/kernels.py#L1232">scikit-learn/sklearn/gaussian_process/kernels.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 1232
      in
      <a data-pjax="true" class="commit-tease-sha" href="/scikit-learn/scikit-learn/commit/f339609ab80ee79924cc739edc774182f4870c7c">f339609</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L1232" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1232"></td>
          <td id="LC1232" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">elif</span> <span class="pl-c1">not</span> <span class="pl-s1">self</span>.<span class="pl-s1">anisotropic</span> <span class="pl-c1">or</span> <span class="pl-s1">length_scale</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>] <span class="pl-c1">==</span> <span class="pl-c1">1</span>: </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
I think the gradient should be further divided by the <strong>length_scale</strong>.<p></p>
<p dir="auto">It seems to me that rather than computing the <strong>gradient</strong>, we are computing <strong>gradient * hyperparameter</strong> here. Am I missing something?</p>
<p dir="auto">Thanks!<br>
Junteng</p>
<h4 dir="auto">Description</h4>
<p dir="auto">As discovered in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="208141122" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/8373" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/8373/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/8373">#8373</a>, the docstring states that if <code class="notranslate">eval_gradient</code> is true, the gradient with respect to the kernel hyperparameters is returned, whereas actually it returns the gradient with respect to the log-transformed hyperparameters. As an example:</p>
<blockquote>
<p dir="auto">Writing the <code class="notranslate">DotProduct</code> kernel as a function of <code class="notranslate">theta</code> instead of <code class="notranslate">sigma_0</code> with <code class="notranslate">sigma_0 = exp(theta)</code> gives us:<br>
<code class="notranslate">k(x_i, x_j) = sigma_0 ^ 2 + x_i \cdot x_j = exp(theta) ^ 2 + x_i \cdot x_j = exp(2*theta) + x_i \cdot x_j</code><br>
Now the gradient with respect to theta is :<br>
<code class="notranslate">2*exp(2*theta) = 2*exp(theta)^2 = 2*\sigma_0^2</code></p>
</blockquote>