title: HPA NOT scale up on Custom Metrics after updating targetValue
body: <p>I have an AWS EKS cluster, and I have set up the Prometheus and the Prometheus-Adapter.
  The thing is, I have tried different values for my HPA metrics target value (the threshold one), in some cases, I can see the scale-up works, but the other fails.</p>
  <p>Provided my HPA:</p>
  <pre><code>apiVersion: autoscaling/v2beta2
  kind: HorizontalPodAutoscaler
  metadata:
    name: tianbing-xxxxxxxxx-xxxxxxx-hpa
    labels:
  spec:
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: tianbing-xxxxxxxxx-xxxxxxx
    minReplicas: 1
    maxReplicas: 5
    metrics:
      - type: Pods
        pods:
          metric:
            name: plex_queue_size
          target:
            type: AverageValue
            averageValue: 2
    behavior:
      scaleUp:
        stabilizationWindowSeconds: 0
      scaleDown:
        stabilizationWindowSeconds: 300
  </code></pre>
  <p>When I set the target value to 2 in this case, I can see my HPA works perfectly ✅ , here is the logs:
  You can see when my current value is 8, it was successfully scaled to 8/2 = 4 nodes ✅ ,:</p>
  <pre><code>$ kubectl get hpa -w
  NAME                             REFERENCE                               TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   &lt;unknown&gt;/2   1         5         1          31s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   0/2           1         5         1          46s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   8/2           1         5         1          3m34s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   8/2           1         5         4          3m49s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   8/2           1         5         4          4m5s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   7666m/2       1         5         5          4m20s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   7666m/2       1         5         5          4m25s
  </code></pre>
  <p>However, now, if I update my HPA file, changing the target value to a larger one, say 5. (I delete the whole HPA and create a new one.)</p>
  <pre><code>metrics:
      - type: Pods
        pods:
          metric:
            name: plex_queue_size
          target:
            type: AverageValue
            averageValue: 5
  </code></pre>
  <p>From the log, you can see I can scrape the target number (0 in the beginning), but I after I have waited for a long time, there is NO scale-up behavior happens ❌ . The current value is scraped but no scale-up action happens ❌.
  In my case, I should have 8/5, which is 2 desired replicas, however, nothing happens. ❌</p>
  <p>This is what I have when I get the hpa and describe the HPA:</p>
  <pre><code>NAME                             REFERENCE                              TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   &lt;unknown&gt;/5   1         5         1          75s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   0/5           1         5         1          91s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   8/5           1         5         1          3m32s
  tianbing-xxxxxxxxx-xxxxxxx-hpa   Deployment/tianbing-xxxxxxxxx-xxxxxxx   7/5           1         5         1          4m18s
  
  
  
  kubectl describe horizontalpodautoscaler.autoscaling/tianbing-xxxxxxxxx-xxxxxxx-hpa
  Name:                        tianbing-xxxxxxxxx-xxxxxxx-hpa
  Namespace:                    default
  Labels:                       app.kubernetes.io/managed-by=Helm
  Annotations:                  meta.helm.sh/release-name: tianbing
                                meta.helm.sh/release-namespace: default
  CreationTimestamp:            Wed, 10 Mar 2021 14:14:13 -0800
  Reference:                    Deployment/tianbing-xxxxxxxxx-xxxxxxx
  Metrics:                      ( current / target )
    &quot;plex_queue_size&quot; on pods:  7 / 5
  Min replicas:                 1
  Max replicas:                 5
  Deployment pods:              1 current / 1 desired
  Conditions:
    Type            Status  Reason              Message
    ----            ------  ------              -------
    AbleToScale     True    ReadyForNewScale    recommended size matches current size
    ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from pods metric plex_queue_size
    ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
  Events:
    Type     Reason                        Age                    From                       Message
    ----     ------                        ----                   ----                       -------
    Warning  FailedGetPodsMetric           6m10s (x5 over 7m12s)  horizontal-pod-autoscaler  unable to get metric plex_queue_size: unable to fetch metrics from custom metrics API: the server could not find the metric plex_queue_size for pods
    Warning  FailedComputeMetricsReplicas  6m10s (x5 over 7m12s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric plex_queue_size: unable to fetch metrics from custom metrics API: the server could not find the metric plex_queue_size for pods
  </code></pre>
  <p>Note: From the description, I saw some <code>Event</code> for <code>FailedGetPodsMetric</code> and <code>FailedComputeMetricsReplicas</code>, I believe that is just because the initial setup of HPA, since from HPA, I can scrape the metrics and also the <code>AbleToScale</code> and <code>ScalingActive</code> are also <code>True</code>.</p>