title: istio outlier detection breaking routing with no metrics
body: <p>we have been using istio for some time, but have recently discovered an issue we cant explain with outlier detection.  We have 50+ microservices and have discovered that on some of them &quot;atleast 2-3&quot; traffic does not seem to be load balancing we have tracked this down to outlier detection as once we remove it from the destination rule load balancing works correctly.</p>
  <p>the image shows &lt;1% of the traffic going to the pod ending in 8kh2p
  <a href="https://i.stack.imgur.com/Lfyva.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Lfyva.png" alt="enter image description here" /></a><br />
  My main issue is that even tho we can replicate the issue and resolve it by removing outlier detection, We are seeing no metrics to show that the circuit breaker/outlier detection has been tripped.  As per this github issue - <a href="https://github.com/istio/istio/issues/8902" rel="nofollow noreferrer">https://github.com/istio/istio/issues/8902</a> -  we should be able to track it with something similar to</p>
  <pre><code>sum(istio_requests_total{response_code=&quot;503&quot;, response_flags=&quot;UO&quot;}) by (source_workload, destination_workload, response_code) 
  </code></pre>
  <p>I have also found some envoy documentation to where i should be able to track it with</p>
  <pre><code>envoy_cluster_circuit_breakers_default_cx_open
  </code></pre>
  <p>none of these metrics seem to show anything being triggered.</p>
  <p>I do want to point out a similar post on <a href="https://stackoverflow.com/questions/64677815/istio-stop-loadbalancing-when-add-configs-for-outlier-detections">stackoverflow.com</a> which did not seem to fix our issue</p>
  <p>If anyone could help figure out why things are not load balancing correctly with outlier detection on or at least a way that we can track that its being tripped would be much appreciated. -</p>
  <pre><code>kind: DestinationRule
  apiVersion: networking.istio.io/v1alpha3
  metadata:
    name: some-service-dr
    namespace: some-namespace
  spec:
    host: some-service.some-namespace.svc.cluster.local
    trafficPolicy:
      connectionPool:
        http:
          http1MaxPendingRequests: 1000
          idleTimeout: 3s
          maxRequestsPerConnection: 1
        tcp:
          maxConnections: 500
      outlierDetection:
        consecutive5xxErrors: 0 (disabling as our services expect 500s back)
        consecutiveGatewayErrors: 5 (502, 503, 504 should trigger this)
        interval: 10s
        maxEjectionPercent: 50
      tls:
        mode: ISTIO_MUTUAL
  </code></pre>
  <p>Our virtual-services look like</p>
  <pre><code>kind: VirtualService
  apiVersion: networking.istio.io/v1alpha3
  metadata:
    name: some-service-vs
    namespace: some-namespace
  spec:
    hosts:
      - some-service.some-namespace.svc.cluster.local
    http:
      - retries:
          attempts: 5
          perTryTimeout: 30s
          retryOn: 'connect-failure,refused-stream,reset'
        route:
          - destination:
              host: some-service.some-namespace.svc.cluster.local
              port:
                number: 80
    exportTo:
      - .
  </code></pre>
  <p>Peer Authentication</p>
  <pre><code>kind: PeerAuthentication
  apiVersion: security.istio.io/v1beta1
  metadata:
    name: some-service-tls-policy
    namespace: some-namespace
  spec:
    selector:
      matchLabels:
        app: some-service
    mtls:
      mode: STRICT
    portLevelMtls: ~
  </code></pre>
  <p>Kubernetes version v1.21.x</p>
  <p>Istio version 1.10.x</p>
  <p>Prometheus version 2.28.x</p>
  <p><strong>UPDATE</strong></p>
  <p>I have updated our destination rule to attempt changing consecutive5xxErrors and consecutiveGatewayErrors both to 0 and the issue still persists with 2 pods one pod takes 100% of traffic with no traffic being loadbalanced to the other one.
  New setting below</p>
  <pre><code>outlierDetection:
    interval: 10s
    maxEjectionPercent: 50
  </code></pre>