title: Accessing service using istio ingress gives 503 error when mTLS is enabled
body: <p>I have a mutual TLS enabled Istio mesh. My setup is as follows</p>
  
  <p><a href="https://i.stack.imgur.com/Vcxa4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Vcxa4.png" alt="enter image description here"></a></p>
  
  <ol>
  <li>A service running inside a pod (Service container + envoy)</li>
  <li>An envoy gateway which stays in front of the above service. An Istio Gateway and Virtual Service attached to this. It routes <code>/info/</code> route to the above service.</li>
  <li>Another Istio Gateway configured for ingress using the default istio ingress pod. This also has Gateway+Virtual Service combination. The virtual service directs <code>/info/</code> path to the service described in 2</li>
  </ol>
  
  <p>I'm attempting to access the service from the ingress gateway using a curl command such as:</p>
  
  <pre><code>$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v
  </code></pre>
  
  <p>But I'm getting a 503 not found error as below:</p>
  
  <pre><code>$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v
  Note: Unnecessary use of -X or --request, GET is already inferred.
  *   Trying 10.105.138.94...
  * Connected to istio-ingressgateway.istio-system (10.105.138.94) port 80 (#0)
  &gt; GET /info/ HTTP/1.1
  &gt; Host: istio-ingressgateway.istio-system
  &gt; User-Agent: curl/7.47.0
  &gt; Accept: */*
  &gt; Authorization: Bearer ...
  &gt;
  &lt; HTTP/1.1 503 Service Unavailable
  &lt; content-length: 57
  &lt; content-type: text/plain
  &lt; date: Sat, 12 Jan 2019 13:30:13 GMT
  &lt; server: envoy
  &lt;
  * Connection #0 to host istio-ingressgateway.istio-system left intact
  </code></pre>
  
  <p>I checked the logs of <code>istio-ingressgateway</code> pod and the following line was logged there</p>
  
  <pre><code>[2019-01-13T05:40:16.517Z] "GET /info/ HTTP/1.1" 503 UH 0 19 6 - "10.244.0.5" "curl/7.47.0" "da02fdce-8bb5-90fe-b422-5c74fe28759b" "istio-ingressgateway.istio-system" "-"
  </code></pre>
  
  <p>If I logged into istio ingress pod and attempt to send the request with curl, I get a successful 200 OK.</p>
  
  <pre><code># curl hr--gateway-service.default/info/ -H "Authorization: Bearer $token" -v
  </code></pre>
  
  <p>Also, I managed to get a successful response for the same curl command when the mesh was created in mTLS disabled mode. There are no conflicts shown in mTLS setup.</p>
  
  <p>Here are the config details for my service mesh in case you need additional info.</p>
  
  <p><strong>Pods</strong></p>
  
  <pre><code>$ kubectl get pods --all-namespaces
  NAMESPACE       NAME                                                       READY     STATUS      RESTARTS   AGE
  default         hr--gateway-deployment-688986c87c-z9nkh                    1/1       Running     0          37m
  default         hr--hr-deployment-596946948d-c89bn                         2/2       Running     0          37m
  default         hr--sts-deployment-694d7cff97-gjwdk                        1/1       Running     0          37m
  ingress-nginx   default-http-backend-6586bc58b6-8qss6                      1/1       Running     0          42m
  ingress-nginx   nginx-ingress-controller-6bd7c597cb-t4rwq                  1/1       Running     0          42m
  istio-system    grafana-85dbf49c94-lfpbr                                   1/1       Running     0          42m
  istio-system    istio-citadel-545f49c58b-dq5lq                             1/1       Running     0          42m
  istio-system    istio-cleanup-secrets-bh5ws                                0/1       Completed   0          42m
  istio-system    istio-egressgateway-7d59954f4-qcnxm                        1/1       Running     0          42m
  istio-system    istio-galley-5b6449c48f-72vkb                              1/1       Running     0          42m
  istio-system    istio-grafana-post-install-lwmsf                           0/1       Completed   0          42m
  istio-system    istio-ingressgateway-8455c8c6f7-5khtk                      1/1       Running     0          42m
  istio-system    istio-pilot-58ff4d6647-bct4b                               2/2       Running     0          42m
  istio-system    istio-policy-59685fd869-h7v94                              2/2       Running     0          42m
  istio-system    istio-security-post-install-cqj6k                          0/1       Completed   0          42m
  istio-system    istio-sidecar-injector-75b9866679-qg88s                    1/1       Running     0          42m
  istio-system    istio-statsd-prom-bridge-549d687fd9-bspj2                  1/1       Running     0          42m
  istio-system    istio-telemetry-6ccf9ddb96-hxnwv                           2/2       Running     0          42m
  istio-system    istio-tracing-7596597bd7-m5pk8                             1/1       Running     0          42m
  istio-system    prometheus-6ffc56584f-4cm5v                                1/1       Running     0          42m
  istio-system    servicegraph-5d64b457b4-jttl9                              1/1       Running     0          42m
  kube-system     coredns-78fcdf6894-rxw57                                   1/1       Running     0          50m
  kube-system     coredns-78fcdf6894-s4bg2                                   1/1       Running     0          50m
  kube-system     etcd-ubuntu                                                1/1       Running     0          49m
  kube-system     kube-apiserver-ubuntu                                      1/1       Running     0          49m
  kube-system     kube-controller-manager-ubuntu                             1/1       Running     0          49m
  kube-system     kube-flannel-ds-9nvf9                                      1/1       Running     0          49m
  kube-system     kube-proxy-r868m                                           1/1       Running     0          50m
  kube-system     kube-scheduler-ubuntu                                      1/1       Running     0          49m
  </code></pre>
  
  <p><strong>Services</strong></p>
  
  <pre><code>$ kubectl get svc --all-namespaces
  NAMESPACE       NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                                                                   AGE
  
  default         hr--gateway-service                     ClusterIP      10.100.238.144   &lt;none&gt;         80/TCP,443/TCP                                                                                                            39m
  default         hr--hr-service                          ClusterIP      10.96.193.43     &lt;none&gt;         80/TCP                                                                                                                    39m
  default         hr--sts-service                         ClusterIP      10.99.54.137     &lt;none&gt;         8080/TCP,8081/TCP,8090/TCP                                                                                                39m
  default         kubernetes                              ClusterIP      10.96.0.1        &lt;none&gt;         443/TCP                                                                                                                   52m
  ingress-nginx   default-http-backend                    ClusterIP      10.109.166.229   &lt;none&gt;         80/TCP                                                                                                                    44m
  ingress-nginx   ingress-nginx                           NodePort       10.108.9.180     192.168.60.3   80:31001/TCP,443:32315/TCP                                                                                                44m
  istio-system    grafana                                 ClusterIP      10.102.141.231   &lt;none&gt;         3000/TCP                                                                                                                  44m
  istio-system    istio-citadel                           ClusterIP      10.101.128.187   &lt;none&gt;         8060/TCP,9093/TCP                                                                                                         44m
  istio-system    istio-egressgateway                     ClusterIP      10.102.157.204   &lt;none&gt;         80/TCP,443/TCP                                                                                                            44m
  istio-system    istio-galley                            ClusterIP      10.96.31.251     &lt;none&gt;         443/TCP,9093/TCP                                                                                                          44m
  istio-system    istio-ingressgateway                    LoadBalancer   10.105.138.94    &lt;pending&gt;      80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31219/TCP,8060:31482/TCP,853:30034/TCP,15030:31544/TCP,15031:32652/TCP   44m
  istio-system    istio-pilot                             ClusterIP      10.100.170.73    &lt;none&gt;         15010/TCP,15011/TCP,8080/TCP,9093/TCP                                                                                     44m
  istio-system    istio-policy                            ClusterIP      10.104.77.184    &lt;none&gt;         9091/TCP,15004/TCP,9093/TCP                                                                                               44m
  istio-system    istio-sidecar-injector                  ClusterIP      10.100.180.152   &lt;none&gt;         443/TCP                                                                                                                   44m
  istio-system    istio-statsd-prom-bridge                ClusterIP      10.107.39.50     &lt;none&gt;         9102/TCP,9125/UDP                                                                                                         44m
  istio-system    istio-telemetry                         ClusterIP      10.110.55.232    &lt;none&gt;         9091/TCP,15004/TCP,9093/TCP,42422/TCP                                                                                     44m
  istio-system    jaeger-agent                            ClusterIP      None             &lt;none&gt;         5775/UDP,6831/UDP,6832/UDP                                                                                                44m
  istio-system    jaeger-collector                        ClusterIP      10.102.43.21     &lt;none&gt;         14267/TCP,14268/TCP                                                                                                       44m
  istio-system    jaeger-query                            ClusterIP      10.104.182.189   &lt;none&gt;         16686/TCP                                                                                                                 44m
  istio-system    prometheus                              ClusterIP      10.100.0.70      &lt;none&gt;         9090/TCP                                                                                                                  44m
  istio-system    servicegraph                            ClusterIP      10.97.65.37      &lt;none&gt;         8088/TCP                                                                                                                  44m
  istio-system    tracing                                 ClusterIP      10.109.87.118    &lt;none&gt;         80/TCP                                                                                                                    44m
  kube-system     kube-dns                                ClusterIP      10.96.0.10       &lt;none&gt;         53/UDP,53/TCP                                                                                                             52m
  </code></pre>
  
  <p><strong>Gateway and virtual service described in point 2</strong></p>
  
  <pre><code>$ kubectl describe gateways.networking.istio.io hr--gateway
  Name:         hr--gateway
  Namespace:    default
  API Version:  networking.istio.io/v1alpha3
  Kind:         Gateway
  Metadata:
    ...
  Spec:
    Selector:
      App:                           hr--gateway
    Servers:
      Hosts:
        *
      Port:
        Name:      http2
        Number:    80
        Protocol:  HTTP2
      Hosts:
        *
      Port:
        Name:      https
        Number:    443
        Protocol:  HTTPS
      Tls:
        Mode:  PASSTHROUGH
  
  $ kubectl describe virtualservices.networking.istio.io hr--gateway
  Name:         hr--gateway
  Namespace:    default
  Labels:       app=hr--gateway
                Annotations:  &lt;none&gt;
  API Version:  networking.istio.io/v1alpha3
  Kind:         VirtualService
  Metadata:
    ...
  Spec:
    Gateways:
      hr--gateway
    Hosts:
      *
    Http:
      Match:
        Uri:
          Prefix:  /info/
      Rewrite:
        Uri:  /
      Route:
        Destination:
          Host:  hr--hr-service
  </code></pre>
  
  <p><strong>Gateway and virtual service described in point 3</strong></p>
  
  <pre><code>$ kubectl describe gateways.networking.istio.io ingress-gateway
  Name:         ingress-gateway
  Namespace:    default
  Labels:       &lt;none&gt;
  Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"ingress-gateway","namespace":"default"},"spec":{"sel...
  API Version:  networking.istio.io/v1alpha3
  Kind:         Gateway
  Metadata:
    ...
  Spec:
    Selector:
      Istio:  ingressgateway
    Servers:
      Hosts:
        *
      Port:
        Name:      http2
        Number:    80
        Protocol:  HTTP2
  
  $ kubectl describe virtualservices.networking.istio.io hr--gateway-ingress-vs
  Name:         hr--gateway-ingress-vs
  Namespace:    default
  Labels:       app=hr--gateway
                API Version:  networking.istio.io/v1alpha3
  Kind:         VirtualService
  Metadata:
  Spec:
    Gateways:
      ingress-gateway
    Hosts:
      *
    Http:
      Match:
        Uri:
          Prefix:  /info/
      Route:
        Destination:
          Host:  hr--gateway-service
  Events:        &lt;none&gt;
  </code></pre>