title: Unable to access pod over HTTPS via Istio Gateway (running as ELB) on AWS EKS with mTLS enabled on the cluster
body: <p>We have installed Istio 1.0.5 on a newly launched EKS cluster via Helm.</p>
  
  <pre><code>helm install \
      --wait \
      --name istio \
      --namespace istio-system \
      --set tracing.enabled=true \
      --set kiali.enabled=true \
      --set grafana.enabled=true \
      --set global.mtls.enabled=true \
      --set servicegraph.enabled=true \
      --set global.proxy.includeIPRanges="10.10.0.0/16\,172.20.0.0/16" \
      install/kubernetes/helm/istio 
  
  (the IP ranges are pod and service CIDR ranges of the cluster)
  </code></pre>
  
  <p>This is what we want to achieve:
  <a href="https://i.stack.imgur.com/771Ao.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/771Ao.png" alt="istio ingress gateway"></a></p>
  
  <p>To enable attaching ACM certs to the ELB launched by the istio-ingressgateway, following annotations were added to the istio-ingressgateway service:</p>
  
  <pre><code>service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
  service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:ACCOUNT:certificate/MY_CERT
  service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
  </code></pre>
  
  <p>The Ingress Gateway Loadbalancer service is getting created as expected:</p>
  
  <p><a href="https://i.stack.imgur.com/XmL7B.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XmL7B.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/VylOH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VylOH.png" alt="enter image description here"></a></p>
  
  <p>We are following this link to implement the above scenario:</p>
  
  <p><a href="https://doc.istio.cn/en/docs/examples/advanced-gateways/ingress-sni-passthrough/" rel="nofollow noreferrer">https://doc.istio.cn/en/docs/examples/advanced-gateways/ingress-sni-passthrough/</a></p>
  
  <p>The certificates and secrets are created according to the above Istio documentation. The Service and Deployment resources are created as follows:</p>
  
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    name: my-nginx
    labels:
      run: my-nginx
  spec:
    ports:
    - port: 443
      protocol: TCP
    selector:
      run: my-nginx
  
  ---
  
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: my-nginx
  spec:
    selector:
      matchLabels:
        run: my-nginx
    replicas: 1
    template:
      metadata:
        labels:
          run: my-nginx
      spec:
        containers:
        - name: my-nginx
          image: nginx
          ports:
          - containerPort: 443
          volumeMounts:
          - name: nginx-config
            mountPath: /etc/nginx
            readOnly: true
          - name: nginx-server-certs
            mountPath: /etc/nginx-server-certs
            readOnly: true
        volumes:
        - name: nginx-config
          configMap:
            name: nginx-configmap
        - name: nginx-server-certs
          secret:
            secretName: nginx-server-certs
  </code></pre>
  
  <p>The Gateway and VirtualService resources are created as follows:</p>
  
  <pre><code>apiVersion: networking.istio.io/v1alpha3
  kind: Gateway
  metadata:
    name: mygateway
  spec:
    selector:
      istio: ingressgateway # use istio default ingress gateway
    servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: PASSTHROUGH
      hosts:
      - nginx.example.com
  
  ---
  
  apiVersion: networking.istio.io/v1alpha3
  kind: VirtualService
  metadata:
    name: nginx
  spec:
    hosts:
    - nginx.example.com
    gateways:
    - mygateway
    tls:
    - match:
      - port: 443
        sni_hosts:
        - nginx.example.com
      route:
      - destination:
          host: my-nginx
          port:
            number: 443
  </code></pre>
  
  <p>The resources are getting created as expected:</p>
  
  <p><a href="https://i.stack.imgur.com/Qcy68.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Qcy68.png" alt="enter image description here"></a></p>
  
  <p>We are able to get a response from the nginx container from within the pod:</p>
  
  <pre><code>curl --insecure --verbose https://localhost
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/b3BQM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/b3BQM.png" alt="enter image description here"></a></p>
  
  <p>We are also able to get a response from the nginx pod via its service from another pod:</p>
  
  <pre><code>curl --insecure --verbose https://my-nginx
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/GS0qT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GS0qT.png" alt="enter image description here"></a></p>
  
  <p>However, when we try to access the nginx pod via the Istio ingress gateway, the service is not accessible:</p>
  
  <pre><code>curl --insecure --verbose https://a1a99cb2d34f711e9865b0295f80a9c0-303710594.us-east-1.elb.amazonaws.com
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/wxfAC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wxfAC.png" alt="enter image description here"></a></p>
  
  <p>After enabling access log for the ELB, this is the relevant log line:</p>
  
  <pre><code>2019-02-22T12:36:08.960931Z a1a99cb2d34f711e9865b0295f80a9c0 3.84.67.219:56786 - -1 -1 -1 503 0 0 0 "GET https://a1a99cb2d34f711e9865b0295f80a9c0-303710594.us-east-1.elb.amazonaws.com:443/ HTTP/1.1" "curl/7.55.1" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
  </code></pre>
  
  <p>We got inside the istio-ingressgateway pod and tried accessing the nginx pod via its service name(my-nginx), service ip(172.20.198.12), service dns(my-nginx.default.svc.cluster.local), and pod ip(10.10.157.58).
  But none of them were accessible:</p>
  
  <p><a href="https://i.stack.imgur.com/HGFlT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HGFlT.png" alt="enter image description here"></a></p>
  
  <p>Do we need to create any extra VirtualService or ServiceEntry?
  When we tried accessing a Deployment exposed on port 80, via the Istio ingressgateway, we were able to successfully hit the pod.
  The issue arises only when we try to enable HTTPS on the Istio ingressgateway service.
  Is this occuring because we have enabled mTLS?
  Please help us find any pointers as to how to proceed.</p>