title: AWS NLB + NGINX Ingress + websockets == 502 Bad Gateway or Bad Handshake
body: <p>Ingress controller configuration (no changes to the file below)</p>
  <p><a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/aws/deploy.yaml" rel="nofollow noreferrer">https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/aws/deploy.yaml</a></p>
  <p><strong>Service</strong></p>
  <pre><code>kind: Service
  apiVersion: v1
  metadata:
    name: websockets-service
    namespace: development
    annotations:
      argocd.argoproj.io/sync-wave: &quot;10&quot;
  spec:
    selector:
      app: project-websocket
    ports:
    #- name: http
    #  protocol: TCP
    #  port: 80
    #  targetPort: 9000
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
    type: ClusterIP
  </code></pre>
  <p><strong>Ingress</strong></p>
  <pre><code>apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    name: websockets-ingress
    namespace: development
    annotations:
      kubernetes.io/ingress.class: &quot;nginx&quot;
      nginx.ingress.kubernetes.io/proxy-send-timeout: &quot;3600&quot;
      nginx.ingress.kubernetes.io/proxy-read-timeout: &quot;3600&quot;
      nginx.ingress.kubernetes.io/proxy-connect-timeout: &quot;3600&quot;
  
      # nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/secure-backends: &quot;true&quot;
      nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;
      nginx.ingress.kubernetes.io/websocket-services: websockets-service
      nginx.org/websocket-services: websockets-service
  
      nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_set_header HOST $host;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass_request_headers on;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection &quot;Upgrade&quot;;
      argocd.argoproj.io/sync-wave: &quot;10&quot;
  spec:
    tls:
      - hosts:
        - backend-dev.project.com
        secretName: tls-secret
    rules:
    - host: backend-dev.project.com
      http:
        paths:
        - path: /ws/
          backend:
            serviceName: websockets-service
            servicePort: 443
  
  </code></pre>
  <p>trying various CLI websocket clients:</p>
  <pre><code>➜ ws --insecure wss://backend-dev.project.com/ws/table/123
  websocket: bad handshake
  
  ➜ websocat -k  wss://backend-dev.project.com/ws/table/123
  websocat: WebSocketError: Received unexpected status code (502 Bad Gateway)
  websocat: error running
  </code></pre>
  <p>Everything work if I proxy port from the service:</p>
  <pre><code>k port-forward svc/websockets-service 8443:443
  Forwarding from 127.0.0.1:8443 -&gt; 443
  Forwarding from [::1]:8443 -&gt; 443
  Handling connection for 8443
  
  ➜ websocat -k wss://localhost:8443/ws/table/123
  {&quot;connect&quot;: {&quot;channel_name&quot;: &quot;specific.239f96e4a93a470688a12fc0bc8d0374!9d92256f6f65446386fb17b59d1aac57&quot;, &quot;table_token&quot;: &quot;123&quot;}}
  
  and in logs I see
  
  127.0.0.1:50070 - - [21/Nov/2020:05:11:50] &quot;WSCONNECTING /ws/table/123&quot; - -
  127.0.0.1:50070 - - [21/Nov/2020:05:11:50] &quot;WSCONNECT /ws/table/123&quot; - -
  
  </code></pre>
  <p><strong>SSL info</strong></p>
  <pre><code>➜ curl -vvI https://backend-dev.project.com
  *   Trying 44.235.41.143:443...
  * Connected to backend-dev.project.com (44.235.41.143) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * TLSv1.3 (IN), TLS handshake, Server hello (2):
  * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  * TLSv1.3 (IN), TLS handshake, Certificate (11):
  * TLSv1.3 (IN), TLS handshake, CERT verify (15):
  * TLSv1.3 (IN), TLS handshake, Finished (20):
  * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  * TLSv1.3 (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: CN=*.project.com
  *  start date: Oct 28 00:00:00 2020 GMT
  *  expire date: Oct 28 23:59:59 2021 GMT
  *  subjectAltName: host &quot;backend-dev.project.com&quot; matched cert's &quot;*.project.com&quot;
  *  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multi-use
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  * Using Stream ID: 1 (easy handle 0x5579ce4a9af0)
  &gt; HEAD / HTTP/2
  &gt; Host: backend-dev.project.com
  &gt; user-agent: curl/7.69.1
  &gt; accept: */*
  &gt; 
  * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  * old SSL session ID is stale, removing
  * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
  &lt; HTTP/2 200 
  HTTP/2 200 
  &lt; date: Sat, 21 Nov 2020 05:13:22 GMT
  date: Sat, 21 Nov 2020 05:13:22 GMT
  &lt; content-type: text/html; charset=utf-8
  content-type: text/html; charset=utf-8
  &lt; content-length: 143552
  content-length: 143552
  &lt; vary: Accept-Encoding
  vary: Accept-Encoding
  &lt; server-timing: TimerPanel_utime;dur=31.268000000011398;desc=&quot;User CPU time&quot;, TimerPanel_stime;dur=0.0;desc=&quot;System CPU time&quot;, TimerPanel_total;dur=31.268000000011398;desc=&quot;Total CPU time&quot;, TimerPanel_total_time;dur=31.340599060058594;desc=&quot;Elapsed time&quot;, SQLPanel_sql_time;dur=0;desc=&quot;SQL 0 queries&quot;, CachePanel_total_time;dur=0;desc=&quot;Cache 0 Calls&quot;
  server-timing: TimerPanel_utime;dur=31.268000000011398;desc=&quot;User CPU time&quot;, TimerPanel_stime;dur=0.0;desc=&quot;System CPU time&quot;, TimerPanel_total;dur=31.268000000011398;desc=&quot;Total CPU time&quot;, TimerPanel_total_time;dur=31.340599060058594;desc=&quot;Elapsed time&quot;, SQLPanel_sql_time;dur=0;desc=&quot;SQL 0 queries&quot;, CachePanel_total_time;dur=0;desc=&quot;Cache 0 Calls&quot;
  &lt; x-content-type-options: nosniff
  x-content-type-options: nosniff
  &lt; x-frame-options: DENY
  x-frame-options: DENY
  &lt; vary: Cookie, Origin
  vary: Cookie, Origin
  &lt; set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
  set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
  &lt; strict-transport-security: max-age=15724800; includeSubDomains
  strict-transport-security: max-age=15724800; includeSubDomains
  
  &lt; 
  * Connection #0 to host backend-dev.project.com left intact
  
  </code></pre>