title: Debugging SSL Handshake problems of kubernetes pods with the internet
body: <p>I recently installed k3s on debian 10 and I am having problems connecting to the internet from inside the pods. Ping, DNS, HTTP, works but TLS is having problems with the Handshake. I made a tcpdump and see ClientHello's and then immediatly a response from the server with internal error SSL error 80. On the VM where cluster runs on everything works fine, just not inside the pods.</p>
  <p>Example in container:</p>
  <pre><code>pacman -Syu
  :: Synchronizing package databases...
  error: failed retrieving file 'core.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed retrieving file 'core.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed retrieving file 'core.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed to update core (download library error)
  error: failed retrieving file 'extra.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed retrieving file 'extra.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed retrieving file 'extra.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed to update extra (download library error)
  error: failed retrieving file 'community.db' from mirror.pkgbuild.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed retrieving file 'community.db' from mirror.rackspace.com : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed retrieving file 'community.db' from mirror.leaseweb.net : error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  error: failed to update community (download library error)
  error: failed to synchronize all databases
  </code></pre>
  <pre><code>curl -4 -v https://www.google.com
  *   Trying 157.230.127.168:443...
  * Connected to www.google.com (157.230.127.168) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/certs/ca-certificates.crt
  *  CApath: none
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * TLSv1.3 (IN), TLS alert, internal error (592):
  * error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  * Closing connection 0
  curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  </code></pre>
  <p>On VM:</p>
  <pre><code>curl -4 -v https://www.google.com
  * Expire in 0 ms for 6 (transfer 0x556e74843fb0)
  &lt;snip&gt;A lot of these expire messages&lt;/snip&gt;
  * Expire in 10 ms for 1 (transfer 0x556e74843fb0)
  *   Trying 172.217.23.100...
  * TCP_NODELAY set
  * Expire in 200 ms for 4 (transfer 0x556e74843fb0)
  * Connected to www.google.com (172.217.23.100) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *   CAfile: none
    CApath: /etc/ssl/certs
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * TLSv1.3 (IN), TLS handshake, Server hello (2):
  * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  * TLSv1.3 (IN), TLS handshake, Certificate (11):
  * TLSv1.3 (IN), TLS handshake, CERT verify (15):
  * TLSv1.3 (IN), TLS handshake, Finished (20):
  * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  * TLSv1.3 (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
  *  start date: Mar 11 15:00:19 2021 GMT
  *  expire date: Jun  3 15:00:18 2021 GMT
  *  subjectAltName: host &quot;www.google.com&quot; matched cert's &quot;www.google.com&quot;
  *  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multi-use
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  * Using Stream ID: 1 (easy handle 0x556e74843fb0)
  &gt; GET / HTTP/2
  &gt; Host: www.google.com
  &gt; User-Agent: curl/7.64.0
  &gt; Accept: */*
  &gt; 
  * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  * old SSL session ID is stale, removing
  * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
  &lt; HTTP/2 200 
  &lt; date: Fri, 26 Mar 2021 01:32:52 GMT
  &lt; expires: -1
  &lt; cache-control: private, max-age=0
  &lt; content-type: text/html; charset=ISO-8859-1
  &lt; p3p: CP=&quot;This is not a P3P policy! See g.co/p3phelp for more info.&quot;
  &lt; server: gws
  &lt; x-xss-protection: 0
  &lt; x-frame-options: SAMEORIGIN
  &lt; set-cookie: NID=212=0Nbn1MMUvMRUNa8iBWgekw-YrWof3yeeBb22v94ZrQ4KkMeVm5wouwkK9ElA353VmZCp_TOXah6KfC_KQX7S48W-IgXlQUz1z4ytqKnDzSXM7X40rLw8tBwMi4oH7eLyE4nGGAfsSlUne28lwMNLobMxYls54iUbgM4x9-kuMCA; expires=Sat, 25-Sep-2021 01:32:52 GMT; path=/; domain=.google.com; HttpOnly
  &lt; alt-svc: h3-29=&quot;:443&quot;; ma=2592000,h3-T051=&quot;:443&quot;; ma=2592000,h3-Q050=&quot;:443&quot;; ma=2592000,h3-Q046=&quot;:443&quot;; ma=2592000,h3-Q043=&quot;:443&quot;; ma=2592000,quic=&quot;:443&quot;; ma=2592000; v=&quot;46,43&quot;
  &lt; accept-ranges: none
  &lt; vary: Accept-Encoding
  &lt;hmtl&gt;...,&lt;/html&gt;
  </code></pre>
  <p>Does anybody have an idea how I can debug this or fix it?</p>
  <p><strong>UPDATE</strong></p>
  <p>Here a curl from inside the VM and a pod. I fixed the Webserver to a known working one on the Internet (Vercel Hosted Website) aswell so we debug the same server. But it is always the same error inside the pod. no matter which server I contact.</p>
  <pre class="lang-sh prettyprint-override"><code>kubectl run -i --tty --rm debug --image=archlinux --restart=Never -- bash
  </code></pre>
  <p>Pod:</p>
  <pre><code>[root@debug /]# curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ &gt; /dev/null
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 157.230.127.168:443...
  * Connected to wegmueller.it (157.230.127.168) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/certs/ca-certificates.crt
  *  CApath: none
  } [5 bytes data]
  * TLSv1.2 (OUT), TLS handshake, Client hello (1):
  } [229 bytes data]
  * TLSv1.2 (IN), TLS alert, internal error (592):
  { [2 bytes data]
  * error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  * Closing connection 0
  curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
  [root@debug /]#
  </code></pre>
  <p>VM:</p>
  <pre><code>curl -v --tlsv1.2 --tls-max 1.2 https://wegmueller.it/ &gt; /dev/null
  * Expire in 0 ms for 6 (transfer 0x559d896cafb0)
  * Expire in 1 ms for 1 (transfer 0x559d896cafb0)
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 0 ms for 1 (transfer 0x559d896cafb0)
  * Expire in 1 ms for 1 (transfer 0x559d896cafb0)
  &lt;snip a ton of expire messages&gt;
  * Expire in 50 ms for 1 (transfer 0x559d896cafb0)
  *   Trying 76.76.21.21...
  * TCP_NODELAY set
  * Expire in 200 ms for 4 (transfer 0x559d896cafb0)
  * Connected to wegmueller.it (76.76.21.21) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *   CAfile: none
    CApath: /etc/ssl/certs
  } [5 bytes data]
  * TLSv1.2 (OUT), TLS handshake, Client hello (1):
  } [223 bytes data]
  * TLSv1.2 (IN), TLS handshake, Server hello (2):
  { [106 bytes data]
  * TLSv1.2 (IN), TLS handshake, Certificate (11):
  { [2458 bytes data]
  * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  { [300 bytes data]
  * TLSv1.2 (IN), TLS handshake, Server finished (14):
  { [4 bytes data]
  * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  } [37 bytes data]
  * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
  } [1 bytes data]
  * TLSv1.2 (OUT), TLS handshake, Finished (20):
  } [16 bytes data]
  * TLSv1.2 (IN), TLS handshake, Finished (20):
  { [16 bytes data]
  * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: CN=wegmueller.it
  *  start date: Mar 16 21:08:10 2021 GMT
  *  expire date: Jun 14 21:08:10 2021 GMT
  *  subjectAltName: host &quot;wegmueller.it&quot; matched cert's &quot;wegmueller.it&quot;
  *  issuer: C=US; O=Let's Encrypt; CN=R3
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multi-use
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  } [5 bytes data]
  * Using Stream ID: 1 (easy handle 0x559d896cafb0)
  } [5 bytes data]
  &gt; GET / HTTP/2
  &gt; Host: wegmueller.it
  &gt; User-Agent: curl/7.64.0
  &gt; Accept: */*
  &gt; 
  { [5 bytes data]
  * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
  } [5 bytes data]
  &lt; HTTP/2 200 
  &lt; date: Sun, 28 Mar 2021 15:24:59 GMT
  &lt; content-type: text/html; charset=utf-8
  &lt; content-disposition: inline; filename=&quot;index&quot;
  &lt; cache-control: public, max-age=0, must-revalidate
  &lt; content-length: 3154
  &lt; access-control-allow-origin: *
  &lt; etag: W/&quot;637eb49039bc1f6a7aa9e4d88a56b0708b722983f403c3c2f717371b2b25a472&quot;
  &lt; accept-ranges: bytes
  &lt; x-vercel-cache: MISS
  &lt; age: 0
  &lt; server: Vercel
  &lt; x-vercel-id: cdg1::z6h9l-1616945098788-71384c5744be
  &lt; strict-transport-security: max-age=63072000
  &lt; 
  { [1112 bytes data]
  100  3154  100  3154    0     0   3604      0 --:--:-- --:--:-- --:--:--  3600
  * Connection #0 to host wegmueller.it left intact
  </code></pre>