title: Exposing bare metal kubernetes cluster to internet
body: <p>I am trying to setup own single-node kubernetes cluster on bare metal dedicated server. I am not that experienced in dev-ops but I need some service to be deployed for my own project. I already did a cluster setup with <code>juju</code> and <code>conjure-up kubernetes</code> over <code>LXD</code>. I have running cluster pretty fine.</p>
  
  <pre><code>$ juju status
  
  Model                         Controller                Cloud/Region         Version  SLA          Timestamp
  conjure-canonical-kubern-3b3  conjure-up-localhost-db9  localhost/localhost  2.4.3    unsupported  23:49:09Z
  
  App                    Version  Status  Scale  Charm                  Store       Rev  OS      Notes
  easyrsa                3.0.1    active      1  easyrsa                jujucharms  195  ubuntu
  etcd                   3.2.10   active      3  etcd                   jujucharms  338  ubuntu
  flannel                0.10.0   active      2  flannel                jujucharms  351  ubuntu
  kubeapi-load-balancer  1.14.0   active      1  kubeapi-load-balancer  jujucharms  525  ubuntu  exposed
  kubernetes-master      1.13.1   active      1  kubernetes-master      jujucharms  542  ubuntu
  kubernetes-worker      1.13.1   active      1  kubernetes-worker      jujucharms  398  ubuntu  exposed
  
  Unit                      Workload  Agent  Machine  Public address  Ports           Message
  easyrsa/0*                active    idle   0        10.213.117.66                   Certificate Authority connected.
  etcd/0*                   active    idle   1        10.213.117.171  2379/tcp        Healthy with 3 known peers
  etcd/1                    active    idle   2        10.213.117.10   2379/tcp        Healthy with 3 known peers
  etcd/2                    active    idle   3        10.213.117.238  2379/tcp        Healthy with 3 known peers
  kubeapi-load-balancer/0*  active    idle   4        10.213.117.123  443/tcp         Loadbalancer ready.
  kubernetes-master/0*      active    idle   5        10.213.117.172  6443/tcp        Kubernetes master running.
    flannel/1*              active    idle            10.213.117.172                  Flannel subnet 10.1.83.1/24
  kubernetes-worker/0*      active    idle   7        10.213.117.136  80/tcp,443/tcp  Kubernetes worker running.
    flannel/4               active    idle            10.213.117.136                  Flannel subnet 10.1.27.1/24
  
  Entity  Meter status  Message
  model   amber         user verification pending
  
  Machine  State    DNS             Inst id        Series  AZ  Message
  0        started  10.213.117.66   juju-b03445-0  bionic      Running
  1        started  10.213.117.171  juju-b03445-1  bionic      Running
  2        started  10.213.117.10   juju-b03445-2  bionic      Running
  3        started  10.213.117.238  juju-b03445-3  bionic      Running
  4        started  10.213.117.123  juju-b03445-4  bionic      Running
  5        started  10.213.117.172  juju-b03445-5  bionic      Running
  7        started  10.213.117.136  juju-b03445-7  bionic      Running
  </code></pre>
  
  <p>I also deployed Hello world application to output some hello on port <code>8080</code> inside the pod and <code>nginx-ingress</code> for it to re-route the traffic to this service on specified host.</p>
  
  <pre><code>NAME                               READY   STATUS    RESTARTS   AGE
  pod/hello-world-696b6b59bd-fznwr   1/1     Running   1          176m
  
  NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
  service/example-service   NodePort    10.152.183.53   &lt;none&gt;        8080:30450/TCP   176m
  service/kubernetes        ClusterIP   10.152.183.1    &lt;none&gt;        443/TCP          10h
  
  NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/hello-world   1/1     1            1           176m
  
  NAME                                     DESIRED   CURRENT   READY   AGE
  replicaset.apps/hello-world-696b6b59bd   1         1         1       176m
  </code></pre>
  
  <p>When I do <code>curl localhost</code> as expected I have <code>connection refused</code>, which looks still fine as it's not exposed to cluster. when I curl the <code>kubernetes-worker/0</code> with public address <code>10.213.117.136</code> on port <code>30450</code>  (which I get from <code>kubectl get all</code>) </p>
  
  <pre><code>$ curl 10.213.117.136:30450
  Hello Kubernetes!
  </code></pre>
  
  <p>Everything works like a charm (which is obvious). When I do </p>
  
  <pre><code>curl -H "Host: testhost.com" 10.213.117.136
  Hello Kubernetes!
  </code></pre>
  
  <p>It works again like charm! That means ingress controller is successfully routing port 80 based on <code>host</code> rule to correct services. At this point I am 100% sure that cluster works as it should. </p>
  
  <p>Now I am trying to access this service over the internet externally. When I load <code>&lt;server_ip&gt;</code> obviously nothing loads as it's living inside own <code>lxd</code> subnet. Therefore I was thinking forward port <code>80</code> from server <code>eth0</code> to this IP. So I added this rule to iptables </p>
  
  <p><code>sudo iptables -t nat -A PREROUTING -p tcp -j DNAT --to-destination 10.213.117.136</code> (For the sake of example let's route everything not only port 80). Now when I open on my computer <code>http://&lt;server_ip&gt;</code> it loads! </p>
  
  <p>So the real question is how to do that on production? Should I setup this forwarding rule in iptables? Is that normal approach or hacky solution and there is something "standard" which I am missing? The thing is to add this rule with static <code>worker</code> node will make the cluster completely static. IP eventually change, I can remove/add units to workers and it will stop working. I was thinking about writing script which will obtain this IP address from <code>juju</code> like this:</p>
  
  <pre><code>$ juju status kubernetes-worker/0 --format=json | jq '.machines["7"]."dns-name"'
  "10.213.117.136"
  </code></pre>
  
  <p>and add it to IP-tables, which is more okay-ish solution than hardcoded IP but still I feel it's a tricky and there must be a better way.</p>
  
  <p>As last idea I get to run <code>HAProxy</code> outside of the cluster, directly on the machine and just do forwarding of traffic to all available workers. This might eventually also work. But still I don't know the answer what is the <code>correct</code> solution and what is usually used in this case. Thank you! </p>