title: k8s: forwarding from public VIP to clusterIP with iptables
body: <p>I'm trying to understand in depth how forwarding from publicly exposed load-balancer's layer-2 VIPs to services' cluster-IPs works. I've read a high-level overview how <a href="https://metallb.universe.tf/concepts/layer2/" rel="nofollow noreferrer">MetalLB does it</a> and I've tried to replicate it manually by setting keepalived/ucarp VIP and iptables rules. I must be missing something however as it doesn't work ;-]</p>
  
  <p><strong>Steps I took:</strong></p>
  
  <ol>
  <li><p>created a cluster with <code>kubeadm</code> consisting of a master + 3 nodes running k8s-1.17.2 + calico-3.12 on libvirt/KVM VMs on a single computer. all VMs are in <code>192.168.122.0/24</code> virtual network. </p></li>
  <li><p>created a simple 2 pod deployment and exposed it as a <code>NodePort</code> service with <code>externalTrafficPolicy</code> set to <code>cluster</code>:<br>
  <code>$ kubectl get svc dump-request
  NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
  dump-request   NodePort    10.100.234.120   &lt;none&gt;        80:32292/TCP   65s</code><br>
  I've verified that I can reach it from the host machine on every node's IP at 32292 port.</p></li>
  <li><p>created a VIP with <code>ucarp</code> on all 3 nodes:<br>
  <code>ucarp -i ens3 -s 192.168.122.21 -k 21 -a 192.168.122.71 -v 71 -x 71 -p dump -z -n -u /usr/local/sbin/vip-up.sh -d /usr/local/sbin/vip-down.sh</code> (example from knode1)<br>
  I've verified that I can ping the <code>192.168.122.71</code> VIP. I even could ssh through it to the VM that was currently holding the VIP.<br>
  Now if kube-proxy was in <code>iptables</code> mode, I could also reach the service on its node-port through the VIP at <code>http://192.168.122.71:32292</code>. However, to my surprise, in <code>ipvs</code> mode this always resulted in connection timing out.</p></li>
  <li><p>added an iptables rule on every node for packets incoming to <code>192.168.122.71</code> to be forwarded to to service's cluster-IP <code>10.100.234.120</code>:<br>
  <code>iptables -t nat -A PREROUTING -d 192.168.122.71 -j DNAT --to-destination 10.100.234.120</code><br>
  (later I've also tried to narrow the rule only to the relevant port, but it didn't change the results in any way:<br>
  <code>iptables -t nat -A PREROUTING -d 192.168.122.71 -p tcp --dport 80 -j DNAT --to-destination 10.100.234.120:80</code>)</p></li>
  </ol>
  
  <p><strong>Results:</strong></p>
  
  <p>in <code>iptables</code> mode all requests to <code>http://192.168.122.71:80/</code> resulted in connection timing out.</p>
  
  <p>in <code>ipvs</code> mode it worked partially:<br>
  if the <code>192.168.122.71</code> VIP was being held by a node that had a pod on it, then about 50% requests were succeeding and they were <em>always</em> served by the local pod. the app was also getting the real remote IP of the host machine (<code>192.168.122.1</code>). the other 50% (being sent to the pod on anther node presumably) were timing out.<br>
  if the VIP was being held by a node without pods then all requests were timing out.</p>
  
  <p>I've also checked if it affects the results in anyway to keep the rule on all nodes at all times vs. to keep it only on the node holding the VIP and deleting it at the release of the VIP: results were the same in both cases.</p>
  
  <p>Does anyone know why it doesn't work and how to fix it? I will appreciate help with this :)</p>