<p dir="auto"><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container_bridge.go#L122-L143">https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container_bridge.go#L122-L143</a><br>
container_bridge.go assumes that the virtual IP of services &amp; pods will be in the <code class="notranslate">10.</code> space.<br>
I propose there is no reason to make this assumption.</p>
<p dir="auto">As outlined in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="112308768" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/15932" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/15932/hovercard" href="https://github.com/kubernetes/kubernetes/issues/15932">#15932</a>, cluster admins may need to deploy to hosts in which <code class="notranslate">10.</code> is reserved for the nodes. In such a case, Kubelets must support an alternative range.</p>
<p dir="auto">Today kubelet sets up an iptables MASQUERADE rule for any traffic destined for anything except 10.0.0.0/8.  This is close, but not even correct on GCE, and certainly not right elsewhere.</p>
<p dir="auto">First GCE.  We probably want something like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="iptables -t nat -N KUBE-IPMASQ
iptables -t nat -A KUBE-IPMASQ -d 10.0.0.0/8 -j RETURN
iptables -t nat -A KUBE-IPMASQ -d 172.16.0.0/12 -j RETURN
iptables -t nat -A KUBE-IPMASQ -d 192.168.0.0/16 -j RETURN
iptables -t nat -A KUBE-IPMASQ -j MASQUERADE
iptables -t nat -I POSTROUTING -j KUBE-IPMASQ"><pre class="notranslate"><code class="notranslate">iptables -t nat -N KUBE-IPMASQ
iptables -t nat -A KUBE-IPMASQ -d 10.0.0.0/8 -j RETURN
iptables -t nat -A KUBE-IPMASQ -d 172.16.0.0/12 -j RETURN
iptables -t nat -A KUBE-IPMASQ -d 192.168.0.0/16 -j RETURN
iptables -t nat -A KUBE-IPMASQ -j MASQUERADE
iptables -t nat -I POSTROUTING -j KUBE-IPMASQ
</code></pre></div>
<p dir="auto">This catches all traffic to RFC1918 ranges and masquerades it.  We can probably optimize with CONNMARK or something so we only consider packets from containers.  This is probably still imperfect, but better, for lack of project-wide NAT for egress.</p>
<p dir="auto">For other environments, we really have no idea what the correct policy for this is.  It is closer to "your nodes must handle this" than "we can handle this for you".  It's debatable whether we should even try.</p>
<p dir="auto">Either:<br>
a) We teach kubelet a lot more and let people pass flags to nearly=arbitrarily configure this<br>
b) We tell people to configure this as part of their node setup</p>
<p dir="auto">This popped up when I realized GKE allows users to set up 172.* clusters - any traffic between containers in one of these will get masqueraded - not correct behavior!!  This is not a huge deal right now because kube-proxy has the same effect when traversing services.  As we fix kube-proxy in the wake of 1.0, masquerade will be a bigger deal, especially for micro-segmenting.</p>
<p dir="auto">Additional considerations: VPNs have bizarre and very custom needs.  Every such thing has an as yet unmeasured perf implication.  This also pops up in our GCE firewall thing that <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArtfulCoder/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArtfulCoder">@ArtfulCoder</a> is working on.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dchen1107/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dchen1107">@dchen1107</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/alex-mohr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/alex-mohr">@alex-mohr</a></p>