<p dir="auto">It should be possible to script Kube from the cli via simple commands in kubectl that implement the list-and-watch pattern and pass those objects to scripts.  Examples</p>
<p dir="auto">I want to watch all services and and tweak the external load balancer settings when it runs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl get services --watch --all -t &quot;{{ .portalIP }}\n&quot; | xargs ..."><pre class="notranslate"><code class="notranslate">kubectl get services --watch --all -t "{{ .portalIP }}\n" | xargs ...
</code></pre></div>
<p dir="auto">Needs something that makes it easy to exec a process per change, and potentially filter based on event type.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):<br>
No.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):<br>
kube-proxy<br>
issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="188781581" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/36652" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/36652/hovercard" href="https://github.com/kubernetes/kubernetes/issues/36652">#36652</a> is somewhat related but in essence is NOT the same</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:06:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li>
<p dir="auto"><strong>Cloud provider or hardware configuration</strong>:<br>
x86_64 (virtual box virtual machines)</p>
</li>
<li>
<p dir="auto"><strong>OS</strong> (e.g. from /etc/os-release):<br>
NAME="Ubuntu"<br>
VERSION="16.04.1 LTS (Xenial Xerus)"<br>
ID=ubuntu<br>
ID_LIKE=debian<br>
PRETTY_NAME="Ubuntu 16.04.1 LTS"<br>
VERSION_ID="16.04"<br>
HOME_URL="<a href="http://www.ubuntu.com/" rel="nofollow">http://www.ubuntu.com/</a>"<br>
SUPPORT_URL="<a href="http://help.ubuntu.com/" rel="nofollow">http://help.ubuntu.com/</a>"<br>
BUG_REPORT_URL="<a href="http://bugs.launchpad.net/ubuntu/" rel="nofollow">http://bugs.launchpad.net/ubuntu/</a>"<br>
UBUNTU_CODENAME=xenial</p>
</li>
<li>
<p dir="auto"><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):<br>
Linux k8s-node-0 4.4.0-47-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35537532" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/68" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/68/hovercard" href="https://github.com/kubernetes/kubernetes/pull/68">#68</a>-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</p>
</li>
<li>
<p dir="auto"><strong>Install tools</strong>:</p>
</li>
<li>
<p dir="auto"><strong>Others</strong>:<br>
using flannel v0.6.1 as overlay network<br>
kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy all run directly on the host operating system environment (no containerization) and are all managed by systemd. Same for etcd, docker, and flannel.</p>
</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<ul dir="auto">
<li>
<p dir="auto">when i setup all the kube-proxy instances in my minions to use proxy-mode=iptables, then I use curl to try to http/get my webserver (nginx server with 3 replicas) using the service ip address, the connection hangs. However, when I set proxy-mode=userspace it works as expected.</p>
</li>
<li>
<p dir="auto">when I inspected the iptables rules that are setup when proxy-mode=iptables, I noticed some incorrect configurations:  The chains KUBE-MARK-MASQ and KUBE-MARK-DROP in the nat table have the rules:<br>
'-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000<br>
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000'<br>
This is incorrect because the MARK target only works in the mangle table NOT the nat table. See iptables MARK target documentation.</p>
</li>
</ul>
<p dir="auto">Try this, though it is a bit dated. (<a href="https://www.frozentux.net/iptables-tutorial/chunkyhtml/x4389.html" rel="nofollow">https://www.frozentux.net/iptables-tutorial/chunkyhtml/x4389.html</a>)</p>
<ul dir="auto">
<li>The following is my iptables-save dump from one of the nodes:<br>
~$ sudo iptables-save -t nat</li>
</ul>
<h1 dir="auto">Generated by iptables-save v1.6.0 on Tue Nov 15 11:57:54 2016</h1>
<p dir="auto">*nat<br>
:PREROUTING ACCEPT [9:490]<br>
:INPUT ACCEPT [9:490]<br>
:OUTPUT ACCEPT [0:0]<br>
:POSTROUTING ACCEPT [0:0]<br>
:DOCKER - [0:0]<br>
:KUBE-MARK-DROP - [0:0]<br>
:KUBE-MARK-MASQ - [0:0]<br>
:KUBE-NODEPORTS - [0:0]<br>
:KUBE-POSTROUTING - [0:0]<br>
:KUBE-SEP-7WFUHPTNYMSU2JFN - [0:0]<br>
:KUBE-SEP-GXTJGQCWK4PZV3S3 - [0:0]<br>
:KUBE-SEP-HU4VO5AJOOQZNVVQ - [0:0]<br>
:KUBE-SEP-QZMLUAKRU3OISTT5 - [0:0]<br>
:KUBE-SERVICES - [0:0]<br>
:KUBE-SVC-52I4NKETLMIO5ZQJ - [0:0]<br>
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]<br>
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES<br>
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER<br>
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES<br>
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER<br>
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING<br>
-A POSTROUTING -s 172.16.97.0/24 ! -o docker0 -j MASQUERADE<br>
-A DOCKER -i docker0 -j RETURN<br>
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000<br>
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000<br>
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE<br>
-A KUBE-SEP-7WFUHPTNYMSU2JFN -s 172.16.86.2/32 -m comment --comment "default/webservice:" -j KUBE-MARK-MASQ<br>
-A KUBE-SEP-7WFUHPTNYMSU2JFN -p tcp -m comment --comment "default/webservice:" -m tcp -j DNAT --to-destination 172.16.86.2:80<br>
-A KUBE-SEP-GXTJGQCWK4PZV3S3 -s 172.16.97.2/32 -m comment --comment "default/webservice:" -j KUBE-MARK-MASQ<br>
-A KUBE-SEP-GXTJGQCWK4PZV3S3 -p tcp -m comment --comment "default/webservice:" -m tcp -j DNAT --to-destination 172.16.97.2:80<br>
-A KUBE-SEP-HU4VO5AJOOQZNVVQ -s 10.200.3.13/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ<br>
-A KUBE-SEP-HU4VO5AJOOQZNVVQ -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-HU4VO5AJOOQZNVVQ --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.200.3.13:6443<br>
-A KUBE-SEP-QZMLUAKRU3OISTT5 -s 172.16.49.2/32 -m comment --comment "default/webservice:" -j KUBE-MARK-MASQ<br>
-A KUBE-SEP-QZMLUAKRU3OISTT5 -p tcp -m comment --comment "default/webservice:" -m tcp -j DNAT --to-destination 172.16.49.2:80<br>
-A KUBE-SERVICES -d 192.168.33.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y<br>
-A KUBE-SERVICES -d 192.168.33.28/32 -p tcp -m comment --comment "default/webservice: cluster IP" -m tcp --dport 80 -j KUBE-SVC-52I4NKETLMIO5ZQJ<br>
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS<br>
-A KUBE-SVC-52I4NKETLMIO5ZQJ -m comment --comment "default/webservice:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-QZMLUAKRU3OISTT5<br>
-A KUBE-SVC-52I4NKETLMIO5ZQJ -m comment --comment "default/webservice:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7WFUHPTNYMSU2JFN<br>
-A KUBE-SVC-52I4NKETLMIO5ZQJ -m comment --comment "default/webservice:" -j KUBE-SEP-GXTJGQCWK4PZV3S3<br>
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-HU4VO5AJOOQZNVVQ --mask 255.255.255.255 --rsource -j KUBE-SEP-HU4VO5AJOOQZNVVQ<br>
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-HU4VO5AJOOQZNVVQ<br>
COMMIT</p>
<h1 dir="auto">Completed on Tue Nov 15 11:57:54 2016</h1>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
The service ip address should work the same regardless of proxy-mode=iptables or proxy-mode=userspace.</p>
<p dir="auto">So if I do 'curl <a href="http://mywebserviceip" rel="nofollow">http://mywebserviceip</a>' it should work despite the kind of plumbing used by kube-proxy</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ul dir="auto">
<li>install ubuntu 16.04 server in 3 VMs</li>
<li>setup etcd cluster, flanneld overlay network, and docker container runtime</li>
<li>setup kube-apiserver, kube-controller-manager, kube-scheduler in the master node.</li>
<li>setup kubelet and kube-proxy in all the nodes. Setup kube-proxy to use --proxy-mode=iptables</li>
<li>deploy a replication controller yaml config file using kubectl, set it up to deploy an nginx container</li>
<li>deploy a service yaml config file using kubectl, set it up to use the nginx pods as endpoints</li>
<li>from the command line of anyone of the nodes, type 'curl http://&lt;service-ip-address'. Notice that it hangs.</li>
<li>change --proxy-mode=userspace for one of the nodes and restart kube-proxy there. Try invoke curl again multiple times; notice that sometimes curl successfully returns you a webpage and sometimes hangs.</li>
<li>change --proxy-mode=userspace for all the nodes and restart kube-proxy in all. Try invoke curl again multiple times; notice that curl always succeeds.</li>
</ul>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
I made the following manual changes to my iptables configuration (adding KUBE-MARK-MASQ to the mangle table) and i can now successfully query my service when kube-proxy=iptables</p>
<h1 dir="auto">Generated by iptables-save v1.6.0 on Tue Nov 15 11:23:19 2016</h1>
<p dir="auto">*mangle<br>
:PREROUTING ACCEPT [21850:2940486]<br>
:INPUT ACCEPT [21850:2940486]<br>
:FORWARD ACCEPT [0:0]<br>
:OUTPUT ACCEPT [21274:1877657]<br>
:POSTROUTING ACCEPT [21274:1877657]<br>
:KUBE-MARK-MASQ - [0:0]<br>
:KUBE-SERVICES - [0:0]<br>
:KUBE-SVC-52I4NKETLMIO5ZQJ - [0:0]<br>
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]<br>
-A PREROUTING -j KUBE-SERVICES<br>
-A OUTPUT -j KUBE-SERVICES<br>
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000<br>
-A KUBE-SERVICES -d 192.168.33.1/32 -p tcp -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y<br>
-A KUBE-SERVICES -d 192.168.33.28/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-52I4NKETLMIO5ZQJ<br>
-A KUBE-SVC-52I4NKETLMIO5ZQJ -j KUBE-MARK-MASQ<br>
-A KUBE-SVC-NPX46M4PTMTKRN6Y -j KUBE-MARK-MASQ<br>
COMMIT</p>
<h1 dir="auto">Completed on Tue Nov 15 11:23:19 2016</h1>