<p dir="auto">So running the simplest example of nginx on hyperkube everything goes well. I can reach nginx from other pods in the cluster and from the host. But when I try to curl the service ip from within the pod itself then I get no response.</p>
<p dir="auto">The iptables rules below illustrate the problem</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="carsonanderson ~ # iptables-save |egrep &quot;10.0.0.25|KUBE-SVC-4N57TFCL4MD7ZTDA|KUBE-SEP-YAIKK2SEJQZ5ZUOS|MARK&quot;
:KUBE-MARK-MASQ - [0:0]
:KUBE-SEP-YAIKK2SEJQZ5ZUOS - [0:0]
:KUBE-SVC-4N57TFCL4MD7ZTDA - [0:0]
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-SEP-YAIKK2SEJQZ5ZUOS -s 172.17.0.2/32 -m comment --comment &quot;default/nginx:&quot; -j KUBE-MARK-MASQ
-A KUBE-SEP-YAIKK2SEJQZ5ZUOS -p tcp -m comment --comment &quot;default/nginx:&quot; -m tcp -j DNAT --to-destination 172.17.0.2:80
-A KUBE-SERVICES -d 10.0.0.25/32 -p tcp -m comment --comment &quot;default/nginx: cluster IP&quot; -m tcp --dport 80 -j KUBE-SVC-4N57TFCL4MD7ZTDA
-A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment &quot;default/nginx:&quot; -j KUBE-SEP-YAIKK2SEJQZ5ZUOS"><pre class="notranslate">carsonanderson <span class="pl-k">~</span> <span class="pl-c"><span class="pl-c">#</span> iptables-save |egrep "10.0.0.25|KUBE-SVC-4N57TFCL4MD7ZTDA|KUBE-SEP-YAIKK2SEJQZ5ZUOS|MARK"</span>
:KUBE-MARK-MASQ - [0:0]
:KUBE-SEP-YAIKK2SEJQZ5ZUOS - [0:0]
:KUBE-SVC-4N57TFCL4MD7ZTDA - [0:0]
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-SEP-YAIKK2SEJQZ5ZUOS -s 172.17.0.2/32 -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/nginx:<span class="pl-pds">"</span></span> -j KUBE-MARK-MASQ
-A KUBE-SEP-YAIKK2SEJQZ5ZUOS -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/nginx:<span class="pl-pds">"</span></span> -m tcp -j DNAT --to-destination 172.17.0.2:80
-A KUBE-SERVICES -d 10.0.0.25/32 -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/nginx: cluster IP<span class="pl-pds">"</span></span> -m tcp --dport 80 -j KUBE-SVC-4N57TFCL4MD7ZTDA
-A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/nginx:<span class="pl-pds">"</span></span> -j KUBE-SEP-YAIKK2SEJQZ5ZUOS</pre></div>
<p dir="auto">I think it's because we are marking it for masquerade and then sending the traffic out of the network when the source is one of the pods serving a service.</p>
<p dir="auto">The  <code class="notranslate">-A KUBE-SEP-YAIKK2SEJQZ5ZUOS -s 172.17.0.2/32 -m comment --comment "default/nginx:" -j KUBE-MARK-MASQ</code> line should probably also take into account if the destination is the same as the source and route back into itself directly or jump to the KUBE-SVC-XXX rule instead of hitting the masquerade table and going out to the world.</p>
<p dir="auto">I have a Kubernetes 1.2.2 setup with kube-proxy in iptables mode.</p>
<p dir="auto">There is a Pod A with Pod-IP 10.2.94.5.<br>
There is a Service A with IP 10.3.249.19, which redirects all traffic to this pod.<br>
There is Pod B with Pod-IP 10.2.94.2.<br>
Pod A and B are on the same machine.</p>
<p dir="auto">When I access service A from pod B, everything works fine. However, when I try to access service A from pod A, iptables seems to drop the request.</p>
<p dir="auto">In the examples below, I'm trying to access a squid as a proof-of-concept. The practical use case would be to run a GlusterFS (which needs access to its own outside IP when creating volumes).<br>
I did a bit iptables digging and it seems like there is a masquerading rule KUBE-MARK-MASQ, which catches the request and prevents forwarding it.</p>
<p dir="auto"><strong>Trace for accessing service A from pod B (successful)</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[248369.783562] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783580] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783594] TRACE: nat:PREROUTING:rule:1 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783645] TRACE: nat:KUBE-SERVICES:rule:16 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783670] TRACE: nat:KUBE-SVC-NQAN5OLURYDUMCKE:rule:1 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783676] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783694] TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783700] TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783705] TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783709] TRACE: filter:DOCKER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783715] TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783720] TRACE: nat:POSTROUTING:rule:1 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783725] TRACE: nat:KUBE-POSTROUTING:return:2 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783729] TRACE: nat:POSTROUTING:rule:2 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783733] TRACE: nat:FLANNEL:rule:1 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783815] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783824] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783831] TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783836] TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783841] TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783846] TRACE: filter:DOCKER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783850] TRACE: filter:FORWARD:rule:3 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)"><pre class="notranslate"><code class="notranslate">[248369.783562] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783580] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783594] TRACE: nat:PREROUTING:rule:1 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783645] TRACE: nat:KUBE-SERVICES:rule:16 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783670] TRACE: nat:KUBE-SVC-NQAN5OLURYDUMCKE:rule:1 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783676] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783694] TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783700] TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783705] TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783709] TRACE: filter:DOCKER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783715] TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783720] TRACE: nat:POSTROUTING:rule:1 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783725] TRACE: nat:KUBE-POSTROUTING:return:2 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783729] TRACE: nat:POSTROUTING:rule:2 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783733] TRACE: nat:FLANNEL:rule:1 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307) 
[248369.783815] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783824] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783831] TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783836] TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783841] TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783846] TRACE: filter:DOCKER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5) 
[248369.783850] TRACE: filter:FORWARD:rule:3 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)
</code></pre></div>
<p dir="auto"><strong>Trace for accessing service A from pod A (unsuccessful):</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[246579.045435] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045451] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045460] TRACE: nat:PREROUTING:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045504] TRACE: nat:KUBE-SERVICES:rule:18 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045528] TRACE: nat:KUBE-SVC-NQAN5OLURYDUMCKE:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045534] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045548] TRACE: nat:KUBE-MARK-MASQ:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045556] TRACE: nat:KUBE-MARK-MASQ:return:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) MARK=0x4000 
[246579.045563] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) MARK=0x4000 
... (repeats this chunk multiple times as it tries connecting again)"><pre class="notranslate"><code class="notranslate">[246579.045435] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045451] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045460] TRACE: nat:PREROUTING:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045504] TRACE: nat:KUBE-SERVICES:rule:18 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045528] TRACE: nat:KUBE-SVC-NQAN5OLURYDUMCKE:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045534] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045548] TRACE: nat:KUBE-MARK-MASQ:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) 
[246579.045556] TRACE: nat:KUBE-MARK-MASQ:return:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) MARK=0x4000 
[246579.045563] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) MARK=0x4000 
... (repeats this chunk multiple times as it tries connecting again)
</code></pre></div>
<p dir="auto">The interesting parts from <code class="notranslate">iptables -t nat -L</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
FLANNEL    all  --  10.2.0.0/16          anywhere            
MASQUERADE  all  -- !10.2.0.0/16          10.2.0.0/16         

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain FLANNEL (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             10.2.0.0/16         
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4 

Chain KUBE-MARK-MASQ (50 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination         

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere             /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-AIYB5BTNB4W7CP2T (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.2.94.5            anywhere             /* default/squid:http */
DNAT       tcp  --  anywhere             anywhere             /* default/squid:http */ tcp to:10.2.94.5:3128

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
[...]
KUBE-SVC-NQAN5OLURYDUMCKE  tcp  --  anywhere             10.3.249.19          /* default/squid:http cluster IP */ tcp dpt:http-alt
[...]
KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-NQAN5OLURYDUMCKE (1 references)
target     prot opt source               destination         
KUBE-SEP-AIYB5BTNB4W7CP2T  all  --  anywhere             anywhere             /* default/squid:http */"><pre class="notranslate"><code class="notranslate">Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
FLANNEL    all  --  10.2.0.0/16          anywhere            
MASQUERADE  all  -- !10.2.0.0/16          10.2.0.0/16         

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain FLANNEL (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             10.2.0.0/16         
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4 

Chain KUBE-MARK-MASQ (50 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination         

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere             /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-AIYB5BTNB4W7CP2T (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.2.94.5            anywhere             /* default/squid:http */
DNAT       tcp  --  anywhere             anywhere             /* default/squid:http */ tcp to:10.2.94.5:3128

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
[...]
KUBE-SVC-NQAN5OLURYDUMCKE  tcp  --  anywhere             10.3.249.19          /* default/squid:http cluster IP */ tcp dpt:http-alt
[...]
KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-NQAN5OLURYDUMCKE (1 references)
target     prot opt source               destination         
KUBE-SEP-AIYB5BTNB4W7CP2T  all  --  anywhere             anywhere             /* default/squid:http */
</code></pre></div>