pod to pod can not be reached through service clusterIP running in the same minion few days ago, i am facing a strange issue which is "a pod can not reach another pod running in the same minion/host through servide ClusterIP whereas they can reach each other skipping going through service clusterIP"

i am running kubernetes 1.9.6 on top of openstack (ubuntu 16.04 hosts) with:


cni 0.7.0
cri-containerd 1.0.0-beta
flannel 0.9.1
etcd 3.0.13


the other scenarios were working fine. even between pods running in different hosts, they can reach each other through service clusterIP.

what is going on here !!! any support will be more than welcome
The response from the called service is not DNAT'ed back to set src to the service clusterIP. So the response packet is going directly via cni0 bridge and not getting into iptables, preserving pod's IP.

to fix this i added the following option to kube-proxy and issue was gone.

--masquerade-all=true