pod to pod can not be reached through service clusterIP running in the same minion
The response from the called service is not DNAT'ed back to set src to the service clusterIP. So the response packet is going directly via cni0 bridge and not getting into iptables, preserving pod's IP.

to fix this i added the following option to kube-proxy and issue was gone.

--masquerade-all=true