Does Kubernetes internal networking keep traffic to co-located Pods where possible? In Kubernetes, if I run a Service Foo on every Node, and also Service Bar on every Node, i.e. with DaemonSets. If a Pod from Service Foo needs to make a request to Service Bar, does Kubernetes networking magic optimise things so communication will be done within the same Node, i.e. without a external network call? Or will a random Pod Bar be picked meaning usually Bar will be on a different Node, so there will be a network hop?

Context to question: I'm thinking to run CoreDNS as a Daemonset, because with the cache plugin, I'm hoping most DNS queries will not need to use the network as there will always be a CoreDNS Pod co-located within the same Node and most requests will be cached. Currently we only run 3 CoreDNS pods and we have 12 Nodes and we see some DNS failures at times of network traffic surges - particularly during a deployment when lot of logging is done, which have to be shipped across the network to ElasticSearch.



In case specifics matter:


The CoreDNS service runs as clusterIP: 172.20.0.10
The services that use CoreDNS run as type: NodePort
The Kubernetes cluster is AWS EKS, v1.14.9. Networking is done with the AWS VPC CNI.
This looks to be answered on offcail docs here

NodeLocal DNSCache improves Cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet.
With this architecture, Pods will reach out to the dns caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking.
Having a local cache will help improve the latency.


Image Reference : https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/#architecture-diagram