title: how to configure kubespray DNS for bare-metal
body: <p>I am relatively new to kubernetes and have a project for my University class, to build a kubernetes Cluster on bare metal.</p>
  <p>For this I have set up a PoC Environment, out of 6 Machines (of which 3 are KVM Machines on one Node) all the administration is done by MAAS, meaning DHCP, and DNS is administered by that one Machine. I have a DNS Zone delegated to the MAAS DNS server k8s.example.com where all the machines are inside. The whole network is in its own VLan 10.0.10.0/24, with the metallb IPRange reserved from DHCP.
  This is a picture to illustrate the simple cluster:</p>
  <p><a href="https://i.stack.imgur.com/jPEg6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jPEg6.png" alt="k8s overview" /></a></p>
  <p>software wise, all hosts are using ubuntu 20.04 and I use kubespray to deploy everything, meaning kubernetes, metallb and nginx-ingress-controller. My corresponding values for kubespray are:</p>
  <pre><code>dashboard_enabled: false
  ingress_nginx_enabled: true
  ingress_nginx_host_network: true
  
  kube_proxy_strict_arp: true
  
  metallb_enabled: true
  metallb_speaker_enabled: true
  metallb_ip_range:
    - &quot;10.0.10.100-10.0.10.120&quot;
  
  kubeconfig_localhost: true
  </code></pre>
  <p>My Problem is, that I am unable getting DNS out of the cluster to the Internet to work.
  I had a wildcard A Record set for *.k8s.example.com to the nginx-Ingress external ip, which worked fine for every pod to be accessible from outside.
  The Problem was, every container inside the Cluster could not reach the internet anymore. Every request was routed via the ingress. Meaning if I tried to reach <a href="http://www.google.net" rel="nofollow noreferrer">www.google.net</a> it would try to reach <a href="http://www.google.net.k8s.example.com" rel="nofollow noreferrer">www.google.net.k8s.example.com</a> which makes kind of sense. Only every .com domain could be reached without problems (example <a href="http://www.google.com" rel="nofollow noreferrer">www.google.com</a>) after removing the Wildcard A record it worked fine. All pods inside of the cluster have no problem reaching each other.</p>
  <p>There are several configuration possibility I see, where it makes sense to tweak around, yet after 2 weeks I really would prefer a solution that is based on best practice and done right.</p>
  <p>I would really love to be able to work with a wildcard A record, but I fear that might not be possible.</p>
  <p>I hope I supplied every Information needed to give you enough overview to understand my Problem.</p>
  <p>EDIT:
  I used the standard kubespray DNS config as i was told it would suffice:</p>
  <pre><code>DNS configuration.
  # Kubernetes cluster name, also will be used as DNS domain
  cluster_name: cluster.local
  # Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
  ndots: 2
  # Can be coredns, coredns_dual, manual or none
  dns_mode: coredns
  # Set manual server if using a custom cluster DNS server
  # manual_dns_server: 10.x.x.x
  # Enable nodelocal dns cache
  enable_nodelocaldns: true
  nodelocaldns_ip: 169.254.25.10
  nodelocaldns_health_port: 9254
  # nodelocaldns_external_zones:
  # - zones:
  #   - example.com
  #   - example.io:1053
  #   nameservers:
  #   - 1.1.1.1
  #   - 2.2.2.2
  #   cache: 5
  # - zones:
  #   - https://mycompany.local:4453
  #   nameservers:
  #   - 192.168.0.53
  #   cache: 0
  # Enable k8s_external plugin for CoreDNS
  enable_coredns_k8s_external: false
  coredns_k8s_external_zone: k8s_external.local
  # Enable endpoint_pod_names option for kubernetes plugin
  enable_coredns_k8s_endpoint_pod_names: false
  
  # Can be docker_dns, host_resolvconf or none
  resolvconf_mode: docker_dns
  # Deploy netchecker app to verify DNS resolve as an HTTP service
  deploy_netchecker: false
  # Ip address of the kubernetes skydns service
  skydns_server: &quot;{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}&quot;
  skydns_server_secondary: &quot;{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}&quot;
  dns_domain: &quot;{{ cluster_name }}&quot;
  </code></pre>
  <p>What I noticed is, that the etc resolv.conf of pods looks like this:</p>
  <pre><code>/ $ cat /etc/resolv.conf 
  nameserver 169.254.25.10
  search flux-system.svc.cluster.local svc.cluster.local cluster.local k8s.example.com maas
  options ndots:5
  </code></pre>
  <p>for example on the node, which is managed by MAAS, it is:</p>
  <pre><code># This file is managed by man:systemd-resolved(8). Do not edit.
  #
  # This is a dynamic resolv.conf file for connecting local clients to the
  # internal DNS stub resolver of systemd-resolved. This file lists all
  # configured search domains.
  #
  # Run &quot;resolvectl status&quot; to see details about the uplink DNS servers
  # currently in use.
  #
  # Third party programs must not access this file directly, but only through the
  # symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
  # replace this symlink by a static file or a different symlink.
  #
  # See man:systemd-resolved.service(8) for details about the supported modes of
  # operation for /etc/resolv.conf.
  
  nameserver 127.0.0.53
  options edns0 trust-ad
  search k8s.example.com maas
  </code></pre>