title: Cannot resolve kubernetes.default or services
body: <p>I cannot resolve service from kubernetes.</p>
  
  <pre><code>kubectl get pods -l k8s-app=kube-dns --namespace kube-system
  NAME                       READY   STATUS    RESTARTS   AGE   IP
  coredns-86c58d9df4-gn62b   1/1     Running   0          18d   10.244.0.58
  coredns-86c58d9df4-svmk5   1/1     Running   0          18d   10.244.0.59
  </code></pre>
  
  <p>containers do not resolve any domains, including kubernetes.default</p>
  
  <pre><code>kubectl exec -ti busybox -- sh
  / # nslookup kubernetes.
  defaultServer:    10.96.0.10
  Address 1: 10.96.0.10
  
  nslookup: can't resolve 'kubernetes.default'
  command terminated with exit code 1
  </code></pre>
  
  <p>Logs from dns pods do not show any queries (note coredns is configured to log queries)</p>
  
  <pre><code>kubectl logs --namespace=kube-system coredns-86c58d9df4-gn62b
  .:53
  2019-01-18T21:44:34.271Z [INFO] CoreDNS-1.2.6
  2019-01-18T21:44:34.271Z [INFO] linux/amd64, go1.11.2, 756749c
  CoreDNS-1.2.6
  linux/amd64, go1.11.2, 756749c
   [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
  [INFO] Reloading
   [INFO] plugin/reload: Running configuration MD5 = 2394cf331ea25e9aacc36ddf69fafcdb
  [INFO] Reloading complete
  2019-02-04T22:23:21.266Z [INFO] 127.0.0.1:39695 - 58939 "HINFO IN 4718439545634584094.2038959545847864189. udp 57 false 512" NXDOMAIN qr,rd,ra 133 0.021492508s
  </code></pre>
  
  <p>The kube-node coredns is hosted on is running ubuntu xenial. </p>
  
  <p><a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues" rel="nofollow noreferrer">I noticed there is a known issue on ubuntu hosts</a></p>
  
  <p>I applied custom kubelet conf, setting <code>--resolv-conf=/run/systemd/resolve/resolv.conf</code> </p>
  
  <pre><code>❯ systemctl status kubelet.service
  ● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Wed 2019-02-06 01:05:42 GMT; 5min ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 27867 (kubelet)
      Tasks: 30 (limit: 4915)
     CGroup: /system.slice/kubelet.service
             └─27867 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-drive --resolv-conf=/run/systemd/resolve/resolv.conf
  </code></pre>
  
  <p>However I still cannot resolve any services. </p>