title: microk8s + ingress: ingressed service always resolves to 127.0.0.1 and not pod ip
body: <p>I am learning about microk8s and how ingress works.</p>
  <p>I have a single node microk8s (v1.18.4) with the following add-ons: DNS, ingress, RBAC, storage</p>
  <p>I am trying to get it working with the microbot example. I've read (and reread) through the tutorial but, the address, once the ingress manifest is applied results with the microbot service, routed to <code>127.0.0.1</code> (and not the internal pod IP).</p>
  <p>I am attempting to access the app: <code>http://192.168.91.166/microbot</code> via web external to the vm that it's running inside (and have tried curl while logged into the vm) it would result in an error page being returned. <code>192.168.91.166</code> is the vm's ip.</p>
  <pre class="lang-sh prettyprint-override"><code># kubectl get all -A
  NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
  development   pod/microbot-7dcc49575f-fqttp                 1/1     Running   1          22h
  ingress       pod/nginx-ingress-microk8s-controller-phjgl   1/1     Running   1          21h
  kube-system   pod/coredns-588fd544bf-lb8mx                  1/1     Running   3          37h
  kube-system   pod/hostpath-provisioner-75fdc8fccd-4klwh     1/1     Running   5          37h
  
  NAMESPACE     NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
  default       service/kubernetes   ClusterIP   10.152.183.1    &lt;none&gt;        443/TCP                  37h
  development   service/microbot     ClusterIP   10.152.183.19   &lt;none&gt;        80/TCP                   22h
  kube-system   service/kube-dns     ClusterIP   10.152.183.10   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   37h
  
  NAMESPACE   NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
  ingress     daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           &lt;none&gt;          21h
  
  NAMESPACE     NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
  development   deployment.apps/microbot               1/1     1            1           22h
  kube-system   deployment.apps/coredns                1/1     1            1           37h
  kube-system   deployment.apps/hostpath-provisioner   1/1     1            1           37h
  
  NAMESPACE     NAME                                              DESIRED   CURRENT   READY   AGE
  development   replicaset.apps/microbot-7dcc49575f               1         1         1       22h
  kube-system   replicaset.apps/coredns-588fd544bf                1         1         1       37h
  kube-system   replicaset.apps/hostpath-provisioner-75fdc8fccd   1         1         1       37h
  
  </code></pre>
  <p><strong>microbot svc description</strong></p>
  <pre class="lang-sh prettyprint-override"><code># kubectl describe svc microbot -n development
  Name:              microbot
  Namespace:         development
  Labels:            &lt;none&gt;
  Annotations:       Selector:  name=microbot
  Type:              ClusterIP
  IP:                10.152.183.19
  Port:              &lt;unset&gt;  80/TCP
  TargetPort:        8080/TCP
  Endpoints:         10.1.62.21:8080
  Session Affinity:  None
  Events:            &lt;none&gt;
  
  </code></pre>
  <p><strong>ingress description</strong></p>
  <pre class="lang-sh prettyprint-override"><code># kubectl describe ingress microbot -n development
  Name:             microbot
  Namespace:        development
  Address:          127.0.0.1
  Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
  Rules:
    Host        Path  Backends
    ----        ----  --------
    *           
                /microbot   microbot:80 (10.1.62.21:8080)
  Annotations:  kubernetes.io/ingress.class: nginx
                nginx.ingress.kubernetes.io/rewrite-target: /
  Events:
    Type    Reason  Age   From                      Message
    ----    ------  ----  ----                      -------
    Normal  UPDATE  14s   nginx-ingress-controller  Ingress development/microbot
  </code></pre>
  <p>ingress pod log</p>
  <pre class="lang-sh prettyprint-override"><code># kubectl logs -n ingress nginx-ingress-microk8s-controller-phjgl
  -------------------------------------------------------------------------------
  NGINX Ingress controller
    Release:       0.33.0
    Build:         git-589187c35
    Repository:    https://github.com/kubernetes/ingress-nginx
    nginx version: nginx/1.19.0
  
  -------------------------------------------------------------------------------
  
  W0715 13:52:22.191579       6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
  W0715 13:52:22.192057       6 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  I0715 13:52:22.194678       6 main.go:218] Creating API client for https://10.152.183.1:443
  W0715 13:52:53.301344       6 main.go:259] Initial connection to the Kubernetes API server was retried 1 times.
  I0715 13:52:53.301361       6 main.go:262] Running in Kubernetes cluster version v1.18+ (v1.18.4-1+6f17be3f1fd54a) - git (clean) commit 6f17be3f1fd54a88681869d1cf8bedd5a2174504 - platform linux/amd64
  I0715 13:52:53.538979       6 main.go:103] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
  I0715 13:52:53.539788       6 main.go:111] Enabling new Ingress features available since Kubernetes v1.18
  E0715 13:52:53.542526       6 main.go:120] Unexpected error searching IngressClass: ingressclasses.networking.k8s.io &quot;nginx&quot; is forbidden: User &quot;system:serviceaccount:ingress:nginx-ingress-microk8s-serviceaccount&quot; cannot get resource &quot;ingressclasses&quot; in API group &quot;networking.k8s.io&quot; at the cluster scope
  W0715 13:52:53.542574       6 main.go:123] No IngressClass resource with name nginx found. Only annotation will be used.
  I0715 13:52:53.590982       6 nginx.go:263] Starting NGINX Ingress controller
  I0715 13:52:53.604026       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;ingress&quot;, Name:&quot;nginx-ingress-tcp-microk8s-conf&quot;, UID:&quot;5f1b23d0-297c-4b2f-ac43-f3dc7256f684&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;14859&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-ingress-tcp-microk8s-conf
  I0715 13:52:53.604099       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;ingress&quot;, Name:&quot;nginx-ingress-udp-microk8s-conf&quot;, UID:&quot;abbfd3c7-c38c-43df-ae0c-7c26abcdb9cc&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;14860&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-ingress-udp-microk8s-conf
  I0715 13:52:53.604190       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;ingress&quot;, Name:&quot;nginx-load-balancer-microk8s-conf&quot;, UID:&quot;bc525770-be0a-4303-b26d-7e1c1d424d39&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;14858&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-load-balancer-microk8s-conf
  I0715 13:52:54.694319       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;development&quot;, Name:&quot;microbot&quot;, UID:&quot;58d26f3e-d19a-4c53-8a47-1223b3ccc300&quot;, APIVersion:&quot;networking.k8s.io/v1beta1&quot;, ResourceVersion:&quot;22517&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' Ingress development/microbot
  I0715 13:52:54.791730       6 nginx.go:307] Starting NGINX process
  I0715 13:52:54.791929       6 leaderelection.go:242] attempting to acquire leader lease  ingress/ingress-controller-leader-nginx...
  I0715 13:52:54.793717       6 controller.go:139] Configuration changes detected, backend reload required.
  I0715 13:52:54.805276       6 leaderelection.go:252] successfully acquired lease ingress/ingress-controller-leader-nginx
  I0715 13:52:54.805481       6 status.go:86] new leader elected: nginx-ingress-microk8s-controller-phjgl
  I0715 13:52:55.235056       6 controller.go:155] Backend successfully reloaded.
  I0715 13:52:55.235148       6 controller.go:164] Initial sync, sleeping for 1 second.
  2020/07/15 15:49:54 [error] 38#38: *57190 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  2020/07/15 15:49:54 [error] 38#38: *57190 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  2020/07/15 15:49:54 [error] 38#38: *57190 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  127.0.0.1 - - [15/Jul/2020:15:49:54 +0000] &quot;GET /microbot HTTP/1.1&quot; 502 157 &quot;-&quot; &quot;curl/7.68.0&quot; 81 0.000 [development-microbot-80] [] 10.1.62.21:8080, 10.1.62.21:8080, 10.1.62.21:8080 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 f4a04290c54d7562d01c2b2ef7b68d6b
  2020/07/15 15:50:03 [error] 36#36: *57305 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  2020/07/15 15:50:03 [error] 36#36: *57305 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  2020/07/15 15:50:03 [error] 36#36: *57305 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  127.0.0.1 - - [15/Jul/2020:15:50:03 +0000] &quot;GET /microbot HTTP/1.1&quot; 502 157 &quot;-&quot; &quot;curl/7.68.0&quot; 81 0.000 [development-microbot-80] [] 10.1.62.21:8080, 10.1.62.21:8080, 10.1.62.21:8080 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 58813a2861ae850b96d10ab9482aaaae
  2020/07/15 15:54:44 [error] 37#37: *60738 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  2020/07/15 15:54:44 [error] 37#37: *60738 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  2020/07/15 15:54:44 [error] 37#37: *60738 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
  127.0.0.1 - - [15/Jul/2020:15:54:44 +0000] &quot;GET /microbot HTTP/1.1&quot; 502 157 &quot;-&quot; &quot;curl/7.68.0&quot; 81 0.000 [development-microbot-80] [] 10.1.62.21:8080, 10.1.62.21:8080, 10.1.62.21:8080 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 b2bc837c1eb56d5e598f8160e4f2e3cb
  I0715 16:36:08.010736       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;development&quot;, Name:&quot;microbot&quot;, UID:&quot;58d26f3e-d19a-4c53-8a47-1223b3ccc300&quot;, APIVersion:&quot;networking.k8s.io/v1beta1&quot;, ResourceVersion:&quot;94444&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'UPDATE' Ingress development/microbot
  I0715 16:36:08.011329       6 controller.go:139] Configuration changes detected, backend reload required.
  I0715 16:36:08.116361       6 controller.go:155] Backend successfully reloaded.
  I0715 16:41:19.262503       6 store.go:383] removing ingress microbot based on annotation kubernetes.io/ingress.class
  I0715 16:41:19.262648       6 controller.go:139] Configuration changes detected, backend reload required.
  I0715 16:41:19.262688       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;development&quot;, Name:&quot;microbot&quot;, UID:&quot;58d26f3e-d19a-4c53-8a47-1223b3ccc300&quot;, APIVersion:&quot;networking.k8s.io/v1beta1&quot;, ResourceVersion:&quot;94444&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'DELETE' Ingress development/microbot
  I0715 16:41:19.343023       6 controller.go:155] Backend successfully reloaded.
  I0715 16:42:53.063307       6 store.go:324] ignoring delete for ingress microbot based on annotation kubernetes.io/ingress.class
  I0715 16:43:26.671104       6 store.go:349] ignoring add for ingress microbot based on annotation kubernetes.io/ingress.class with value 
  </code></pre>
  <p>deployment manifest:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Namespace
  metadata:
    name: development
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: microbot
    namespace: development
  spec:
    strategy:
      type: Recreate
    replicas: 1
    selector:
      matchLabels:
        name: microbot
    template:
      metadata:
        labels:
          name: microbot
      spec:
        containers:
        - name:  microbot
          image: dontrebootme/microbot:v1
          imagePullPolicy: Always
          envFrom:
          ports:
          - containerPort: 8080
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: microbot
    namespace: development
  spec:
    ports:
    - port: 80
      targetPort: 8080
      protocol: TCP 
    selector:
      name: microbot
  ---
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: microbot
    namespace: development
    annotations: 
      kubernetes.io/ingress.class: &quot;nginx&quot;
      nginx.ingress.kubernetes.io/rewrite-target: /
  spec:
    rules:
    - http:
        paths:
        - path: /microbot
          backend:
            serviceName: microbot
            servicePort: 80
  </code></pre>
  <p>I'm wondering if there's something I am not doing correctly. Any help would be much appreciated.</p>