title: Nginx Ingress giving 404 not found for any ingress resource
body: <p>Nginx controller is throwing 404 error for any ingress resource setup in local.</p>
  
  <p>nginx-ingress controller setup was made according to steps detailed in the below document and ingress controller was exposed as NodePort Service in local setup</p>
  
  <p><a href="https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md#5-access-the-live-activity-monitoring-dashboard--stub_status-page" rel="noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md#5-access-the-live-activity-monitoring-dashboard--stub_status-page</a></p>
  
  <p>The following sample ingress resources were used to test ingress
  banana.yaml</p>
  
  <pre><code>kind: Pod
  apiVersion: v1
  metadata:
    name: banana-app
    namespace: nginx-ingress
    labels:
      app: banana
  spec:
    containers:
      - name: banana-app
        image: hashicorp/http-echo
        args:
          - "-text=banana"
  
  ---
  
  kind: Service
  apiVersion: v1
  metadata:
    name: banana-service
    namespace: nginx-ingress
  spec:
    selector:
      app: banana
    ports:
      - port: 5678 
  </code></pre>
  
  <p>apple.yaml</p>
  
  <pre><code>kind: Pod
  apiVersion: v1
  metadata:
    name: apple-app
    namespace: nginx-ingress
    labels:
      app: apple
  spec:
    containers:
      - name: apple-app
        image: hashicorp/http-echo
        args:
          - "-text=apple"
  
  ---
  
  kind: Service
  apiVersion: v1
  metadata:
    name: apple-service
    namespace: nginx-ingress
  spec:
    selector:
      app: apple
    ports:
      - port: 5678 # 
  </code></pre>
  
  <p>ingressfile.yaml</p>
  
  <pre><code>apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: example-ingress
    namespace: nginx-ingress
    annotations:
      ingress.kubernetes.io/rewrite-target: /
      kubernetes.io/ingress.class: nginx
  spec:
    rules:
    - http:
        paths:
          - path: /apple
            backend:
              serviceName: apple-service
              servicePort: 5678
          - path: /banana
            backend:
              serviceName: banana-service
              servicePort: 5678
  </code></pre>
  
  <p>Following is the list of services</p>
  
  <pre><code>NAMESPACE       NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  nginx-ingress   apple-service          ClusterIP   10.111.230.109   &lt;none&gt;        5678/TCP                     95m
  nginx-ingress   banana-service         ClusterIP   10.102.139.127   &lt;none&gt;        5678/TCP                     95m
  nginx-ingress   nginx-ingress          NodePort    10.97.65.187     &lt;none&gt;        80:30207/TCP,443:31031/TCP   6d23h
  </code></pre>
  
  <pre><code>[root@kube01 ingress]# kubectl get ing
  NAME              HOSTS              ADDRESS   PORTS     AGE
  example-ingress   *                            80        131m
  </code></pre>
  
  <pre><code>[root@kube01 ingress]# kubectl describe ing example-ingress
  Name:             example-ingress
  Namespace:        default
  Address:
  Default backend:  default-http-backend:80 (&lt;none&gt;)
  Rules:
    Host  Path  Backends
    ----  ----  --------
    *
          /apple    apple-service:5678 (10.244.2.7:5678)
          /banana   banana-service:5678 (10.244.1.11:5678)
  Annotations:
    ingress.kubernetes.io/rewrite-target:  /
  Events:                                  &lt;none&gt;
  </code></pre>
  
  <p>If I directly curl the pods as follows, it's working as expected</p>
  
  <pre><code>[root@kube01 ingress]# curl http://10.244.2.7:5678/apple
  apple
  [root@kube01 ingress]# curl http://10.244.1.11:5678/banana
  banana
  [root@kube01 ingress]#
  </code></pre>
  
  <p>However if I try to access it through Nodeport ingress controller I'm always getting not found as below</p>
  
  <pre><code>[root@kube01 ingress]# curl http://xx.xx.xx.193:30207/apple
  &lt;html&gt;
  &lt;head&gt;&lt;title&gt;404 Not Found&lt;/title&gt;&lt;/head&gt;
  &lt;body&gt;
  &lt;center&gt;&lt;h1&gt;404 Not Found&lt;/h1&gt;&lt;/center&gt;
  &lt;hr&gt;&lt;center&gt;nginx/1.17.2&lt;/center&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  [root@kube01 ingress]# curl http://xx.xx.xx.193:30207/banana
  &lt;html&gt;
  &lt;head&gt;&lt;title&gt;404 Not Found&lt;/title&gt;&lt;/head&gt;
  &lt;body&gt;
  &lt;center&gt;&lt;h1&gt;404 Not Found&lt;/h1&gt;&lt;/center&gt;
  &lt;hr&gt;&lt;center&gt;nginx/1.17.2&lt;/center&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  [root@kube01 ingress]#
  
  </code></pre>
  
  <p>curl <a href="http://xx.xx.xx.193:30207/apple" rel="noreferrer">http://xx.xx.xx.193:30207/apple</a>
  curl <a href="http://xx.xx.xx.193:30207/banana" rel="noreferrer">http://xx.xx.xx.193:30207/banana</a> 
  The above statements should display apple and banana respectively when accessed from outside or from a browser</p>
  
  <p>Update: I went into the pod where the ingress controller is running and checked its config file and found that ingress resources are not applied to the controller. I tried relaunching the container with no luck. I also tried reapplying the ingress resource but the config file in the pod is not changing. Everything that goes to controller is throwing 404 as in location tag in the config file. Any help is appreciated on why the resource is not getting applied to the controller</p>
  
  <pre><code>root@nginx-ingress-685d7964cd-djvgf:/etc/nginx# cat nginx.conf
  
  user  nginx;
  worker_processes  auto;
  daemon off;
  
  error_log  /var/log/nginx/error.log notice;
  pid        /var/run/nginx.pid;
  
  events {
      worker_connections  1024;
  }
  
  http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;
  
      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
  
  
      access_log  /var/log/nginx/access.log  main;
  
  
      sendfile        on;
      #tcp_nopush     on;
  
      keepalive_timeout 65s;
      keepalive_requests 100;
  
      #gzip  on;
  
      server_names_hash_max_size 512;
  
  
      variables_hash_bucket_size 256;
      variables_hash_max_size 1024;
  
      map $http_upgrade $connection_upgrade {
          default upgrade;
          ''      close;
      }
  
  
  
  
  
      server {
          listen 80 default_server;
          listen 443 ssl default_server;
  
          ssl_certificate /etc/nginx/secrets/default;
          ssl_certificate_key /etc/nginx/secrets/default;
  
          server_name _;
          server_tokens "on";
          access_log off;
  
  
  
          location / {
             return 404;
          }
      }
      # stub_status
      server {
          listen 8080;
  
          allow 127.0.0.1;
          deny all;
          location /stub_status {
              stub_status;
          }
      }
  
      include /etc/nginx/config-version.conf;
      include /etc/nginx/conf.d/*.conf;
  
      server {
          listen unix:/var/run/nginx-502-server.sock;
          access_log off;
  
          location / {
              return 502;
          }
      }
  }
  
  stream {
      log_format  stream-main  '$remote_addr [$time_local] '
                        '$protocol $status $bytes_sent $bytes_received '
                        '$session_time';
  
      access_log  /var/log/nginx/stream-access.log  stream-main;
  
  
  }
  root@nginx-ingress-685d7964cd-djvgf:/etc/nginx#
  
  </code></pre>